Your open-source AI ally. We are committed to making production-grade AI inference as accessible and reliable as electricity, powered by vLLM.
Joined October 2023
- Tweets625
- Following1.4K
- Followers1.3K
- Likes1K
EmbeddedLLM retweeted
Great work from the vLLM Semantic Router, KR Labs, and @AMD teams on Vela 3.0!
Routing decisions and span extraction in one call, with six open models from 0.3B to 27B. More flexibility in what the router can detect and how small it can run.
Introducing Vela 3.0 📡 Name it. Vela finds it.
Six open models (0.3B–27B): decisions + the spans you name. One call.
⚡ 380M: #1 Decision Index ≤500M
🎯 Beats GPT-5-mini at zero-shot classification
🔍 18 ms per span question
🤗 huggingface.co/collections/v…
#Jev #Decison
EmbeddedLLM retweeted
Great work from @AMD and the vLLM Semantic Router team on d3-ultra! Open multimodal decision-making with per-answer probabilities, visual grounding, and video tracking—trained and served on AMD Instinct GPUs.
Apache-2.0 weights on @huggingface.
Meet d3-ultra: our newest open 27B multimodal decision model.
#2 of 230 on the Jev Decision Index text board, +3.03 over d3.
Text, images, video: it boxes what it finds and tracks it through time.
Open weights (Apache-2.0): huggingface.co/vllm-sr/d3-ul…
#Jev #Decision
EmbeddedLLM retweeted
vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64.
DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.
EmbeddedLLM retweeted
🎉 MiniMax M3.1 Flash from @MiniMax_AI has Day-0 support in vLLM!
Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack.
What the model can do, and the vLLM engineering that makes it practical to serve. 👇
Model: huggingface.co/MiniMaxAI/Min…
1/5
MiniMax M3.1 Flash weights are now public. 428B parameters total, 23B activated per token.
Agentic performance: A generational leap in agentic tasks, coding, and cybersecurity — 76.2 on BankerToolBench, 73.7% on DeepSWE v1.1, 61.7 on OSWorld V2, 29.6 on Agents' Last Exam, and 82.2 on CyberGym.
Efficient by design: MiniMax Sparse Attention and DSpark speculative decoding improve inference efficiency.
Hugging Face:
huggingface.co/MiniMaxAI/Min…
ModelScope:
modelscope.cn/models/MiniMax…
API: platform.minimax.io
M Plan: minimax.io/m-planMiniMax
MCode for desktop: agent.minimax.io/download
EmbeddedLLM retweeted
Outstanding work by the team!!! Optimizing World Action Model on AMD GPU by combining kernel level optimization via GEAK (github.com/AMD-AGI/GEAK), fewer denoising steps and compact action expert to reduce the latency from 222.55ms to 8.83ms. @AIatAMD
Fully open
📝 Blog: rocm.blogs.amd.com/artificia…
💻 Code: github.com/AMD-AGI/faster-ac…
🤗 Checkpoint: huggingface.co/collections/a…
EmbeddedLLM retweeted
High throughput on the grill. Zero latency on the drinks.
vLLM maintainers and friends gathered during the PyTorch Conference for an evening of Brazilian steak, drinks, and great conversation! 🥩🥂
Grateful for this community and everyone who keeps vLLM moving forward. 🚀
EmbeddedLLM retweeted
vLLM is all you need to reproduce DeepSeek's performance.
🚨🚨🚨 vLLM is all you need to reproduce DeepSeek’s performance.
With Wide-EP and EPLB, vLLM can match DeepInfer’s decode performance within 6% and generate $39/gpu/hr in revenue at DeepSeek’s off-peak pricing, at 200 P90 TPS on the AgentX workload.
Specifically:
• Under DeepSeek’s decode configuration (DeepSeek-V4.1-Flash, 131K context, nextn=3), vLLM on GB200 achieves throughput within 6% of DeepInfer’s published results across batch sizes 8–256.
• On AgentX, vLLM pushes the Pareto frontier further on GB300, reaching up to 1M tokens/s/GPU and generating $38.96/gpu/hr in revenue at 196 P90 TPS, based on DeepSeek’s off-peak API pricing.
• Also, vLLM extended low-latency kernel support for DeepSeek-V4.1-Flash, achieving up to 826 TPS per user on AgentX.
EmbeddedLLM retweeted
🎉 Step 5 Preview has Day-0 support in vLLM.
Sparse attention, MoE, multi-token prediction and vision, with serving work across Blackwell and Hopper.
The work spans the sparse indexer, FP8 MegaMoE and cache transfer. Here is how the pieces fit together, with kernel results and validation below. 🧵
1/6
EmbeddedLLM retweeted
🎉 Day-0 support for Ling-3.1-flash is here in vLLM!
~560B total parameters, ~25B active per token, and up to 1M context. From a rough brief to websites, mobile UIs, and interactive prototypes—serve it on your own infrastructure with vLLM.
Congrats to @AntLingAGI!
EmbeddedLLM retweeted
ALERT🚨: AMD JUST LANDED SUPPORT FOR MEGAKERNEL DECODE IN UPSTREAM VLLM WELL BEFORE NVIDIA. This marks one of the first times that a major vLLM feature is only supported on ROCm and not on CUDA yet.
MegaKernels is a technique that is massively important for high-interactivity decoding by reducing the # of kernel launches by 10x.
This marks an❗️important milestone❗️for ROCm vLLM to land major ROCm-exclusive features. CUDA vLLM doesn't even have draft PRs for upstream vLLM megakernel support yet.
EmbeddedLLM retweeted
It was an incredible honor to receive the National Medal of Science from @POTUS today.
My parents came to America with big dreams and a deep belief in education and hard work. This country gave me the opportunity to pursue my passion for semiconductors and work with extraordinary people to push the boundaries of what is possible.
I am so excited about what we can accomplish together as America continues to lead the world in science and technology.
.@POTUS presents the National Medal of Science to Sergey Brin, @JensenHuang, @elonmusk, and @LisaSu, and the National Medal of Technology and Innovation to @MichaelDell and @satyanadella.
EmbeddedLLM retweeted
AMD MI355X posts the best margin on every model in SemiAnalysis InferenceX agentic profit estimates:
Kimi K3: 45.7% vs 43.8% (GB300)
GLM 5.3: 63.4% vs 63.2% (GB200)
MiniMax M3: 51.5% vs 48.4% (B300)
DeepSeek V4 Pro: 87.9% (no green data)
inferencex.semianalysis.com
EmbeddedLLM retweeted
Zero mismatches. Bit-for-bit aligned. 🔥
Incredibly proud of our team's work integrating with vime @vllm_project. Rock-solid RL pipelines are now a reality on @AMD ROCm.
Read the deep dive below and star our repo⭐ github.com/RL-Align/RL-Kerne… – at Singapore
Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout.
The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides.
Deep dive:
vllm.ai/blog/2026-09-14-rl-k…
EmbeddedLLM retweeted
Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout.
The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides.
Deep dive:
vllm.ai/blog/2026-09-14-rl-k…
EmbeddedLLM retweeted
Massive shoutout to TJ & Team at @EmbeddedLLM; they are the lead maintainers & CODEOWNERS of vLLM ROCm! Up until May 2026, they did not even have persistent access to an MI355X cluster, but after SemiAnalysis worked with & convinced AMD leadership of something that is quite obvious in hindsight, they finally got a persistent cluster to make MI355X great on AMD!
Super proud of the @EmbeddedLLM Team, highly respected within the @vllm_project community.
EmbeddedLLM retweeted
🎉 Congrats to @XiaomiMiMo on MiMo-V2.6: day-0 support in vLLM for both sizes.
Two MoEs with text, image, video and audio in one checkpoint: Pro is 1.02T total with 42B active, Flash is 309B with 15B active. 1M context, native FP8 weights. Built-in DFlash spec decoding drafts 7 tokens a step.
Same architecture class as V2.5, and the serving pieces carry over: hybrid sliding-window plus global attention, the mimo reasoning and tool-call parsers, DFlash spec decoding.
Thanks to the @XiaomiMiMo team for opening the weights and the RL stack! 🙌
🔗 recipes.vllm.ai/XiaomiMiMo/M…
Introducing Xiaomi MiMo-V2.6 — Pro & Flash.
Frontier intelligence, all the modalities, built in public.
🔹 Two omnimodal models, advancing through scaled reinforcement learning
🔹 Pro performs on par with Claude Opus 5 and GPT-5.6 Sol across most agent benchmarks
🔹 Pro scores 46 on the Artificial Analysis Intelligence Index — the highest among open-source models
🔹 Stronger coding, computer use, 3D reasoning and creative capabilities
🔹 Open model weights, technical report, RL environments and training code
Blog：mimo.xiaomi.com/mimo-v2-6
EmbeddedLLM retweeted
10 things to know before joining the AMD AI Developer Program 👇
1. It’s free. Anyone can join, whether you are a student starting out or a seasoned ML engineer.
2. You get $100 in cloud credits on day one. Enough to prototype, benchmark, and get moving on AMD tech.
3. You don’t need any hardware to start. Build immediately with cloud access.
4. There’s a private Discord with a community of engineers who will provide technical discussion and help.
5. You get early invites to dev events and workshops. Before they’re public. Before they fill up.
6. Monthly Ask the Expert hours with open-source partners. Ask questions. Go deep. Learn the tricks of the trade.
7. Your projects can get featured on official AMD Developer channels and at events.
8. You’re automatically entered into monthly hardware raffles. No forms. No extra steps.
9. You get a free month of @DeepLearningAI Pro. High quality courses for you to expand your skills.
10. The more you participate, the more you unlock. More access, more visibility, more opportunities.
If you’re building, training, or optimizing AI models, we’d love to have you join us!
EmbeddedLLM retweeted
🔎 New blog: day-0 support gets a model running in vLLM. From there, the community keeps making it faster.
@AMD and @EmbeddedLLM wrote up that process for MiniMax M3 on Instinct MI355X: follow the bottleneck.
Read it for the 3–4× serving gains and performance tips you can reuse on the next model optimization. 👇
🔗 vllm.ai/blog/2026-09-10-mini…
EmbeddedLLM retweeted
Model Runner V2 is the default runner for all models as of vLLM v0.29.0. 🎉
594 commits from 277 contributors, 91 of them first-timers.
Highlights:
🌙 Another optimization round on Kimi-K3, latent tail down to Mamba metadata
🐳 DeepSeek-V4 shared experts fused into MegaMoE
🧠 Mamba prefix caching keeps internal prefill checkpoints: 9~25% better TTFT
⚡ Speculative decoding reports per-request acceptance stats over the API
🔄 RL weight sync adds the sharded_rdt P2P backend: each worker pulls only its own TP/EP slice
🆕 New models: Hy4-preview, Qwen3.8-Flash-Next, GraniteSWA, NemotronH Omni Reasoning V3
Thread 👇
