Alex Nails retweeted
SGLang pays special attention to fair, rigorous, and reproducible comparisons.
We’ve seen prefill comparisons that put GB300 results alongside DeepInfer’s published B300 numbers. Comparing across different hardware and parallelism configurations makes it difficult to isolate improvements from the inference engine itself.
We reran DeepInfer on both B300 and GB300. On our B300 hardware, it achieved higher prefill throughput than its published results at several configurations. We used its fastest tested configuration at each point as our baseline, with both engines on the same hardware and matching DP/EP configurations.
Against this stronger baseline, SGLang matches or exceeds DeepInfer’s prefill performance. We provide full reproduction details, including the remaining methodological differences, and plan to submit to AgentX soon.
DeepInfer deserves full credit. Fair comparisons and verifiable results should be the standard across the community.
DeepInfer set the bar. SGLang pushed it higher! 🚀
SGLang now matches DeepInfer performance on BOTH prefill and decode:
• Prefill: Up to 1.04× throughput on 4× GB300 and 4× B300
• Decode: Up to 7.4% throughput on 64× GB300
We also provide an end-to-end reproduction guide and a full serving stack to run it in production.
TPU TPU TPU
BREAKING NEWS🚨🚨: TODAY META CONFIRMS THEY ARE RUNNING TPU INFERENCE IN PRODUCTION. What does this mean for the CUDA moat?
For those who read our public newsletter, this is not a surprise, as on Nov 28, 2025, we publicly wrote that Meta is a TPU customer & has lots of engineers working on TPUs. Our accelerator supply chain model has details about Meta's TPU procurement!
it is out!! hope people enjoy
Muse Spark Open is built to plan, act, and use computer across agentic tasks for life and work.
You can now build a powerful personal AI agent with an open stack, and SGLang to make it:
- Ultra-responsive with NVFP4 MegaMoE and sequence parallelism
- Faster for agentic and coding tasks with our DFlash2 draft model
- Customizable for specialized use cases via fine-tuning with Miles
We’re incredibly proud to have made this possible with @AIatMeta, @NVIDIAAI, @AIatAMD, @intel, and @radixark!
Alex Nails retweeted
The full technical report for Miles is now available.
Miles is built for production-level post-training, with stability, efficiency, and flexibility at its core. The report covers its system design and how it works in practice.
Today we're launching Miles v0.1, an open-source RL framework for LLMs and multimodal models.
RL training is easy to start and hard to debug. Miles helps you ensure your run is correct, use hardware efficiently, and keep RL running at scale.
Over the past 9 months, 72 contributors have landed 1,326 commits, 85 GPU E2E CI tests, battle-testing Miles on frontier open models like Kimi K3, DeepSeek V4, Qwen 3.8, GLM 5.2, Inkling, MiniMax H3, etc.
Miles powers frontier-model development and production RL workloads at @humansand, @periodiclabs, @modal, @DecagonAI, @Eigent_AI, @nebiusai, @IBM and more, on both @NVIDIAAI and @AIatAMD hardware.
Here is what we built, and why teams picked Miles🧵
Alex Nails retweeted
Hardware heterogeneity is no longer an edge case - it is the default.
Join us at PyTorch Conference North America 2026 (Oct 20–21, San Jose) for deep dives into kernel engineering, custom compiler backends, and performance optimization across GPUs, TPUs, NPUs, and custom ASICs.
Featured speakers include:
@AWS / AWS Annapurna Labs / Annapurna ML: Maen Suleiman, Yahav Biran, Esha Lakhotia, Pinak Panigrahi
@Arm: Kavya Sri Chennoju, Thomas Cottenier
@Google: Sandeep Pokkunuri, Chris Jones, Rob Mulla, Claudio Basile
@Huawei: Jiahao Chen, Jiahao Tan
@huggingface: Michael Benayoun
@IBM/ @IBMResearch: Takeshi Yoshimura, Olivier Tardieu, Matthew Arnold, Thomas Parnell, Prasanth Chatarasi, Bardia Mahjour, Viji Srinivasan, David Grove, Antoni Viros Martin, Avery Blanchard
@intel: Yu Guangye, Eikan Wang, Panagiotis Kourdis, Tanima Dey, Huma Abidi, Ashok Emani, Qiacheng Li
@Meta: Richard Zou, Mergen Nachin, Digant Desai
@RedHat: Markell Rawls
Beijing Academy of Artificial Intelligence: Yonghua Lin
@Fujitsu_Global / Fujitsu Research of India: Abhishek Jain, N Maajid Khan
Register here and stay for AGNTCon + MCPCon that same week: bit.ly/464FTbp
Read the Hardware Acceleration & Compute Infrastructure Sessions guide: bit.ly/4yoK5z2
#PyTorchCon
Alex Nails retweeted
open-source RL-as-a-service github.com/radixark/miles
come hang with all of us!
Hanging out next Thursday 9/10 with my friends from NVIDIA Dynamo and SGLang.
We'll have tech talks on inference optimization at every layer of the stack.
Join us!
luma.com/basedynsgl
Alex Nails retweeted
@alxnails from @sgl_project @radixark speaking at our @MistralDevs meetup in SF. @jeffboudier from @huggingface 🤗 is up next!
Alex Nails retweeted
New Blog: Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon.
We built a persistent GPU process that cuts weight loading for Ling-2.6-1T FP8 from ~495s to ~0.63s, by serving post-quantized weights to new engines via CUDA IPC zero-copy.
- ~785× faster weight loading (~495s to ~0.63s on Ling-2.6-1T FP8)
- ~500× on Qwen3-235B FP8
- 93.9% shorter end-to-end boot
- Load + quantize once per GPU, every engine maps the same IPC handles
- Active-standby failover in <1s, no idle GPUs held for replicas
Built together with the Ling Infra team @AntGroup and @Alibaba_Qwen
github.com/sgl-project/sglan…
Exciting for this to be out!!!
![[Feature] Add Muse Glimmer model support by sglang-bot · Pull Request #34262 · sgl-project/sglang](https://pbs.twimg.com/card_img/2115145463213035520/KlacSbDy?format=jpg&name=800x419)
Introducing Muse Glimmer, an open-weight 30B-parameter model optimized for local, always-on agent workflows.
Muse Glimmer delivers strong performance on key agentic use cases and benchmarks compared with leading models in its size category, and is designed to run entirely on consumer hardware like a Mac or PCs with performant GPUs.
In keeping with our long tradition of sharing fundamental AI research, we’re releasing model weights under a permissive Apache 2.0 license.
🧵👇
Alex Nails retweeted
excited to be releasing open weights for muse glimmer today, a 30b model that runs on a single consumer gpu, with open weights for a version of muse spark 1.2 coming soon.
two very different models, both headed into people's hands, with more to come.
Today we're also opening the weights for Muse Glimmer, a great 30B parameter dense model that can run locally. Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model. Meta is a strong supporter of open source and I'm proud of these releases. Congrats to @alexandr_wang and the MSL team for all your great work on these models.
🗣️
RadixArk and Google Cloud are joining forces with the SGLang community to make TPU a drop-in, cost-efficient path to frontier inference.
SGL-JAX already serves the major open model families on the latest TPU generations: Gemma, Qwen, DeepSeek, GLM, Kimi, Ling, MiniMax, MiMo, Grok, plus Wan and Flux for video and image generation.
With this partnership, TPU unlocks SGLang's production feature set: 5D parallelism, Radix Cache, HiCache, quantization, speculative decoding, on TPU Pallas kernels co-built by Google, RadixArk, and the SGLang community. SGL-torchtpu opens a PyTorch-native path to all of it.
SGLang on TPU comes with the same API and the same features, so for teams already running it, getting started is simple. Going forward, TPU support for new open models will land alongside the rest, and that carries forward to each new TPU generation.
Full announcement and SGL-JAX repo 👇
Alex Nails retweeted
Big news: @Google and @RadixArk are partnering to bring @sgl_project to Google Cloud TPUs!
✅ Run SGLang on TPU today via SGL-JAX
✅ Coming soon: SGL-torchtpu for a PyTorch-native experience
✅ No more migration tax—just ultimate flexibility for devs
Learn more 🛠️: goo.gle/3U2JVOC
#GoogleCloud #TPU #SGLang #AI #DevRel
ALT Logo for RadixArk featuring an orange stylized flame icon within an arch on the left. To its right, the text reads "Radix" in black followed by "Ark" in orange.
Alex Nails retweeted
Inkling, @thinkymachines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio.
Serving and RL support are already live: you can run and shape it on an open stack, starting now.
Day 0 support on SGLang @sgl_project and Miles @radixark👇
- Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache
- Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization
- DFlash speculative decoding from @modal for low-latency serving
Launching now, blog and cookbook in the comments ⬇️
Alex Nails retweeted
The standout feature in SGLang v0.5.13 is BCG (Breakable CUDA Graph). It delivers prefill efficiency comparable to PCG (Piecewise CUDA Graph), while being significantly more flexible and compatible with advanced optimizations.
BCG is also a powerful debugging tool — it enables eager execution inside CUDA graph replay, so you can easily print debug info or inspect intermediate states.
Prototype by @csy_789. @Oasis_a19 turned it into a production-ready prefill optimization. I was fortunate to pick the name for this awesome technique. 😊
🎉 SGLang v0.5.13 is out!
First, new model support!
Nemotron 3 Ultra, Step-3.7-Flash, Command A+, plus new diffusion models: Cosmos3, FLUX.2-Klein, Ideogram 4, LingBot-World, SANA-WM, and Ernie-Image.
Here are the highlights for this release:
- Speculative Decoding V2 is now the default! Tree drafting (topk>1) for faster generation
- Breakable CUDA Graphs now make prefill faster
- Qwen 3.5 runs faster on NVIDIA Blackwell with new GDN kernels
- HiCache with UnifiedTree on by default for hybrid SWA/Mamba models
- SGLang-Diffusion now supports realtime generation! Plus progressive resolution
- Multiple performance and feature updates for DeepSeek V4
Thanks to our amazing partners and model makers:
@NVIDIAAI @AMD @intel @awscloud @boson_ai @cohere @bfl_ml @ideogram_ai @deepseek_ai @Kimi_Moonshot @Alibaba_Qwen @StepFun_ai @Baidu_Inc @robbyant_brain
Alex Nails retweeted
NYC, we're bringing the inference + finance crowd together for #NYTechWeek @Techweek_!
SGLang Happy Hour: AI Infra in Finance
🕤Wed, June 3 · 6–9 PM ET
📍1/2 Bond St, New York
Co-hosted with @HOFCapital, @CrusoeAI, @CloudflareDev, @ArklexAI. Lightning talks from inference engineers and researchers shipping into trading, research, compliance, and risk, followed by an open happy hour for networking. More surprise speakers to be announced — stay tuned 👀
Expected attendees from leading quant funds, banks, and trading firms, including Jane Street, Citadel, Two Sigma, Goldman Sachs, Bloomberg, among others.
We've also got a bartender on site and a full bar. Come have a drink with us!
Limited space. RSVP 👇
partiful.com/e/p74X9KDrgoLaD…
Alex Nails retweeted
Training DeepSeek V4 @deepseek_ai at scale? SGLang + Miles is the Day 0 path. @lmsysorg
Miles and SGLang enable full-parameter RL training for DSV4 with stability, efficiency, and broad hardware support.
✅ Verified stability
- Rollout Routing Replay (R3) and indexer replay (experimental)
- Tensor-level validation across the Miles & Megatron mixed-precision training stack
- Step-0 train-inference diff: ~0.02–0.03
✅ Efficient full-parameter RL
- DP / TP / SP / EP / PP / CP support
- Tilelang attention and indexer kernels
- FP8/BF16 rollout and FP8/BF16 training support
✅ Broad hardware support
- Verified training on NVIDIA Hopper and Grace Blackwell clusters
- Ready for DeepSeek V4 RL from Day 0
This is the exclusive Day 0 path to scale DeepSeek V4 with rock-solid reliability. Full technical docs & setup guide below! 👇
#DeepSeekV4 #SGLang #RL
