Engineering @ NVIDIA Dynamo. Planet scale LLM Inference systems - Disagg Serving, Tiered KV, . ex-AWS|Trainium. @GeorgiaTech. Opinions my own.
California, USA
Joined May 2009
- Tweets1.2K
- Following158
- Followers252
- Likes189
Senthilkumar Gopal retweeted
Super excited to share an update to my blog from a few months ago on how we’re optimizing dynamo and the inference frameworks for agentic inference performance. This is a problem that we’ve been thinking a lot about and many teams have already put these optimizations and more into production!
Agentic workloads generate long, expanding context across repeated model calls, tool execution, and parallel subagents. Much of this context remains stored in the KV cache while an agent waits on external tools or pending operations.
Efficient serving relies on preserving and reusing this PyTorch tensor state while maintaining sub-second latency targets across concurrent sessions.
Read our latest blog from @nvidia to learn how NVIDIA Dynamo integrates its router, inference engine, and KV-cache manager to deliver seamless end-to-end serving for agentic workloads: bit.ly/4emlt21
Dynamo 2.0 aspires to be the best post-training inference system and this is just the first step.. Lots of exciting work in progress. Check the Dynamo GitHub for more PRs 😁
Just Dynamo things 😊
Our RL stack now supports NIXL weight transfer, reducing trainer-to-inference transfer time 9x compared with NCCL: from 86 seconds down to single-digit seconds for an 800B-parameter model, and even <4 seconds in our experiments.
For prime-rl users, this means over 25% more throughput end-to-end compared with our previous speed. It also clears the way for fault-tolerant, elastic inference scaling that NCCL's rigid process groups made difficult.
Senthilkumar Gopal retweeted
Join @baseten, @nvidia Dynamo, and SGLang on 9/10 for an evening dedicated to the infrastructure behind reinforcement learning post-training!
We will present the latest on SGLang & Miles. Miles is an RL training framework using SGLang as the rollout inference engine to make RL post-training fast and reliable. Come meet us in person!
🕐 Thursday, September 10, 6:00–9:00 PM
👉 Register now
luma.com/BaseDynSGL
Senthilkumar Gopal retweeted
Super happy to share our intention to join forces with NVIDIA in a $12,930,300,000 acquisition 💛💚
10 years after starting Hugging Face, open-source AI is at an inflection point. Thanks to the community, we’ve shown that it can be a complement, and even an alternative, to closed-source APIs. But for it to happen at larger scale, it needs more compute, more support, more collaboration and more visibility. That’s why we went to talk to Jensen, who offered to do exactly that with us.
In addition to doubling down on NVIDIA’s massive contributions to open-source AI (I called them the “King of American open-source AI” earlier this year), they’ve committed to strongly supporting Hugging Face and our mission while keeping the platform open, independent and compute agnostic. The founders and the team are all staying to keep pushing this mission forward.
Together, we think we can make open source the default way to build AI, with the goal of empowering 100 million AI builders to own their intelligence rather than rent it.
Excited about the next 10 years! 🤗🤗🤗
Senthilkumar Gopal retweeted
Most production inference services run on Kubernetes, where declarative configuration helps coordinate distributed services. Running one inference server in a Slurm job is easy; running a production-style topology across multiple containers and nodes is not.
NVIDIA’s open-source srt-slurm (github.com/NVIDIA/srt-slurm) provides a YAML-based orchestration layer for these deployments. It coordinates sbatch and srun, GPU placement, networking, readiness checks, and cleanup for systems involving prefill/decode disaggregation, multiple workers, Dynamo frontends, KV-cache-aware routers, and KV-cache offloading services.
It’s useful for quickly deploying representative inference systems for testing or benchmarking on GPU clusters that already run Slurm.
Great work to @0xishand and the rest of the @NVIDIAAI team on this!
Senthilkumar Gopal retweeted
Delivery day at our Microsoft DCs as the first production Vera Rubins arrive. A huge thank you to our partners at @nvidia and our Azure hardware and datacenter teams for all the incredible work that brought us to this milestone!
Senthilkumar Gopal retweeted
Qwen 3.8-2.4T recipes for Dynamo, straight out of the oven for Day 0!
We have SGLang and VLLM recipes for chat/agentic use cases!
Let us know what you think!
github.com/ai-dynamo/dynamo/…
Senthilkumar Gopal retweeted
Introducing Dynamo Snapshot, our approach for fast startup for inference workloads on Kubernetes, which reduces startup time from minutes to under 5 seconds.
In production inference deployments demand fluctuates over time. Cold-starting inference workloads can take minutes, leaving idle GPUs that generate no tokens and serve no requests.
Snapshot leverages GMS to enable concurrent weight restoration over a high-speed interconnect, while using Linux native AIO and parallel memfd restoration to accelerate CRIU restore performance.
And this will work with Nvidia Dynamo seamlessly 😊
A vLLM MoE deployment's DP/EP topology used to be locked in at launch — scaling or swapping config meant a full restart, in-flight traffic dropped. Elastic Expert Parallelism changes that. One API call resizes a live deployment:
curl -X POST localhost:8000/scale_elastic_ep \
-d '{"new_data_parallel_size": 16}'
Under the hood: standby comm groups span the target topology, EPLB redistributes experts across the new EP group, and weights are transferred directly between GPUs over NVIDIA NVLink/RDMA. The same runtime reconfiguration path is what fault-tolerant serving needs: evict failed ranks, redistribute their experts, bring replacements back, no restart.
Thanks to @NVIDIAAI, Sky Computing, @anyscalecompute, @RedHat_AI, and the community.
📖 vllm.ai/blog/2026-05-14-elas…
You still need to do this within the original world size 😊 so arbitrary grow/shrink might still be pending. But a great API for Fault tolerance 😉
Senthilkumar Gopal retweeted
The bitter lesson in 26 words:
Don’t be distracted by human knowledge, as AI has been historically.
Instead focus on methods for creating knowledge that scale with computation, like search and learning.
I have a variation of this for analysis of papers. To write the intuition gained, thinking behind why this method works vs. does not work for <problem>, evidence to support and the exact citation from the paper to support the decision.
Senthilkumar Gopal retweeted
Interestingly, long context attention actually enables more opportunities to stream weights into HBM, decreasing the memory requirements for weights stored in HBM at any given time, which pairs well with managing the larger KV cache.
Check out our work on this:
