Shekhar (@indianspeedster) header
Shekhar (@indianspeedster) profile picture

Distributed Training & Inference | MTS @amd | @nyuniversity ‘24 | Views are my own. https://nitter.cf/t.co/cjd2SigzVO

San Jose, CA
Joined March 2016
Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)Photo by Shekhar (@indianspeedster)
First Hackathon Win in United States....!!!
This year, #Tandon students pulled out every trick in the #book at the annual #HackDibner Competition. Congrats to first place winners Trick Coders, computer engineering grad students Priyanka Bose & Chandra Shekhar Pandey. #NYUTandonMade https://nitter.cf/t.co/d1usnBEsDY
1
11
5,930
Shekhar retweeted
We also find that at sufficiently high bs, local grouped GEMM (EP1) is more efficient than MegaMoE
1
2
11
447
💯/💯
Replying to @yunta_tsai
A petition is underway to reinstate Jensen and I’m signing it too
1
1
1
55
Shekhar retweeted
Replying to @yunta_tsai
A petition is underway to reinstate Jensen and I’m signing it too
7,508
20,259
3,133
228,993
10,857,295
Shekhar retweeted
In March, I emailed Jensen when @inferact's compute provider took back our capacity, leaving us unable to develop @vllm_project for Blackwell. He replied within the SAME minute and had the issue resolved overnight. The entire NVIDIA team, led by Jensen, operates like an unstoppable armada. Jensen moves fast, so is his "exo-skeleton". Everyone in the industry respects and trusts him. That's how you push open intelligence forward. Thank you Jensen, for saving Inferact and vLLM.
In June @NaderLikeLadder emailed Jensen about a crazy idea we had over dinner to make Local AI 10x better on NVIDIA hardware. Jensen replied in 7 minutes. He saw this as important work, and approved a war room for us to work on this at NVIDIA HQ. For a month, NVIDIA HQ was the @exolabs satellite office. We were overwhelmed by the support we got. We were back to back the first week with smart NVIDIANs coming in to our war room to help us. Creators of CUDA, Dynamo, people who built NVIDIA. Long story short with NVIDIA we managed to improve their Local AI playbooks by 9.6x (almost the 10x goal we set out with). We’ll be publishing a whitepaper on this work. We were just a small startup with a crazy idea. Jensen took a bet on us. Jensen cares deeply about Open Source AI and Local AI. I don't believe Jensen is replaceable, he's a 1 of 1 human, but I hope NVIDIA will continue to aggressively support Open Source AI and Local AI, and continue to take bets on small startups with crazy ideas. Thank you Jensen
31
317
30
4,517
207,705
There's still a lot of juice left in existing draft model architectures, just from training them better. DStill keeps the DFlash drafter architecture unchanged and gets up to 28% higher acceptance length and 25% more speedup at the same inference cost. Great work by @shreyask269 and @madhav_kanda_ . A few training insights from the paper I liked: 1. Running vanilla DFlash for 8 denoising passes lowers acceptance length (4.45 → 3.70 on GPT-OSS-120B), because it only saw fully masked blocks in training. A drafter trained on partially revealed blocks improves with 8 passes (4.50 → 5.31). 2. They distill that 8-pass teacher into a one-pass student with the same architecture. The student is trained to predict the teacher's final block from any state on its refinement trajectory, and recovers 29-36% of the multi-step gain in one pass. 3. Distillation uses a curriculum: the student trains on both intermediate and fully masked states, and the share of fully masked states rises from 0.5 to 0.85 over training. This beats training on fully masked states only. 4. Longer training doesn't explain the gain. DFlash trained for twice as many epochs gets slightly lower acceptance length. The DStill gains over DFlash hold at concurrency up to 32, and the method also improves DSpark (+4.5% acceptance length).
Can diffusion drafters benefit from multi-step denoising while still drafting in a single pass? Excited to share DStill, our new work on accelerating block diffusion speculative decoding! 🚀 We learn from the drafter’s denoising trajectories during training to improve draft acceptance, while keeping drafting to a single forward pass at inference. ⚡ Up to 6.45× speedup over autoregressive decoding. 📄 arxiv.org/abs/2610.27379 A thread 🧵👇
3
7
422
Shekhar retweeted
Megakernel blog is out now We implement a full layer (MoE + Attention) megakernel, achieving 1100 toks/s for Muse Spark, 2.1x faster than baseline. Nvidia's Instruction cache makes it really hard to write bigger kernels. We ended up tuning both instruction and data caches at all levels for extreme latency hiding! With this, some warps of next operation can start running before previous operation has finished - redefining what GPU utilization means. Full blog post: lmsys.org/blog/2026-11-05-mu…
Muse Spark Open: 517 → 1,109 tokens/s on 8× GB300 with SGLang megakernels. One kernel launch per layer, overlapping compute, memory loads and communication. Total launches per decode step drop from 1,450 to 97. lmsys.org/blog/2026-11-05-mu… In our new technical post, Pranjal Shankhdhar explains what makes this work: - Prefetching weights into shared memory and L2 before they’re needed. - Warming the instruction cache so large kernels don’t stall fetching their own code. - Using data itself as a readiness signal for cross-GPU communication. - Fusing the draft model to make drafting 2.6× faster. The post also includes measured timelines, CUDA/PTX examples, and the tradeoffs behind stopping at one megakernel per layer.
13
62
9
601
34,306
I finally got the time to read the JetSpec++ paper. Some thoughts. It reports impressive results on Qwen3-8B: up to 7.45× speedup over autoregressive decoding, 1.54× over DSpark, and 1,361 tok/s on GB200 GPUs. But imo the headline numbers are misleading. The AR baseline runs on a single GB200, while JetSpec++ uses two. Table 3 compares JetSpec++ on 2 GPUs against DSpark and DFlash on 1. Simply running two replicas (DP=2) would, by linear extrapolation, give DSpark 1,804 tok/s and DFlash 1,591 tok/s, both exceeding JetSpec++'s 1,361 tok/s on MATH-500. I'd argue JetSpec++'s speedup over DSpark relies heavily on the additional compute: acceptance lengths are nearly identical (9.41 vs 8.98 on MATH-500), so almost all of the 1.51× comes from hiding drafting with an additional GPU. In fairness, JetSpec++ optimizes single-request latency, where DP=2 wouldn't help. But even latency-sensitive serving runs at low concurrency, not batch size 1, and the paper reports nothing beyond batch 1. And in edge scenarios, where batch-1 latency does matter, a second accelerator dedicated to drafting is a big ask.
Introducing JetSpec++: we find causal parallel drafting paired with hidden states predictor and anchor token decoder can effectively draft the next future block before the current one is verified, and spending the freed time on self refinement. This allows drafting to be pipelined with verification, hiding almost all of its drafting cost. JetSpec++ reaches up to 1.54x end-to-end speedup over DSpark and 7.45x over autoregressive decoding while keeping lossless, at 1,361 TPS on Qwen3-8B and 367 TPS on Qwen3.8-27B in BF16. With NVFP4 and kernel optimizations, JetSpec++ further reaches over 4,000 TPS on Qwen3-8B and over 900 TPS on Qwen3.8-27B, on two GB200 GPUs. ⚡️ Project page: jetspec-project.github.io/je…
1
2
6
257
Inferact is 🐐
Inferact now serves more than 7 trillion tokens a day. That traffic runs on the Inferact Platform, which serves frontier models with optimized vLLM across heterogeneous hardware and diverse workloads. For context, OpenRouter averaged 19 trillion tokens a day of open-weight traffic over the past 30 days, across all of its providers. The platform improves through a data flywheel. Production traffic shows us where to improve vLLM, partnerships with accelerators like NVIDIA, AMD, and Google bring those improvements to new hardware, and partnerships with model labs like MiniMax and Kimi bring them to new models from launch day. Next, we’re automating that flywheel with agents that find and validate optimizations on their own, so the platform improves with every token it serves. inferact.ai/news/7-trillion-…
3
112
Shekhar retweeted
Third post in our AMD GPU engineering deep-dive series! Jungwook walks through warp-pipelining (aka block ping-pong) across MI300, MI350, and MI450: explicitly staggering wave groups to overlap matrix compute with data preparation, achieving performance on par with hand-tuned hipBLASLt assembly for compute-bound GEMMs. Another showcase of the team's deep investment in foundational optimization techniques: exposing low-level scheduling control through high-level Gluon block programming. Less code, fewer tokens (especially for agents!), easier to verify, and no need to trade away performance. rocm.blogs.amd.com/software-…
4
13
101
3,751
Shekhar retweeted
Wrote a new blog on GPU comms with a lot of SGLang internals.
3
40
447
10,393
Shekhar retweeted
🚨🚨🚨 vLLM is all you need to reproduce DeepSeek’s performance. With Wide-EP and EPLB, vLLM can match DeepInfer’s decode performance within 6% and generate $39/gpu/hr in revenue at DeepSeek’s off-peak pricing, at 200 P90 TPS on the AgentX workload. Specifically: • Under DeepSeek’s decode configuration (DeepSeek-V4.1-Flash, 131K context, nextn=3), vLLM on GB200 achieves throughput within 6% of DeepInfer’s published results across batch sizes 8–256. • On AgentX, vLLM pushes the Pareto frontier further on GB300, reaching up to 1M tokens/s/GPU and generating $38.96/gpu/hr in revenue at 196 P90 TPS, based on DeepSeek’s off-peak API pricing. • Also, vLLM extended low-latency kernel support for DeepSeek-V4.1-Flash, achieving up to 826 TPS per user on AgentX.
36
96
45
1,035
114,753
Replying to @teortaxesTex
"i don't like a particular benchmark so i just slap my own benchmark into the same X/Y axis" what the fuck are you even saying
2
2
47
1,295
Shekhar retweeted
Introducing d3-turbo: our newest state-of-the-art multimodal foundation decision model⚡ #1 on both Jev Decision Index boards. It points(box), follows and sees images in your questions. StateOnce brings 15x faster than d3 flash 🤗 huggingface.co/vllm-sr/d3-tu… #vLLM #Jev #Decision
29
44
4
429
21,378
Shekhar retweeted
Today, we are announcing an engineering collaboration with @AMD to strengthen production agentic inference on AMD Instinct GPUs in @vllm_project. AMD brings the Instinct platform, ROCm engineering, and the AITER kernel library, while Inferact and the vLLM community bring model integration, serving features, and day-0 enablement. Together, we are focused on agentic workloads with long-running sessions, long prefixes, and tool calling, and each configuration will ship as a public per-model serving recipe. This work is already running in production. It shipped with this summer’s major open-model releases, and Inferact uses it on AMD Instinct clusters for our Inferact Platform customers. Everything this collaboration produces lands in open source. Read the full announcement: inferact.ai/news/inferact-co…
17
23
5
126
11,234
Shekhar retweeted
AMD GPUs go 🏎️
Today, we are announcing an engineering collaboration with @AMD to strengthen production agentic inference on AMD Instinct GPUs in @vllm_project. AMD brings the Instinct platform, ROCm engineering, and the AITER kernel library, while Inferact and the vLLM community bring model integration, serving features, and day-0 enablement. Together, we are focused on agentic workloads with long-running sessions, long prefixes, and tool calling, and each configuration will ship as a public per-model serving recipe. This work is already running in production. It shipped with this summer’s major open-model releases, and Inferact uses it on AMD Instinct clusters for our Inferact Platform customers. Everything this collaboration produces lands in open source. Read the full announcement: inferact.ai/news/inferact-co…
1
9
364
Shekhar retweeted
1/ 🚀 New on the vLLM blog: Finish RL Training Faster — Bitwise-Consistent, High-Performance, Agent-Driven Same weights don't guarantee the same policy: inference optimizations can change the logprobs that training recomputes. We make rollout and training bitwise identical while retaining high-performance inference paths. Here's how 🧵 vllm.ai/blog/2026-10-14-trai…
9
12
2
52
5,287
Shekhar retweeted
ALERT🚨🚨: @vllm_project, @Inferact & NVIDIA TOOK CO-DESIGN TO THE EXTREME WITH A GB300 NVL72 INFERENCE OPTIMIZATION TO GET 🚀 7.2x 🚀 BETTER PERF PER TCO THAN LITERALLY EVERY OTHER AI CHIP, using an inference co-design optimization called wideEP take full advantage of the NVL72 BACKPLANE. We explain below👇️ (1/3)🧵
3
8
380
Been thinking a bit about what the future holds for coding in an agentic world and I have a pretty interesting specific example that is near and dear to my heart and which I want to write down: CUDA streams.
3
6
139
8,619