Andy Luo (@linluo77) profile picture

AI@AMD

Joined April 2008
Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)Photo by Andy Luo (@linluo77)
Andy Luo retweeted
🚀 ATOM Meets Mooncake: Scaling Agentic LLM Serving on AMD Instinct GPUs! A deep dive into how NeuroSpark AI deploys AMD's ATOM and Mooncake to serve GLM-5.3 across 32 AMD Instinct MI350X GPUs. With ATOM, ATOMesh and Mooncake's shared KV pool: 🔥 95–96% KV cache reuse on production traffic ⚡ Up to 2× throughput vs. per-server KV pools in AgentX benchmarks 🚀 Scalable agentic LLM serving across servers Read the full story 👇 kvcache.ai/blog/atom-meets-m…
5
3
21
866
7
16
4
150
31,601
AMD really cooked here. Kimi K3.1 Pro on our 8xMI350X (default SGLang recipe) is a whole different experience at 300 tok/s (up from like 60 with K3), the speed makes it often a better choice than Astra/Opus.
17
36
2
505
28,152
Andy Luo retweeted
Muse Spark Open 🤝 AMD Instinct GPUs. Day 0 support is here for the first open weights model: Muse Spark Open. Start building 👇 amd.com/en/developer/resourc…
7
13
3
75
8,519
~440 tok/s/user P90 interactivity on MiniMax M3 at concurrency=1 on MI355X. How? TileRT MegaKernel → FlyDSL MonoKernel → ATOM, fusing Indexer + GEMM + MLA + AllReduce + MoE into a single kernel launch. Open software stacks turn ideas into production performance—fast. #AMD #ROCm
1
5
22
1,127
Andy Luo retweeted
Great work from the @TileRT_AI and @AIatAMD teams, who got GLM-5.3 to 469 tok/s single-user decode with vLLM on 8× MI355X on @SemiAnalysis_ AgentX. The run uses a disaggregated setup where vLLM handles prefill and TileRT handles latency-critical decode through vLLM's V1 connector interface. 1/2
AMD MI355X becomes the first official TileRT result no AgentX! This configuration achieves an astounding 470 TPS on GLM 5.3 (FP8). This is over 40% faster than GB300 TRTLLM using FP4! Great work to the TileRT x AMD team! (1/3)🧵
19
18
1
108
13,663
Andy Luo retweeted
2 ways to win on OpenRouter, serving GLM5.3/MI355X: 1. Speed: TileRT hits 469 tps/user on AgentX, ~45% higher interactivity v. GB300/FP4: amd.com/en/developer/resourc… 2. Profits: $11.18/GPU/hr at 67 tps (SemiAnalysis AgentX, 60% utilization) => $11.18*24*265*8 = $560k profit/node/year
6
4
3
59
8,029
We welcome more hardware manufacturers to join us in jointly optimizing inference performance, OSS forever!
AMD MI355X now is close to Nvidia B200 on MiniMax-M3 serving. Thanks to @AMD team. released dual technical blogs detailing full-stack ATOM + ATOMesh co-optimization for our 428B MSA sparse MoE multimodal base model. • EAGLE3 speculative decoding for lower generation latencyCore upgrades: • MSA-aligned Page-16 KV cache for 1M ultra-long context • Layer-wise FP8 quantization avoiding vision/MoE precision drop • Distributed P/D separation architecture for large-scale clusters 🔗 Blog 1: rocm.blogs.amd.com/artificia… 🔗 Blog 2: rocm.blogs.amd.com/software-…
15
23
3
567
43,583
Agent workloads bring long contexts, bursty traffic, and frequent tool calls, creating new demands for AI serving. Together with @AMD , we rebuilt the stack for Kimi K2.6 on AMD Instinct™ MI355X, with scheduler-aware multi-tier KV caching. Up to 3.2× smaller p99 TTFT and 7.7% higher total-token throughput, with no accuracy tax. Happy to see more Kimi running on AMD chips! Tech blog: amd.com/en/developer/resourc…
53
107
17
1,692
120,041
Great to see this out! Worked with the team on bringing efficient large-scale RL post-training to MI300-class clusters — lots more coming 🚀 @linluo77 @lizli202503 @roaner
🚀 New blog: ROCm Support for Miles: Large-Scale RL Post-Training on AMD Instinct™ GPUs Together with @AMD, Miles brings end-to-end RL pipelines to MI300/350-class clusters: ⚡️ Rollout generation dominates RL compute, and AMD’s HBM bandwidth directly addresses this bottleneck 🧠 AIME accuracy improved from 0.665 → 0.729 across training on Qwen3-30B-A3B with GRPO 💾 MI300X delivers ~1.1–1.3k tok/GPU/s rollout throughput ⏱️ Mean step time 388.5s on a single 8-GPU MI300X node (32×8 sampling, 8k response cap) 🔧 Multi-turn agentic training validated ... and more optimizations to come 🔥
2
2
294
Andy Luo retweeted
🚀 New blog: ROCm Support for Miles: Large-Scale RL Post-Training on AMD Instinct™ GPUs Together with @AMD, Miles brings end-to-end RL pipelines to MI300/350-class clusters: ⚡️ Rollout generation dominates RL compute, and AMD’s HBM bandwidth directly addresses this bottleneck 🧠 AIME accuracy improved from 0.665 → 0.729 across training on Qwen3-30B-A3B with GRPO 💾 MI300X delivers ~1.1–1.3k tok/GPU/s rollout throughput ⏱️ Mean step time 388.5s on a single 8-GPU MI300X node (32×8 sampling, 8k response cap) 🔧 Multi-turn agentic training validated ... and more optimizations to come 🔥
3
18
4
66
12,985
From 1 to 3,204 TFLOPS ⚡ FP8 GEMM optimization on AMD MI355X — pure HIP/C++, no assembly. 1→5 (LDS tiling) 5→30 (Matrix Cores) 30→507 (global→LDS) 507→1,166 (double buffer) 1,166→2,288 (multi-wave) 2,288→3,204 (8-wave ping-pong) Exceeds hipBLASLt at 8K matrix. Full blog: rocm.blogs.amd.com/software-…
1
9
1
62
4,339
Andy Luo retweeted
Day 0 support for Qwen 3.5 is here on AMD Instinct GPUs. By providing Day 0 support on AMD Instinct GPUs with both SGLang and vLLM, we ensure that developers have the compute power and optimized software stack needed to run these massive, high-context models at production scale. Learn how to get started → amd.com/en/developer/resourc…
6
4
1
42
12,571
Day-0 support for the Qwen 3.5 model on AMD GPUs, achieved through tight collaboration with the @Alibaba_Qwen team just in time for Chinese New Year's Eve. Support is available via both @sgl_project and @vllm_project . With SGLang: Launch rocm/sgl-dev:v0.5.8.post1-rocm720-mi30x-20260215 Inside container, run: python3 -m sglang.launch_server \ --port 8000 \ --model-path Qwen/Qwen3.5-397B-A17B \ --tp-size 8 \ --attention-backend triton \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder With vLLM: Launch rocm/vllm-dev:nightly_main_20260211 Inside container, run: pip install git+lnkd.in/gw4k6mqE VLLM_ROCM_USE_AITER=1 \ vllm serve Qwen/Qwen3.5-397B-A17B \ --port 8000 \ --tensor-parallel-size 8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder You can now run the following text/image/video input examples on Hugging Face: huggingface.co/Qwen/Qwen3.5-… huggingface.co/Qwen/Qwen3.5-… huggingface.co/Qwen/Qwen3.5-… Accuracy has been fully verified, and all AMD support code has been upstreamed. The upcoming SGLang and vLLM releases will support Qwen 3.5 out-of-the-box on AMD MI300X, MI325X, and MI355X GPUs.
5
5
1
47
6,496
✅ Day 0 support of MiniMax-2.5 on AMD GPU 2 MI300X GPUs are all you need, instead of 4 Hopper GPUs, to run it in full context. uv pip install vllm --extra-index-url lnkd.in/gJdnn3kJ VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.5 \ --tensor-parallel-size 2 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2_append_think \ --enable-auto-tool-choice \ --trust-remote-code
9
4
53
15,003
Qwen3 Optimization on AMD GPU with SGLang, full blog: lmsys.org/blog/2026-02-11-Qw…
1
6
22
1,526
Run #openclaw (aka #clawdbot and #moltbot) with local Minimax-M2.1 on AMD Developer Cloud for free. Check out the blog by @DuneDudeMahdi - lnkd.in/g7az4G5q
2
7
717
How to run the new kimi-k2.5 on AMD GPU: uv venv source .venv/bin/activate uv pip install vllm --extra-index-url wheels.vllm.ai/rocm/0.15.0/r… vllm serve moonshotai/Kimi-K2.5 -tp 8 --mm-encoder-tp-mode data --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --trust-remote-code
8
1
20
6,495
How to improve LLM serving perf 3-10x with LMCache on AMD GPU for long-document QA and multi-round conversations - blog.lmcache.ai/en/2026/01/0…
122