Local AI Alex (@AI__Alex) header
Local AI Alex (@AI__Alex) profile picture

Chem engineer / chemist & founder. Running open models on 4× DGX Spark (will be 8 soon). Writing on local AI: real industrial use & my occasional perspective

Brisbane, Queensland
Joined September 2023
Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)Photo by Local AI Alex (@AI__Alex)
NVIDIA's own tooling clusters up to 4 DGX Sparks on a switch. I've designed for 8: 1 TB unified memory, a 200G RDMA fabric, and four cluster modes switched in software. 100% local inference for two businesses. 18 drawings below. Tell me what you'd change.
3
1
8
469
Local AI Alex retweeted
Banger paper from Amazon on self-improving agents. (bookmark it) CompoundingKernel writes GPU kernels from scratch with over a thousand agents: - 235 problems, every one with a verified correct kernel - First place on the public L1, L2 and Quantization boards of SOL-ExecBench on NVIDIA B200 - 4.40x faster than the reference baseline on the 209 public problems The agents start from each problem's PyTorch definition, with no reference kernels and no official scores. An archive outside the model keeps every correct kernel and its lineage. An LLM controller reads the archive, picks a parent, and gives each child agent one direction. Checks outside the model decide correctness. The controller beat a UCB-style tree search on 35 of 38 problems at equal wall time. It chose parents about as well, but made 2.4x as many correct attempts. One restart fix raised the fleet's evaluation rate 3.6x. If you scale an agent search, measure how many reliable evaluations per hour it runs before adding agents. arxiv.org/abs/2611.19230 Chat with Paper: academy.dair.ai/papers/compo…
17
16
4
144
10,381
I claimed the “ai__alex” handle on @ThePirateFace 🏴‍☠️ Where open weight models never die, and are immortalized as torrents. You'll need one too, for what comes next. Claim yours ↓ pirateface.co/claim?ref=ai__…
4
90
Everybody who cares about local AI needs to the following right now: 1. Purchase large, inexpensive (>20TB) external drives. 2. Ask an LLM for a Python script to auto-download Hugging Face model collections fitting those drives. 3. Unplug and store the drives for later use. Do this right now; do this today! After the hf takedown, head to pirate face to upload and seed. This is now an urgent matter & we need enough ppl need to act immediately. Local AI Influencers: please tweet similar advice to your audiences. @thepirateface @LLMSherpa
12
14
1
92
3,082
Local AI Alex retweeted
🤖 Made with AI
84
877
68
9,946
374,145
And so begins the third chapter: -- Jensen's Revenge!!!
1
60
Can a ~A$26,000 Mac Studio M5 Ultra (512GB) really beat an A$120,000+ 8x NVIDIA RTX PRO 6000 rig at running giant LLMs? 🤔 Recent Qwen 3.8 Flash Next benchmarks show the M5 Ultra hitting 125.8 tok/s vs 122.3 tok/s on 8x RTX Pro 6000s. Wait... how?! Here is what's going on 🧵👇
4
1
1
6
1,029
4/ Now look at Prompt Processing & Dense Models - where NVIDIA dominates 🔥 In prompt processing, the 8x RTX Pro 6000 rig hits 9,230 tok/s vs 3,427 tok/s on the M5 Ultra (2.7x faster). On dense 405B models or 10+ concurrent users, Blackwell is 4x to 7x faster 📈
1
111
5/ The takeaway for local AI builds 🛠️ ✅ Solo dev running huge sparse MoEs (DeepSeek R1, Qwen Flash)? A ~A$26k 512GB M5 Ultra gives you flagship token speeds at ~350W. ✅ Multi-user serving, huge 250K prompts, or dense 405B models? Get the 4x RTX PRO 6000 Max-Q rig 🏆
153
Why do almost all serious AI servers stop at 8 GPUs (TP=8) per node instead of 6, 10, or 16? It’s not an arbitrary industry convention. 8 GPUs is the exact mathematical and physical asymptote where 5 independent laws of transformer linear algebra and silicon physics converge. A thread on the first-principles maths of TP=8: 🧵👇
1
3
136
6/7 Law 5: The "HGX-8" Industry Feedback Loop Every frontier training cluster built by NVIDIA (DGX / HGX H200 / B200) and AMD (MI300X / MI350X) has 8 GPUs per node baseboard. Because researchers at DeepSeek, Meta, Qwen, Mistral, and Cohere train and benchmark on 8-GPU nodes, they intentionally design model dimensions (64/128 query heads, 8 KV heads, FFN dimensions divisible by 8 × 128 = 1,024) to shard with 100% mathematical symmetry across 8 GPUs. When you build a TP=4 or TP=8 node, you are riding the exact mathematical rails the model weights were born on.
1
69
7/7 The Practical Takeaway for Local AI Builders: 1. Never build a 3-GPU, 5-GPU, or 6-GPU rig if you want fast single-stream Tensor Parallelism—stick strictly to 2, 4, or 8 GPUs. 2. If you want a future-proof 8-slot EPYC server (NPS1 BIOS, ACS/IOMMU off, native P2P DMA), either populate 4 high-VRAM cards (TP=4) with 4 open slots to grow, or go the full 8 cards (TP=8). 3. If you ever need 16 GPUs, build two TP=8 nodes rather than one 16-GPU PCIe box. Physics and linear algebra always win in the end.
15
Trying to cluster 4x NVIDIA DGX Sparks right now? The MikroTik CRS804 and CRS812 switches are completely unobtainium. People are trying wild breakout cable "spaghetti" meshes or settling for broken ring setups. Here is how to actually get the full 200 Gbps per node without an 804/812 (and what traps to avoid) 🧵👇
1
2
162
9/ Is the 8x 100G DAC setup actually *better* than a MikroTik CRS804 or CRS812? If you can get an 804/812 (I run an 812 with 8 Sparks), it's still the cleanest desktop setup: half the cables & whisper-quiet. Where 8x 100G wins: ✅ 25G NRZ signalling links up instantly (no finicky 50G PAM4 breakout/FEC91 tuning) ✅ Used Mellanox SN2700 has a true Spectrum ASIC (gold-standard lossless RoCEv2 vs Marvell Prestera) Where 804/812 wins: ✅ Only 1 cable per Spark (vs 2 thick twinax cables) ✅ Office-quiet (SN2700 is 1U server loud!)
1
42
10/ Once your 4 Sparks are wired up, don't jump straight into vLLM/NCCL without testing raw RoCEv2 bandwidth first. Run ib_write_bw across both rails to verify you're hitting ~197 Gbps per node: 🖥️ On Spark 1 (Server): ib_write_bw -d rocep1s0f0 -F --report_gbits & ib_write_bw -d rocep1s0f1 -p 18516 -F --report_gbits 🖥️ On Spark 2 (Client): ib_write_bw -d rocep1s0f0 <IP1> -F --report_gbits & ib_write_bw -d rocep1s0f1 <IP2> -p 18516 -F --report_gbits If both hit ~98 Gbps simultaneously, your 200G fabric is ready to rip! 🚀
30