Kyle Widmann (@KyleWidmann) header
Kyle Widmann (@KyleWidmann) profile picture

@Chutes_AI | Shaping secure, decentralized private serverless compute

Joined November 2015
Kyle Widmann retweeted
Kappa run is now stopped, not too bad overall! We passed llama 3.2 1b on ARC-C, OBQA, TQA, etc., nearly matched on ARC-E, SciQ, etc. Considering that was 9T tokens and we hit those numbers on the 576B checkpoint, AND we did so around 90% cheaper per token, I'd call it a solid preliminary win! Issues with the run: 1. In one of the Gated DeltaNet-2 heads, the per-channel decay gate underflowed, causing state wipe, and ourput RMSNorm amplified near-zero outputs ~1000x into fleet wide grad spikes. 2. Very late/slow nodes were so far behind their evidence was discarded, not a huge effect but we should have had better tolerance for stale data and partial updates. 3. We packed samples but didn't do masking/etc. which is sub-optimal for pretraining (not a huge diff but worth fixing anyways). 4. The first and last layers being GDN2 may be suboptimal also, vs. SWA, no clear evidence but might be worth switching up the order a bit for those two. We have a fix for this GDN2 issue, but while that was still running I also tested EDA (arxiv.org/abs/2606.26560v1) and it actually addresses some of the root cause directly (and seems to have better loss than GDN2 anyways) so we'll probably switch to EDA once all tests come back clean and pass the kappa's instability regime (and waiting until we have highly optimized tilelang kernels for EDA to keep MFU/throughput high). Onwards and upwards 🚀
13
23
4
133
19,146
Kyle Widmann retweeted
Just for fun I kicked off a run of a 176b parameter model on ~140 steps to prove feasibility - 4 separate nodes across the internet using the "Parallax" method, works like a charm. Still need a more concrete plan on dataset curation, phases, context elongation, etc. etc. before a full run is ready of this scale, but at least we know 176b should be no problem at all.
You can force an architecture into decentralized training, or you can craft an architecture specifically for decentralized training (and ultimately insanely performant inference). Stop trying to fight gravity. "Parallax", so far, is working exceptionally well. Surrogates, expert offload, stratified decoupled diloco, etc. All pretty great to make training work at scale across the internet. But... The biggest benefits are actually AFTER training, via specific model architecture choices that drastically reduce VRAM and increase throughput. These giant MoEs are wasting so much VRAM and compute for all of the many routed experts, it's insane actually. So much research already showing that MoEs can be massively compressed/pruned/ternary-distilled/etc., not to mention they are generally comparable to around 2.5-3x dense model equivalents of the ACTIVE parameter count. KV cache is a huge bottleneck, and these huge routed experts are often "dead weight" that consume it needlessly. We need to do better, so we will.
21
49
17
273
94,164