Chanhyuk Lee

Just read a fun paper where they adapt off-the-shelf Qwen models with under 30B training tokens, and the result runs at 2.54× the throughput of a same-size autoregressive model at comparable quality. The idea is to have an LLM write the way image generators draw: it still goes left to right, but a chunk at a time, and each chunk starts as random noise that gets refined into real tokens all at once. Everything else works like a normal LLM, which is why they could just convert an existing model instead of training one from scratch. Each token in a chunk gets cleaned up at its own pace, and the model learns those paces itself during training. What it ends up learning is that earlier tokens should come into focus first. Nobody told it to write left to right. It figured that out on its own. Learning all this also made it a better normal LLM. Compared with the same Qwen model fine-tuned the usual way on the same data and budget, its average score across six benchmarks went up at every size (0.5B: 28.7 → 31.6, 7B: 58.4 → 60.9, 8B: 56.8 → 63.1). The authors think it's a multi-token prediction effect: practicing on whole chunks makes it better at predicting the next token too. That normal-LLM side is also what makes it fast, because the model can check its own drafts. It takes one quick shot at a whole chunk, then checks it the normal next-token way and keeps everything up to the first token it disagrees with. Since every token gets checked, the output is exactly what it would've written one token at a time (greedy decoding), just faster. With the 7B model on a single H100, that's about 2× faster and a bit more accurate with 4-token chunks, or 2.54× faster at about the same accuracy with 16-token chunks. For comparison, other speedup tricks they tested, like DeepSeek-style multi-token prediction and MARS, keep the quality but top out at 1.4×. The diffusion LLMs they compared against all lose quality, and lose more the faster you push them.