Ali Naeimi retweeted
i'll let you in on something: inference optimization has moved up the abstraction ladder. it's not about handwriting the fastest kernels anymore. it's about how well you can define your optimization context, I/O, constraints, and testing environment for your AI agents.
Ali Naeimi retweeted
TLDR: we integrate many computational and statistical improvements for muon-style updates here: github.com/microsoft/dion .
Comp: Gram Newton Schulz, Row Selection, Megabatching, Cudagraph Capture, Symmetric Kernels
Stat: Joint learning rate+fraction, row norm, fused rounding
Newton-Schulz is what makes Muon work, but it's also a massive pain compared to Adam. *Dion3* makes it ~6x faster and more parallelizable too.
Combines our work on row-filtered gradients, Gram-NS alg, and GPU kernels in one package. Try pip install dion!
arxiv.org/abs/2608.11612
Ali Naeimi retweeted
Newton-Schulz is what makes Muon work, but it's also a massive pain compared to Adam. *Dion3* makes it ~6x faster and more parallelizable too.
Combines our work on row-filtered gradients, Gram-NS alg, and GPU kernels in one package. Try pip install dion!
arxiv.org/abs/2608.11612





