Introducing Underdog, your Private Personal AI on devices you already own Today we're announcing our backing from @a16z @khoslaventures @HummingbirdVC Anthology (@AnthropicAI @MenloVentures) @patrickc @naval @rauchg @polynoamial @Thom_Wolf @Mascobot @tszzl @OfficialLoganK and other top AI leaders Our mission is to provide free, capable, reliable and private AI to billions of people. Underdog’s Law: today’s frontier intelligence reaches your devices in six months. We’re starting with fast, capable AI that runs on consumer hardware. By co-designing models and inference engines, we’re pushing the frontier of capability, speed, power and data efficiency. Our research also spans agentic commerce, confidential inference, and how AI will reshape the internet economy. Privacy and capability no longer needs to be a tradeoff. If you believe in this future, join us. Time to build.

There have been some follow-up questions on the relationship between XMs and IMLE, so I’m going to provide some clarifications (some of this was in E.3 of the paper). IMLE is a great paper we respect and cite in XM, and I highly recommend people check it out. In fact, I’d encourage people to read the two papers side by side. I think this would help clarify the different motivations and let the merits of each speak for themselves. That being said, XMs are a generalization of IMLE and of best-of-K methods more broadly, and the recent claim that “XM is just a special case of IMLE” is inaccurate. XMs are about one thing, which is increasing generative expressivity by factoring the training loop, and IMLE is one specific instance of that (end-to-end Forward XMs). Because of this, >90% of the results in the XM paper go against the IMLE theory, and have not been studied by IMLE (including the entire 3rd pretraining axis portion of the paper, which was done by combining XMs with existing scalable reconstructive generative models, which IMLE has never focused on). Additionally, the XM paper's main contribution is empirical insights on generative expressivity, not best-of-K. In the paper, we explicitly state, “We do not claim to invent best-of-K”, and we also cite IMLE within the first two paragraphs of the approach section and in other locations. 🧵Thread:

🚨New Optimizer Paper AMUSE: Anytime MUon with Stable gradient Evaluation AMUSE combines Muon with Schedule-Free-style gradient evaluation for stable anytime training without LR decay. • Stronger 124M / 720M / 1B pretraining • Strong ImageNet / ViT fine-tuning performance.

Modded-NanoGPT Optimization Benchmark Hundreds of neural network optimizers have been proposed in the literature, recently including dozens citing Muon: MARS, SWAN, REG, ADANA, Newton-Muon, TrasMuon, AdaMuon, HTMuon, COSMOS, Conda, ASGO, SAGE, and Magma, to name a few. The majority of this innovation is happening in the public research community. But the community currently lacks a widely accepted, easily accessible way to compare and make sense of the deluge of methods. As a result, promising new ideas get buried, and spurious results go unchallenged. To help address these issues, I'm releasing a new optimization benchmark. It's designed for maximum simplicity and speed: Just a single file containing ~350 lines of plain PyTorch, which can complete a baseline LM training within 20 minutes of booting up a fresh 8xH100 machine. It also works with {1,2,4}xH100 or A100. These attributes make the new benchmark more accessible than any prior work. The rules are simple: The optimization algorithm can be changed arbitrarily, with the goal being to minimize the number of training steps needed to reach 3.28 val loss on FineWeb (this is the same target loss as in the main speedrun). Modifying the architecture or dataloader, on the other hand, is not allowed. Wallclock time is unlimited, in order to give a fair chance to optimizers which would need kernel work or larger scale to become wallclock-efficient. Like the main NanoGPT speedrun, submissions are open, and new results will be publicly broadcast. Beyond just improving the step count record, another goal of the benchmark is to collaboratively produce well-tuned baselines for as many optimizers as possible. For example, any improvement to the benchmark's best hyperparameters for AdamW would be considered a worthwhile new result. This benchmark is not intended to be the final measure of optimizer quality across all domains. Convenient shared experimental infrastructure which covers the full space of possibilities -- across varying batch size, tokens per parameter, model scale, epoch count, and architecture -- is desirable, but far beyond the current status quo. This benchmark is only meant to be one step towards that goal. To start the benchmark off, I've spent ~20 runs tuning baselines for Muon and AdamW. From time to time over the next few weeks, I'll add another optimizer from the literature, with my best effort at finding good hyperparameters. Researchers interested in neural network optimization are invited to join in by picking an optimizer and giving it a try on the benchmark. All optimizers are welcome, and even runs that don't necessarily have the best hyperparameters are desirable additions to the repo, because each new run adds to the collective knowledge.

HeavyBall 3.0.0 is finally out. Key features: * FSDP * DDP * End-to-End Compilation (2.5x speedup) * Higher-precision PSGDKron (grey, vs. HB2's blue) * Faster Muon and SOAP * PSGD-PRO (yellow) * LATHER, a SOAP-like optimizer * HyperBall * explicit `consume_grad` * simplified API