Ben (@SolidlySheafy) profile picture

Learning

Joined March 2024
Over 15,000 submissions to One Layer Deeper. Nobody solved the problem as intended, but there are some interesting ideas for learning reusable operations and composing them into deeper computations coreauto.com/blog/what-the-m…
12
25
5
189
36,398
In deep learning, deeper is almost always better. If it isn’t, that means you’re just not very good at optimization.
This quoted post is unavailable.
1
6
2
98
14,552
This party was very full, we couldn’t invite everyone who applied because we didn’t have the space host everyone. My sincere apologies- I also missed the house warming core party due to life reasons. Now back to recap of Core Party, we finally beat the allegations that we are a car company. First, We have our @MillionInt introducing the lab and what we do and why now. James Demmel gave us a beautiful talk. He had said that day he compressed a semester course into a 20min talk. We were quite lucky to having him and huge thanks! His talk is quite relevant to building reliable automation particularly kernels generations for next generation hardware. He talks about hard to beat grader for numerics of gemms which is his recent work. The legendary @marksaroufim talked about how we got community to accelerate the Nvidia B200 GPU kernels: QR, Eigh, Chol. I am very glad he was a gentleman and didn’t troll me when I wasn’t there. This was a byproduct of the great Muon vs Shampoo non-feud on X with @kellerjordan0 TPUs had better kernels for these because Google was the home for higher order optimization (particularly KFAC and Shampoo) now you can get end to end wall clock time improvements on latest GPUs incorporating these advanced kernels into training pipeline. It can be thought as a great unhobbling for this line of research. Great work by the GPUMode community members. Legit. We also have @SolidlySheafy who we have found as a kindred spirit in pushing the frontiers, so we partnered with his team in building out an algorithmic competition as well as Nvidia’s Christoph Klein (@nvidia) give an overview of Dense Linear Algebra If you read it so far, should we do a core party again? What would you like to see there?
We had the @coreauto housewarming party a couple of weeks back. It was tons of fun and should hopefully give you a better idea of where we're spending some of our time improving model architectures and systems 00:00: Introduction 00:40: @MillionInt (@coreauto), Building the world’s most automated AI lab 12:10: James Demmel (@UCBerkeley), Dense Numerical Linear Algebra: A brief history and current trends 42:11: Christoph Klein (@nvidia), Dense Linear Algebra at NVIDIA 47:54: @marksaroufim (@coreauto), Linear Algebra Kernels for the Age of Research 51:43: @SolidlySheafy (@tilderesearch), One Layer Deeper…
5
2
114
17,433
KL Shampoo has a very much something I love. We used to call it LogDet divergence derivation for Shampoo, instead of frobenius. It falls out pretty directly. A good title for this work: Shampoo Strikes Back: The KL Awakens Every Step This work is cool, Tilde improved the coupled newton method we introduced to work at low precision, and run fast so you can run that every step. Also the additional improvements around MuP. I am nerdsniped to work out the fwd and bwd error of the technique they use introduced to see if something provable can be said about them.
Introducing Online KL Shampoo (OKLS), an optimizer that brings a KL-optimal approximation of full-matrix AdaGrad to language-model training. Diagonal optimizers ignore correlations between gradient coordinates. Full-matrix AdaGrad captures this geometry but requires quadratic state. Muon considers correlations but not their history. OKLS closes this gap using KL-optimal Kronecker factors, whitening matrix gradients across both row and column directions while remaining naturally scale-invariant. The main challenge is computing fresh inverse-square-root preconditioners at every step. Even one-step staleness can destabilize training. We make zero-staleness preconditioning practical with Scaled CANS Coupled Newton–Schulz: 10 iterations, 27 FP16 GEMMs, and FP32 accumulation. OKLS achieves 1.45× the parameter efficiency of Muon while retaining 98% of its training throughput. Across 200M–1B models, an OKLS model matches a Muon model roughly 1.5× larger.
2
8
149
14,553
Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable? @coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper" We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases. This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way. The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation. Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
14
57
10
493
78,569
We wanted to create a competition for arch x opt co-design! The problem is *very hard* Incredibly excited to collaborate on this with the amazing folks at @coreauto Give it a try :)
onelayerdeeper.ai/ There were (and still are) kernel competitions. Now we want to present something new: optimizer competition. Show your skills, and learn along the way optimizing through hard problems.
1
8
2
93
12,579
onelayerdeeper.ai/ There were (and still are) kernel competitions. Now we want to present something new: optimizer competition. Show your skills, and learn along the way optimizing through hard problems.
3
12
3
172
43,092
Find ways to beat this benchmark that tries to understand the limits of deep learning architectures and optimization? We are funding compute, there is a nice way to test it out. Its a single GPU problem. Cracked team @tilderesearch x @coreauto collaboration.
onelayerdeeper.ai/ There were (and still are) kernel competitions. Now we want to present something new: optimizer competition. Show your skills, and learn along the way optimizing through hard problems.
7
10
1
112
15,440
~1/7~ Today we are releasing Aurora on Arxiv → a leverage-aware spectral optimizer built for the MLP matrices in language models. Aurora achieves strong pre-training efficiency gains, achieves state-of-the-art performance on the optimizer track of modded nanoGPT speedrun among spectral optimizers, and allows for the effective training of very wide MLPs. 🧵
3
16
4
130
15,500
The future of using your phone is not tapping glass. We're launching Willow iOS 2.0 to make your voice actually work. Including Scribe on iOS, faster speed, better accuracy, and a smoother experience across the places you already write. Now on App Store.
26
5
7
73
20,103
Modded-NanoGPT optimization result #17 (2026/05/06): @Li_Yang_2019, one of the authors of Aurora, submitted an Aurora-based run to the benchmark. Its performance (3175 steps) reached halfway between SOAP-Muon (3125 steps) and Contra-NorMuon (3225 steps).
2
6
77
5,143
When Amy was diagnosed with a brain tumor, I did what anyone would do: I trusted the system. Two surgeries later, the tumor was still there. So I started using AI to research her condition full time. Within my first week, I found a paper that leading pituitary scientists told me they'd never seen. I wrote about what happened next for @TheFP. Grateful to their team for helping me tell this story! thefp.com/p/i-used-ai-to-fig…
14
38
14
301
55,554
Reminder that we're always hiring for engineers and researchers! Work includes building infra for large-scale distributed training and research in model architecture and optimizers. If you're interested in working on a foundational understanding of models, dm us or apply using the link in the comments.
8
8
1
112
45,089
My girlfriend has a brain tumor. I’m going to cure it myself. 🧵
370
255
77
7,476
1,121,933
What if your benchmark scores are lying to you? Today, I'm excited to share @Microsoft's DevBench, the first telemetry-grounded code generation benchmark, covering six languages, and the first to combine synthetic generation with manual expert review for contamination resistance. 📄 arxiv.org/abs/2601.11895 💻 github.com/microsoft/devbenc… 1/7
2
10
1
27
3,907
new brand for Tilde Research inspired by transformer architecture — nodes and connections that map how intelligence flows
11
3
1
53
8,701
AxiomProver solved Erdos problem #481 - took 5 hours #124 simplified version took over 24 hours (oof) and was not as succinct as we'd like it to be
21
27
5
266
23,842
🎉Excited to announce today that @erikqu_ and I have been accepted into the X25 @ycombinator batch. Erik and I met in 2014 in high school doing coding competitions together. >10 years later, we're building @operative_sh - the world's first web app coding & debugging agent. 🧵
21
11
2
110
9,364