
machine learning @neuralink. prev cto @tilderesearch & cs @stanford.
Joined December 2020
- Tweets368
- Following655
- Followers4K
- Likes1.6K
Pinned Tweet
i trained Llama 2 on a $5 computer with less processing power than a thermostat.
it’s called 🥧 PiTorch: an ML library for Raspberry Pi Zeros.
i cover how to make it from scratch, from writing assembly GPU kernels to sending bytes over wires. (no hardware background needed!)
Mason Wang retweeted
I joined neuralink with a background in aerospace engineering and firefighting. I didn’t know anything about surgery when I joined, but I was able to observe a couple of cases right away to learn the ropes.
I can promise you this picture is exactly how it feels like being in the operating room during a brain surgery as an engineer.
Sure we can design the tools from our desks and everything will work great in theory, but to make reliable hardware that will help millions of people, you NEED to close the loop. You NEED to be in the field.
This is true for engineers in any discipline. You can not make good parts if you are not seeing how the customer actually uses (and misuses!) them, what useless features you’ve left in, and what useful features you’ve left out.
The load cases that you didn’t think of in your FEA. The silliness of some of your “risk mitigations” and how unlikely they are to actually be performed. The random traditional way of doing things that users need to unlearn. You can’t know this without seeing it first hand.
If you are an engineer- please make sure you go to the field- and by field i mean literally every step of the process- Go to the machine shop and show them your drawings before you get the exorbitant quote. Go sit with the assembly technicians and hear them curse you out. Write your own verification and validation tests and check that they’re being run properly and know what margins you have and what assumptions they rely on. Make a few extra samples just so you can break them in creative ways and get an intuitive sense of how strong they are. Go to the training sessions for your users and see what questions they ask and what parts they struggle with.
Whatever it takes, get the job done right!
Mason Wang retweeted
Introducing Online KL Shampoo (OKLS), an optimizer that brings a KL-optimal approximation of full-matrix AdaGrad to language-model training.
Diagonal optimizers ignore correlations between gradient coordinates. Full-matrix AdaGrad captures this geometry but requires quadratic state. Muon considers correlations but not their history. OKLS closes this gap using KL-optimal Kronecker factors, whitening matrix gradients across both row and column directions while remaining naturally scale-invariant.
The main challenge is computing fresh inverse-square-root preconditioners at every step. Even one-step staleness can destabilize training. We make zero-staleness preconditioning practical with Scaled CANS Coupled Newton–Schulz: 10 iterations, 27 FP16 GEMMs, and FP32 accumulation.
OKLS achieves 1.45× the parameter efficiency of Muon while retaining 98% of its training throughput. Across 200M–1B models, an OKLS model matches a Muon model roughly 1.5× larger.
A challenge I never thought I'd have:
Most of our renders of the brain were too beautiful to be used, and would make people think it was fake
nitter.cf/alephneuro/status/2070…
LFG @levchizhov and team :)
We recently obtained the highest-resolution 3D images of the human brain ever taken from outside the skull. This is the first look.
Introducing Aleph, a research lab building brain interfaces for the telepathic future. (1/n)
4
23
2,332
incredible work from @SolidlySheafy and team!
their new optimizer treats attention's paired weight matrices as one object instead of tuning each in isolation for more stable training (the bangers from tilde continue)
Introducing Compositional Muon, an optimizer that extends Muon from individual matrices to composed transformer circuits.
Modern optimizers usually draw trust regions around individual parameters. But in attention, the loss often sees compositions like QK^T and OV. Updating each factor independently can therefore control the wrong object.
Compositional Muon closes this gap by deriving partner-whitened update rules. Each factor’s update is shaped by the spectral geometry of the matrix it is composed with, producing more stable composed updates and better effective learning-rate allocation across heads and layers.
For QK, this gives a head-local half-split rule. For OV, the circuit geometry selects a hybrid rule: (V) is optimized per-head, while (W_O) is optimized as the single matrix that aggregates all heads back into the residual stream.
CM improves over Muon at 340M and 1B scale, transfers to the modded-nanoGPT optimization benchmark, and can be approximated cheaply as partner-rescaled Muon via the isotropic rule.
The broader point is optimizer-architecture co-design: better optimizers should not only ask how to update a parameter, but what composed circuit that parameter participates in. CM is one step toward optimizers that respect the functional structure the loss actually sees.
5
34
2,897
if you're in SF or Palo Alto, come join @sambeskind and me for a Victory Club morning workout!
we've had 70 awesome people so far. most say it's the highlight of their week :)
all fitness levels welcome!
link below ⬇️
BANGER FROM @tilderesearch!!
softmax attn is a local constant fit, and the current efficient alternatives (linear attn and SSMs) basically just compress the whole past into a fixed-size state at the cost of recall
parallax adds a learned probe approximating a local linear fit:
i reproduced a model for predicting how ultrasound travels through the skull with 11x less parameters + open-sourced the training code and weights
new mini-post to share learnings and some visualizations :)
also ran 5 new loss-function finetunes to try to close the metric i couldn't match. one of them (E) ends up beating the paper on peak-pressure error
note: all my models below are 3.4M parameters compared to the paper's 38M.
(lower is better for all 3)
1
14
1,277
full credit of DeepTFUS architecture and dataset to Srivastav et al. (arXiv:2505.12998), and thanks for getting back to me with architecture details
read here (strong disclaimer this was just a quick 1.5 day learning sprint): masonjwang.com/projects/repr…






















