My 2nd modded-nanogpt world record attempt: 30.67 seconds? - amund.blog/modded-nanogpt-se…
1
133
My first modded-nanogpt attempt: 34.9 s vs 40.2 s for the current record’s code on the same 8×H100 node. It mostly combines two open PRs by others (#379 + #375), plus a small timing fix. In review, not a record yet.
amund.blog/modded-nanogpt-re…
350
Exp minus Log is all you need for Deep Learning - amund.blog/exp-minus-log-is-…
318
4 bits is all you need (and 3.6 bit you have?) for resource-efficient LLMs?
A few months ago OpenAI published their open weights model(s) GPT-OSS (20B and 120B), and one of the eye-catching characteristics was that it was heavily quantized - in other words “shrunk” or compressed - to 4 bits per parameter (MXFP4) instead of the commonly used 16 bit (BF16).
4 bit (“half byte”) means that the model uses much less memory, energy and also can run efficiently on hardware natively supporting 4 bit MXFP inference (e.g. Nvidia Blackwell B200/300 and AMD MI355X). With 4 bit compared to 16 bit there is some quality loss, but typically marginally.
But can we go even lower and represent the model in fewer bits per parameter, e.g. 3 bit, 2 bit or even 1 bit? There is highly interesting and seemingly promising research on more efficient quantization (few bit representation) of deep neural networks, however perhaps 4 bit is very close to the lower plateau - based on:
a) findings in an article with large empirical studies by Meta, Google DeepMind, Cornell University and Nvidia that language models have a capacity of 3.6 bits per parameter - Language Models Don’t Store Facts: They Compress Patterns - so perhaps 3.6 bit is a lower bound per parameter for efficient representation of deep neural networks?
b) and to a lesser degree personal testing on doing further quantizations on GPT-OSS 20B, it seems like when going to 2 and 3 bit even with (LoRA-based) finetuning it is hard to lift it back to close to 4 bit.
Amund
atsentia.com
604
Amund Tveit retweeted
The *full* Python Documentary will be released this Thursday (Aug 28) at 10am PDT / 19:00 CET. More at discuss.python.org/t/python-… Don't miss the online release party / chat! @TECHDOCU
Exciting times in AI (Agent) devtools and vibe coding — and for the historically inclined, you might enjoy a 2008 throwback: my blog post Increase the automation of Test-Driven Development?amundblog.blogspot.com/2008/…
2
8
1,971
Amund Tveit retweeted
A new opening for multimodal model research: jobs.careers.microsoft.com/g… . Please apply if interested.
Published a paper: Muon Optimizer Accelerates Grokking - arxiv.org/abs/2504.16041 - with colleagues Bjørn Remseth (@rmz) and Arve Skogvold
3
1,073
Grokking transformer implementations in Pytorch and Jax/Flax - amund.blog/pytorch_jax_grokk…
1
308
Amund Tveit retweeted
A real, practical device technology that provides stable, logical, qubits. We are entering the quantum era. This is not hype.
A breakthrough in quantum computing. Majorana 1 brings us closer to harnessing millions of potential qubits working together to solve the unsolvable—from new medicines to revolutionary materials—all on a single chip. #QuantumComputing #QuantumReady
msft.it/6007UxEHz
Amund Tveit retweeted
🪩The @stateofai 2024 has landed! 🪩
Our seventh installment is our biggest and most comprehensive yet, covering everything you *need* to know about research, industry, safety and politics.
As ever, here's my director’s cut (+ video tutorial!) 🧵
Amund Tveit retweeted
A stable qubit needing only 30 physical qubits? @Microsoft and @QuantinuumQC achieve an 800x improvement in logical vs physical error rate, with both entanglement and error corrections. A real landmark in quantum computing. cloudblogs.microsoft.com/qua…
Amund Tveit retweeted
I will be talking about our work on building a digital twin for the Oslo Fjord at this popular science event in Oslo, February 8. Come by for a chat? mn.uio.no/ibv/english/resear…
Amund Tveit retweeted
Starting the year with a small update, phi-2 is now under MIT license, enjoy everyone!
huggingface.co/microsoft/phi…













