Amund Tveit (@atveit) header
Amund Tveit (@atveit) profile picture

Principal AI Engineer Tieto | Past: Microsoft, Google, YouTube | Personal Opinions Only

Trondheim, Norway
Joined May 2007
Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)Photo by Amund Tveit (@atveit)
Early winter
1
44
My first modded-nanogpt attempt: 34.9 s vs 40.2 s for the current record’s code on the same 8×H100 node. It mostly combines two open PRs by others (#379 + #375), plus a small timing fix. In review, not a record yet. amund.blog/modded-nanogpt-re…
350
Exp minus Log is all you need for Deep Learning - amund.blog/exp-minus-log-is-…
318
4 bits is all you need (and 3.6 bit you have?) for resource-efficient LLMs? A few months ago OpenAI published their open weights model(s) GPT-OSS (20B and 120B), and one of the eye-catching characteristics was that it was heavily quantized - in other words “shrunk” or compressed - to 4 bits per parameter (MXFP4) instead of the commonly used 16 bit (BF16). 4 bit (“half byte”) means that the model uses much less memory, energy and also can run efficiently on hardware natively supporting 4 bit MXFP inference (e.g. Nvidia Blackwell B200/300 and AMD MI355X). With 4 bit compared to 16 bit there is some quality loss, but typically marginally. But can we go even lower and represent the model in fewer bits per parameter, e.g. 3 bit, 2 bit or even 1 bit? There is highly interesting and seemingly promising research on more efficient quantization (few bit representation) of deep neural networks, however perhaps 4 bit is very close to the lower plateau - based on: a) findings in an article with large empirical studies by Meta, Google DeepMind, Cornell University and Nvidia that language models have a capacity of 3.6 bits per parameter - Language Models Don’t Store Facts: They Compress Patterns - so perhaps 3.6 bit is a lower bound per parameter for efficient representation of deep neural networks? b) and to a lesser degree personal testing on doing further quantizations on GPT-OSS 20B, it seems like when going to 2 and 3 bit even with (LoRA-based) finetuning it is hard to lift it back to close to 4 bit. Amund atsentia.com
604
AI on Apple Watch
1
292
Exciting times in AI (Agent) devtools and vibe coding — and for the historically inclined, you might enjoy a 2008 throwback: my blog post Increase the automation of Test-Driven Development?amundblog.blogspot.com/2008/…
2
8
1,971
A new opening for multimodal model research: jobs.careers.microsoft.com/g… . Please apply if interested.
3
10
4
59
13,120
Published a paper: Muon Optimizer Accelerates Grokking - arxiv.org/abs/2504.16041 - with colleagues Bjørn Remseth (@rmz) and Arve Skogvold
3
1,073
Amund Tveit retweeted
A real, practical device technology that provides stable, logical, qubits. We are entering the quantum era. This is not hype.
A breakthrough in quantum computing. Majorana 1 brings us closer to harnessing millions of potential qubits working together to solve the unsolvable—from new medicines to revolutionary materials—all on a single chip. #QuantumComputing #QuantumReady msft.it/6007UxEHz
36
144
9
1,387
73,114
🪩The @stateofai 2024 has landed! 🪩 Our seventh installment is our biggest and most comprehensive yet, covering everything you *need* to know about research, industry, safety and politics. As ever, here's my director’s cut (+ video tutorial!) 🧵
31
301
104
1,127
443,888
Amund Tveit retweeted
GitHub Copilot in VS Code + OpenAI o1 is flat out bad ass.
27
122
13
946
287,062
Amund Tveit retweeted
A stable qubit needing only 30 physical qubits? @Microsoft and @QuantinuumQC achieve an 800x improvement in logical vs physical error rate, with both entanglement and error corrections. A real landmark in quantum computing. cloudblogs.microsoft.com/qua…
1
29
2
80
6,964
Starting the year with a small update, phi-2 is now under MIT license, enjoy everyone! huggingface.co/microsoft/phi…
47
257
56
1,845
286,767