Jack Cook (@jackcookjack) header
Jack Cook (@jackcookjack) profile picture

mts @thinkymachines | ex-phd student @mit | systems for ml

San Francisco, CA
Joined January 2011
Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)Photo by Jack Cook (@jackcookjack)
In TML we care a lot about low precision training and high quality bufos :bufo-nvfp4:
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
2
8
1
112
4,480
Jack Cook retweeted
Today on Connectionism: how we trained Inkling 2 Small with a native 4-bit forward pass on @nvidia GB300s, from the first pretraining step through the final RL rollout, one of the many improvements across the stack that took pretraining from weeks to 68 hours.
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
53
102
7
1,398
74,757
Jack Cook retweeted
Congrats @thinkymachines team on Inkling 2 Small! A 594B model pretrained in just 68 hours with NVIDIA NVFP4 on GB300 NVL72 racks. Read how the team made low-precision training work at this scale. 👇
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
227
112
17
1,994
163,491
Jack Cook retweeted
A fantastic read on low-precision training by the amazing @jackcookjack and others at thinky!
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
3
15
883
Jack Cook retweeted
ML researcher: ...then we must de-quantize the tensors. Me: That's not a real thing, that's some gibberish the villain says in a cheap sci-fi movie. ML researcher: ... Claude: That just means storing each number with more memory bits. Me: Oh okay, why didn't you say so.
4
4
1
214
15,251
Replying to @jackcookjack
@jackcookjack cook cooks
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
2
2
52
1,755
we wrote a detailed post on our NVFP4 training, worth a read!
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
15
38
1
828
48,069
Jack Cook retweeted
Whenever I see the bigram Jack + NVFP4, I know it has to be a cool work. Congrats on the release!
Inkling 2 Small is our first model trained partly in NVFP4, with a mixed-precision recipe that enables a great tradeoff between speed and accuracy. Check out our new post to see how we did it, and the lessons we learned along the way!
1
2
25
1,570
Jack Cook retweeted
Replying to @jackcookjack
@jackcookjack is a low precision legend. Give this a read to for a sense of how we do nvfp4 pretraining and RL at thinking machines!
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
1
21
731
the NVFP4 recipe behind Inkling 2 Small: 4-bit forward from the first pretraining step to the final RL rollout, MXFP8 + stochastic rounding for backward and still more to come 🪜
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
3
8
107
5,593
Jack Cook retweeted
When I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs. As it turns out, he was wrong :^) This blog post explains how it can be done
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
8
13
4
301
25,431
Inkling 2 Small’s weights are now available for download on HuggingFace under an Apache 2.0 license. Make Inkling your own! huggingface.co/thinkingmachi…
124
340
189
2,454
475,323
Our v2 small model is here. I've been using it daily for a variety of things. The pre-training ran for 68 hours on GB300s. Post-training was 17 million rollouts over 9 days. Entirely pre-mid-post trained in NVFP4. Available to train and fine-tune on Tinker right now. Open weights next week after external safety testing is fully complete (cant skimp on that). Continuing to decentralize and democratize intelligence...
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor. Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon. thinkingmachines.ai/news/ink…
74
61
17
1,328
90,579
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor. Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon. thinkingmachines.ai/news/ink…
294
585
501
6,303
1,986,613
Today, we are introducing Inkling. Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available. thinkingmachines.ai/news/int… Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
558
1,978
1,189
14,805
8,375,693
Jack Cook retweeted
Today we share the worldview behind our mission. Human values don't average out. Local knowledge can't be centralized. The good future has many AIs, raised in different places, shaped by the people they serve, disagreeing with each other the way we do. thinkingmachines.ai/blog/the…
147
523
205
4,462
1,232,426
Low-precision floats are weird. I have been building up my intuition by playing with them outside of inference/training. @AAAzzam and I cooked up this visualizer for micro-scaling/block quant formats like NVFP4, MXFP4, and friends. Try it: modal.com/llm-almanac/block-…
6
7
131
13,157
Jack Cook retweeted
4/6 --> 6/7... jokes aside jack put this out crazy fast, and it's a very clever idea that I hope it gets standardized in new hardware :)
NVFP4 allows models to be quantized to 4 bits without too much performance degradation, but can we push 4-bit performance even further? Today, we're releasing a new class of low-precision block-scaled data types that natively adapt to your input data: for 4-bit quantization, IF4 (Int/Float 4) allows each scaled group of 16 values to be saved as FP4 or INT4 depending on which option offers less error. Selections are recorded using the scale factor’s sign bit, which is unused in NVFP4, allowing IF4 to offer better performance with no memory overhead! Our data types provide better downstream accuracy in LLMs, they can be implemented efficiently in next-generation hardware accelerators, and they reveal some interesting insights about low-bit quantization! 🧵
1
7
76
11,597
Very cool paper! Glad to see our rotation-based unbiased gradient estimation scheme from Quartet II improve quality as well.
Replying to @jackcookjack
With these two features in place, we find that IF data types (IF3, IF4, and IF6) offer better performance over existing block-scaled data types during quantized training and inference. During training, we show that this can be partly explained by an increased preference for integer quantization after inputs undergo a Hadamard transformation, an important component of backward passes in recently-published 4-bit training recipes. During inference, performance improvements can be primarily explained by the reduced quantization error.
1
4
1,066
doot doot
NVFP4 allows models to be quantized to 4 bits without too much performance degradation, but can we push 4-bit performance even further? Today, we're releasing a new class of low-precision block-scaled data types that natively adapt to your input data: for 4-bit quantization, IF4 (Int/Float 4) allows each scaled group of 16 values to be saved as FP4 or INT4 depending on which option offers less error. Selections are recorded using the scale factor’s sign bit, which is unused in NVFP4, allowing IF4 to offer better performance with no memory overhead! Our data types provide better downstream accuracy in LLMs, they can be implemented efficiently in next-generation hardware accelerators, and they reveal some interesting insights about low-bit quantization! 🧵
3
38
4,444