
mts @thinkymachines | ex-phd student @mit | systems for ml
San Francisco, CA
Joined January 2011
- Tweets168
- Following555
- Followers2.2K
- Likes26.2K
Jack Cook retweeted
In TML we care a lot about low precision training and high quality bufos :bufo-nvfp4:
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Jack Cook retweeted
Today on Connectionism: how we trained Inkling 2 Small with a native 4-bit forward pass on @nvidia GB300s, from the first pretraining step through the final RL rollout, one of the many improvements across the stack that took pretraining from weeks to 68 hours.
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Congrats @thinkymachines team on Inkling 2 Small!
A 594B model pretrained in just 68 hours with NVIDIA NVFP4 on GB300 NVL72 racks.
Read how the team made low-precision training work at this scale. 👇
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Jack Cook retweeted
A fantastic read on low-precision training by the amazing @jackcookjack and others at thinky!
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
ML researcher: ...then we must de-quantize the tensors.
Me: That's not a real thing, that's some gibberish the villain says in a cheap sci-fi movie.
ML researcher: ...
Claude: That just means storing each number with more memory bits.
Me: Oh okay, why didn't you say so.
Jack Cook retweeted
Replying to @jackcookjack
@jackcookjack cook cooks
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Jack Cook retweeted
we wrote a detailed post on our NVFP4 training, worth a read!
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Whenever I see the bigram Jack + NVFP4, I know it has to be a cool work. Congrats on the release!
Jack Cook retweeted
Replying to @jackcookjack
@jackcookjack is a low precision legend. Give this a read to for a sense of how we do nvfp4 pretraining and RL at thinking machines!
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Jack Cook retweeted
the NVFP4 recipe behind Inkling 2 Small: 4-bit forward from the first pretraining step to the final RL rollout, MXFP8 + stochastic rounding for backward
and still more to come 🪜
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
When I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs.
As it turns out, he was wrong :^) This blog post explains how it can be done
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Jack Cook retweeted
Inkling 2 Small’s weights are now available for download on HuggingFace under an Apache 2.0 license.
Make Inkling your own!
huggingface.co/thinkingmachi…
Jack Cook retweeted
Our v2 small model is here.
I've been using it daily for a variety of things.
The pre-training ran for 68 hours on GB300s. Post-training was 17 million rollouts over 9 days.
Entirely pre-mid-post trained in NVFP4.
Available to train and fine-tune on Tinker right now.
Open weights next week after external safety testing is fully complete (cant skimp on that).
Continuing to decentralize and democratize intelligence...
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.
Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.
thinkingmachines.ai/news/ink…
Jack Cook retweeted
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.
Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.
thinkingmachines.ai/news/ink…
Jack Cook retweeted
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/int…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
Jack Cook retweeted
Today we share the worldview behind our mission.
Human values don't average out. Local knowledge can't be centralized. The good future has many AIs, raised in different places, shaped by the people they serve, disagreeing with each other the way we do.
thinkingmachines.ai/blog/the…
Jack Cook retweeted
Low-precision floats are weird. I have been building up my intuition by playing with them outside of inference/training.
@AAAzzam and I cooked up this visualizer for micro-scaling/block quant formats like NVFP4, MXFP4, and friends.
Try it: modal.com/llm-almanac/block-…
Jack Cook retweeted
4/6 --> 6/7...
jokes aside jack put this out crazy fast, and it's a very clever idea that I hope it gets standardized in new hardware :)
NVFP4 allows models to be quantized to 4 bits without too much performance degradation, but can we push 4-bit performance even further?
Today, we're releasing a new class of low-precision block-scaled data types that natively adapt to your input data: for 4-bit quantization, IF4 (Int/Float 4) allows each scaled group of 16 values to be saved as FP4 or INT4 depending on which option offers less error. Selections are recorded using the scale factor’s sign bit, which is unused in NVFP4, allowing IF4 to offer better performance with no memory overhead!
Our data types provide better downstream accuracy in LLMs, they can be implemented efficiently in next-generation hardware accelerators, and they reveal some interesting insights about low-bit quantization! 🧵
Jack Cook retweeted
Very cool paper! Glad to see our rotation-based unbiased gradient estimation scheme from Quartet II improve quality as well.
Replying to @jackcookjack
With these two features in place, we find that IF data types (IF3, IF4, and IF6) offer better performance over existing block-scaled data types during quantized training and inference.
During training, we show that this can be partly explained by an increased preference for integer quantization after inputs undergo a Hadamard transformation, an important component of backward passes in recently-published 4-bit training recipes. During inference, performance improvements can be primarily explained by the reduced quantization error.
Jack Cook retweeted
doot doot
NVFP4 allows models to be quantized to 4 bits without too much performance degradation, but can we push 4-bit performance even further?
Today, we're releasing a new class of low-precision block-scaled data types that natively adapt to your input data: for 4-bit quantization, IF4 (Int/Float 4) allows each scaled group of 16 values to be saved as FP4 or INT4 depending on which option offers less error. Selections are recorded using the scale factor’s sign bit, which is unused in NVFP4, allowing IF4 to offer better performance with no memory overhead!
Our data types provide better downstream accuracy in LLMs, they can be implemented efficiently in next-generation hardware accelerators, and they reveal some interesting insights about low-bit quantization! 🧵





















