Jonathan Geuter (@JonathanGeuter) profile picture

Applied Math PhD @ Harvard & Kempner Institute

Boston, MA, USA
Joined March 2024
Photo by Jonathan Geuter (@JonathanGeuter)Photo by Jonathan Geuter (@JonathanGeuter)Photo by Jonathan Geuter (@JonathanGeuter)
๐—๐—ฒ๐—ณ๐—ณ ๐Ÿญ.๐Ÿญ ๐—ถ๐˜€ ๐—ผ๐˜‚๐˜! ๐Ÿš€ Our open-source Jeff decision models just got a lot better. Jeff-1.1-Large on OpenDecisions: 89.0 in one pass 92.3 with optional reasoning (Jeff-1.0-Large: 84.9, Jev: 80.1) What changed? ๐Ÿงต๐Ÿ‘‡
1
6
1
27
1,746
๐Ÿš€ Our technical report for our K2 horizon models is out! It is the most detailed technical report of any open-source models that has ever been released. If you've ever wanted to see all the details that go into LLM training across all training phases, here it is!
Looking for a reference manual for building a frontier LLM? We wrote one ๐Ÿ‘‡ The K2 Horizon technical report is now on arXiv: 134 pages on how we built six models, from architecture to inference. It may be the most detailed LLM training reference you can read today. โ€ข Architecture: we bring two new attention designs, MoVA and softplus attention gating, to strengthen reasoning. With them, our 36B model nearly matches a 32B dense one on one-seventh of the active parameters. โ€ข Pre-training: we trained four models, from 3.7B to 36B, on the same 22T tokens. Their normalized loss curves collapse onto nearly the same line, so a recipe that works on a small model carries over when scaling up. โ€ข Data: 60+ pages on the part most reports skip. How we generated ~10T synthetic tokens, and why we chose the data mix on predicted scores at scale, which reversed the small-run ranking. โ€ข Post-training: we document our full production-scale run for the 375B flagship. โ€ข Inference: Uno gives an autoregressive model diffusion speed with one small adapter. Up to 2.2ร— faster decoding, lossless by construction, no draft model. This is the last piece of the openness puzzle. K2 Horizon already shipped with 380 checkpoints, 13 TB of pre-training data, 58 training runs and 17 code repositories. The report explains the decisions behind them. More details: arxiv.org/abs/2611.21003
1
2
7
382
โšกNEW: Unlocking lossless speedups in LLMs via discrete diffusion. A new diffusion-augmented model, Uno, combines autoregressive and diffusion weights in one network to generate multiple tokens per forward pass. More in todayโ€™s Deeper Learning blog post: bit.ly/4AmooR0
1
4
11
663
Jonathan Geuter retweeted
Great blog post by @JonathanGeuter and @KempnerInst on our recent Uno paper. Autoregressive LLMs + Diffusion adapters >>> Training Diffusion LLMs from scratch or Speculative Decoding with small drafters Stay tuned for a follow-up to the Uno paper with comparisons against DFlash2 and DSpark. kempnerinstitute.harvard.eduโ€ฆ
Diffusion LLMs have two limitations relative to AR models: (1) Lower quality, and (2) Slower inference at large batch sizes. We address this "Uno" > Retains the AR architecture of LLMs > Each layer has two sets of weights: AR weights and Diffusion weights > Diffusion weights enable parallel sampling from the AR distribution losslessly Results: ๐Ÿ’ฅFaster than all speculative decoding methods: DFlash and EAGLE-3 ๐Ÿ”ฅ Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada Links to the paper, models, code below
11
59
2,516
Great work by Daniel, happy to see people use our decision model training dataset! Check it out here: huggingface.co/datasets/jgeuโ€ฆ
Some things that were useful when my agent and I were building the data for this model: - Check how well you can guess the answer from the options alone. One source looked like a +4.5 gain, but it was mostly "the longest option is right". Real gain was more like +2.3 - Weight wrong options by how common they are. arXiv labels with uniform wrong options got +18 over chance without reading any text - Keep one writing style across classes. With one class written by humans and the other by an LLM, the model just learned to spot LLM style - Shuffle the options per example. Models that read all the options together can flip answers when you reorder them (12-23% in Crivo) - Try different mixes, but with more than one seed. Single runs differed by up to ~8 points on one test set. Adding diverse sources gave the same lift as 4x the rows and steps, at about a quarter of the compute - Check what your trainer silently drops. Ours skipped 53k one-label rows, the only verification data in the mix Data, with source, labeller and licence on every row: huggingface.co/datasets/davaโ€ฆ
4
217
๐Ÿ“„ Paper: arxiv.org/abs/2610.17205 (will be updated to v1.1 soon!) ๐Ÿค— Models: huggingface.co/collections/jโ€ฆ ๐Ÿ“Š Data: huggingface.co/datasets/jgeuโ€ฆ โš™๏ธ Code: github.com/j-geuter/Jeff Joint work with Andrew Cheng and @elmelis.
1
95
@vanstriendaniel might be of interest to you! We've also released our training data now.
2
98
Jonathan Geuter retweeted
Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch. No diffusion teacher. No flow map. No CFG. No JVP. We believe this work will challenge how we think about building one-step models. Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han. ๐Ÿ”ฅ 1.19 FID one-step on ImageNet 256ร—256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L ๐ŸŽจ Scales to text-to-image generation (GenEval 0.82) ๐Ÿค– World modeling on RT-1 / Bridge-V2 Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].
38
132
27
1,374
118,221
Everyone is talking about 'System One models' like Jev: given a state, question, and set of options, they return a calibrated decision at a fraction of the cost of an LLM call. But do we need specialized architectures and recipes to train them? We show you donโ€™t: ๐Ÿงต๐Ÿ‘‡
2
2
1
18
2,998
Our flagship model, Jeff-1.0-Large, scores 84.4 on OpenDecisions, a new state-of-the-art, and 81.5 on JevBench. That ๐—ฏ๐—ฒ๐—ฎ๐˜๐˜€ ๐—๐—ฒ๐˜ƒ ๐—ฏ๐˜† ๐Ÿฐ.๐Ÿฏ ๐—ฎ๐—ป๐—ฑ ๐Ÿฐ.๐Ÿฐ ๐—ฝ๐—ผ๐—ถ๐—ป๐˜๐˜€ and puts it at #๐Ÿฎ ๐—ผ๐—ป ๐˜๐—ต๐—ฒ ๐—๐—ฒ๐˜ƒ๐—•๐—ฒ๐—ป๐—ฐ๐—ต ๐—น๐—ฒ๐—ฎ๐—ฑ๐—ฒ๐—ฟ๐—ฏ๐—ผ๐—ฎ๐—ฟ๐—ฑ (almost tied for first).
1
2
149
๐Ÿ“„ ๐—ฃ๐—ฎ๐—ฝ๐—ฒ๐—ฟ: arxiv.org/abs/2610.17205 ๐Ÿค— ๐— ๐—ผ๐—ฑ๐—ฒ๐—น๐˜€: huggingface.co/collections/jโ€ฆ ๐Ÿ“Š ๐——๐—ฎ๐˜๐—ฎ๐˜€๐—ฒ๐˜: huggingface.co/datasets/jgeuโ€ฆ โš™๏ธ ๐—–๐—ผ๐—ฑ๐—ฒ: github.com/j-geuter/jeff-serโ€ฆ Joint work with Andrew Cheng and @elmelis
1
1
138