Pinned Tweet
๐๐ฒ๐ณ๐ณ ๐ญ.๐ญ ๐ถ๐ ๐ผ๐๐! ๐
Our open-source Jeff decision models just got a lot better.
Jeff-1.1-Large on OpenDecisions:
89.0 in one pass
92.3 with optional reasoning
(Jeff-1.0-Large: 84.9, Jev: 80.1)
What changed?
๐งต๐
๐ Our technical report for our K2 horizon models is out! It is the most detailed technical report of any open-source models that has ever been released. If you've ever wanted to see all the details that go into LLM training across all training phases, here it is!
Looking for a reference manual for building a frontier LLM? We wrote one ๐
The K2 Horizon technical report is now on arXiv: 134 pages on how we built six models, from architecture to inference. It may be the most detailed LLM training reference you can read today.
โข Architecture: we bring two new attention designs, MoVA and softplus attention gating, to strengthen reasoning. With them, our 36B model nearly matches a 32B dense one on one-seventh of the active parameters.
โข Pre-training: we trained four models, from 3.7B to 36B, on the same 22T tokens. Their normalized loss curves collapse onto nearly the same line, so a recipe that works on a small model carries over when scaling up.
โข Data: 60+ pages on the part most reports skip. How we generated ~10T synthetic tokens, and why we chose the data mix on predicted scores at scale, which reversed the small-run ranking.
โข Post-training: we document our full production-scale run for the 375B flagship.
โข Inference: Uno gives an autoregressive model diffusion speed with one small adapter. Up to 2.2ร faster decoding, lossless by construction, no draft model.
This is the last piece of the openness puzzle. K2 Horizon already shipped with 380 checkpoints, 13 TB of pre-training data, 58 training runs and 17 code repositories. The report explains the decisions behind them.
More details: arxiv.org/abs/2611.21003
Jonathan Geuter retweeted
โกNEW: Unlocking lossless speedups in LLMs via discrete diffusion. A new diffusion-augmented model, Uno, combines autoregressive and diffusion weights in one network to generate multiple tokens per forward pass. More in todayโs Deeper Learning blog post: bit.ly/4AmooR0
Jonathan Geuter retweeted
Great blog post by @JonathanGeuter and @KempnerInst on our recent Uno paper.
Autoregressive LLMs + Diffusion adapters >>> Training Diffusion LLMs from scratch or Speculative Decoding with small drafters
Stay tuned for a follow-up to the Uno paper with comparisons against DFlash2 and DSpark.
kempnerinstitute.harvard.eduโฆ
Diffusion LLMs have two limitations relative to AR models:
(1) Lower quality, and
(2) Slower inference at large batch sizes.
We address this "Uno"
> Retains the AR architecture of LLMs
> Each layer has two sets of weights: AR weights and Diffusion weights
> Diffusion weights enable parallel sampling from the AR distribution losslessly
Results:
๐ฅFaster than all speculative decoding methods: DFlash and EAGLE-3
๐ฅ Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada
Links to the paper, models, code below
Great work by Daniel, happy to see people use our decision model training dataset! Check it out here:
huggingface.co/datasets/jgeuโฆ
Some things that were useful when my agent and I were building the data for this model:
- Check how well you can guess the answer from the options alone. One source looked like a +4.5 gain, but it was mostly "the longest option is right". Real gain was more like +2.3
- Weight wrong options by how common they are. arXiv labels with uniform wrong options got +18 over chance without reading any text
- Keep one writing style across classes. With one class written by humans and the other by an LLM, the model just learned to spot LLM style
- Shuffle the options per example. Models that read all the options together can flip answers when you reorder them (12-23% in Crivo)
- Try different mixes, but with more than one seed. Single runs differed by up to ~8 points on one test set. Adding diverse sources gave the same lift as 4x the rows and steps, at about a quarter of the compute
- Check what your trainer silently drops. Ours skipped 53k one-label rows, the only verification data in the mix
Data, with source, labeller and licence on every row: huggingface.co/datasets/davaโฆ
4
217
๐ Paper: arxiv.org/abs/2610.17205 (will be updated to v1.1 soon!)
๐ค Models: huggingface.co/collections/jโฆ
๐ Data: huggingface.co/datasets/jgeuโฆ
โ๏ธ Code: github.com/j-geuter/Jeff
Joint work with Andrew Cheng and @elmelis.
1
95
@vanstriendaniel might be of interest to you! We've also released our training data now.
2
98
Jonathan Geuter retweeted
Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch.
No diffusion teacher. No flow map. No CFG. No JVP.
We believe this work will challenge how we think about building one-step models.
Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han.
๐ฅ 1.19 FID one-step on ImageNet 256ร256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L
๐จ Scales to text-to-image generation (GenEval 0.82)
๐ค World modeling on RT-1 / Bridge-V2
Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].
Everyone is talking about 'System One models' like Jev: given a state, question, and set of options, they return a calibrated decision at a fraction of the cost of an LLM call.
But do we need specialized architectures and recipes to train them?
We show you donโt:
๐งต๐
Our flagship model, Jeff-1.0-Large, scores 84.4 on OpenDecisions, a new state-of-the-art, and 81.5 on JevBench. That ๐ฏ๐ฒ๐ฎ๐๐ ๐๐ฒ๐ ๐ฏ๐ ๐ฐ.๐ฏ ๐ฎ๐ป๐ฑ ๐ฐ.๐ฐ ๐ฝ๐ผ๐ถ๐ป๐๐ and puts it at #๐ฎ ๐ผ๐ป ๐๐ต๐ฒ ๐๐ฒ๐๐๐ฒ๐ป๐ฐ๐ต ๐น๐ฒ๐ฎ๐ฑ๐ฒ๐ฟ๐ฏ๐ผ๐ฎ๐ฟ๐ฑ (almost tied for first).
1
2
149
๐ ๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ: arxiv.org/abs/2610.17205
๐ค ๐ ๐ผ๐ฑ๐ฒ๐น๐: huggingface.co/collections/jโฆ
๐ ๐๐ฎ๐๐ฎ๐๐ฒ๐: huggingface.co/datasets/jgeuโฆ
โ๏ธ ๐๐ผ๐ฑ๐ฒ: github.com/j-geuter/jeff-serโฆ
Joint work with Andrew Cheng and @elmelis
1
1
138









