co-founder @HuggingFace - moonshots get your Microducks and Reachy Mini's at https://nitter.cf/t.co/ovuIUzu5Ov
Joined February 2011
- Tweets5.9K
- Following8K
- Followers137K
- Likes34.4K
Thomas Wolf retweeted
OCR models often get stuck in loops, so I asked Claude Opus if it could stop one from looping by editing its weights.
It changed 0.0085% of them and with no training, it now reads this 1881 page instead of writing 16,399 characters of dots!
Prompt was basically: "read this, find a smallish OCR model and fix its repetition doom loops. You can spend up to 5 hours and $100 on @huggingface Jobs."
It only took under 3 hours in the end and about $10 of Jobs compute.
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Thomas Wolf retweeted
very nice open weight release from ricursive with a modern dense model, benchmark scores look very strong!
beautiful scaling ladder, interesting choices like value residual (interesting contribution pattern 👀) and.. SLIDING WINDOW ATTENTION AGAIN!
ricursive.com/ripple
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: ricursive.com/ripple
Thomas Wolf retweeted
> we developed and trained Ripple-65B Preview for only $32M, which is one-tenth the cost of the previous open-weight SOTA on agentic coding¹
this is like the avengers of open model SOTA
¹ "Note that $32M is our full R&D cost starting from scratch, including pretraining ladder development, data mixing, evals, debugging, synthetic data generation, posttraining recipe development, and the final hero pretraining and posttraining run."
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: ricursive.com/ripple
Thomas Wolf retweeted
Most think vibecoding at scale is slop. Here’s a counterexample.
Today we release a 16M-line formalization of the entire Stacks Project, critical infra underlying modern math, with optimized code quality and its highly interconnected structure preserved.
A mathematical city 🌆
Kimi-K3.1-Pro by Moonshot AI: interesting facts
It's huge, and sparse
• It has 2.8T total parameters, but only 104B are active per token.
• Along with Kimi-K3, it's the largest real model release on the Hub. The next biggest are Qwen3.8-2.4T and DeepSeek-V4-Pro (1.6T).
• It has 896 experts and sends each token to 16 of them. (DeepSeek-V4-Pro has 2 times less and Mistral-Large-4 has 7 times less)
Clever attention for a 1M-token context
• Most of its 93 layers use linear attention (KDA). Every 4th layer uses DeepSeek-style sparse attention, which only looks at the top 2048 tokens.
• This is how it handles a 1M-token context (the same as the other frontier open models)
• It uses a custom activation called SiTU-GLU and has a 401M-parameter vision encoder (MoonViT-V2), so it takes both text and images.
Small on disk for its size
• It was trained to run in 4-bit (MXFP4), so the whole repo is only ~1.56 TB.
• Stored in regular BF16 it would be about 5.6 TB(!)
Benchmarks (from the model card)
• It goes back and forth with Claude Opus 5.5 and GPT-6 Astra. It's top of the card's table on BrowseComp (92.1), HLE with tools (70.0) and SWE Marathon (51.7).
• It's weaker on pure reasoning without tools and on cyber security.
• Moonshot ran most of their own coding evals using the Claude Code harness.
License (a modified MIT)
• It's free to use, modify and redistribute.
• API businesses making over $20M a year need a deal with Moonshot.
• Products with over 100M monthly users or over $20M monthly revenue must display "Kimi K3.1-Pro".
On the Hub
• It went public about 1.5 hours ago and already has 214 likes, about 9× Kimi-K3's first hour. It's #19 on the trending list.
• It isn't gated, but loading it in transformers requires "trust_remote_code"
Kimi-K3.1-Pro is now open-weights
huggingface.co/moonshotai/Ki…
Thomas Wolf retweeted
Kimi-K3.1-Pro is now open-weights
huggingface.co/moonshotai/Ki…
Thomas Wolf retweeted
Evaluating robot models in the real world is hard, but solvable. That's why we're a founding partner of RoboColosseum 🤗
100+ real robots, 140+ in-the-wild scenes, and every rollout, failures included, open on the Hub.
Introducing RoboColosseum
An open robot eval initiative by @huggingface @BitRobotNetwork @eaic_sg @DJiafei
Access:
- 100+ robots (@UnitreeRobotics G1s, @LeRobotHF SO-101s, YAMs, Franka and more)
- 140+ in-the-wild scenes
- Open rollout data
- 10,000 FREE eval hrs
Thread ↓
Thomas Wolf retweeted
It's been a rough few weeks for security.
I pulled together a guide of the highest-impact actions you can take today:
Thomas Wolf retweeted
Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.
We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in parameters per token.
After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.
w/ @akshayvegesna, @EthanBaron78553, @asatheesh6561
Full paper led by @EthanBaron78553, with @ShmuelBerman, @andrewgwils
Thomas Wolf retweeted
Introducing 🧬 FineDNA: the best dataset to teach any language model DNA.
Training Qwen3-14B on it gives the strongest open model.
300B+ tokens of genomic sequences, papers and web pages mixing DNA and text, plus 200k SFT and 98k RL samples.
huggingface.co/datasets/Hugg…
Thomas Wolf retweeted
i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models.
Thomas Wolf retweeted
It looks like Astra didn't natively read Qwen's weights?
As I understand it, Qwen would get stuck in a loop and not answer the question.
Astra used code to perform some standard interp techniques, found a looping direction, and ablated it from the weights.
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
what?
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
WHOOP: $199/year to tell you that you didn't move
Reachy Mini Hoverboard: $149, once, and your robot is free to roam
and it's all open-source: plug in your favorite Opus / Qwen / Gemini
Thomas Wolf retweeted
@scale_ai is teaming with @huggingface to make robots better at long, multi-step tasks, and we’re open-sourcing all of it!
Today @scale_AI is launching FineART-VLA, a model that helps robots complete longer tasks, building on our first open-source robotics dataset.
Built with @huggingface 's LeRobot team, this is Scale’s first open-source robotics release.
Thanks to @Thom_Wolf + team for the partnership.
The momentum on open-source robotics is picking up steam, and high-quality data will determine how far it goes. Go build!
labs.scale.com/blog/fineart
Thomas Wolf retweeted
A little peek inside the factory.
Final adjustments to the Microduck injection molds are underway before our pre-production run.
(Sorry for putting NSFW content on your feed 🫣)
Thomas Wolf retweeted
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
