
RS @IFM_AI | Prev @Articul8_AI @AmazonScience @allen_ai | MS CS @UMassAmherst. Towards super intelligence, One Algorithm at a Time.
Sunnyvale, CA
Joined January 2021
- Tweets370
- Following728
- Followers140
- Likes1.5K
Pinned Tweet
Agentic RL is a whole new beast. Things broke. A lot.
Pods crashed. Registries choked. Runs stalled.
I spent the last few months in the trenches isolating failures and fixing bottlenecks. RL360 is what came out of it: millions of rollouts, scaled across multiple experts.
Proud of this one. Here's how 👇
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
All the methods we used in training our models are shared in detailed in our tech report! We also released the data and code behind this.
P.S If you have any questions or find anything missing, please reach out!
Varad Pimpalkhute retweeted
Introducing PAN 2: an open foundation world model for physical intelligence.
A language model predicts the next word. PAN 2 predicts the next world: what happens after each action, grounded in time, physics and space.
Towards AI that simulates “what if?” before it acts.
More details: ifm.ai/pan/
Sad to see how benchmaxxed models have gotten. Tweak a filename or a small requirement and models “tied” on DeepSWE land 23 points apart.
This is what happens when the industry treats leaderboards as the goal instead of a proxy for good work.
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good at DeepSWE? 🧵
27
Varad Pimpalkhute retweeted
LLM training is about transporting high bandwidth of digits across hardware. This exercise allows us to understand a lot of the bottlenecks in this process.
Our upcoming technical paper (warning: it is loooong) will also share more!
We learned a TON in the last 6 months about Agentic RL infrastructure at scale, and now we're happy to share it with everyone.
RL360 was used for the training of K2 Horizon series (artificialanalysis.ai/models…). It builds on top of miles from @radixark, adding a lot of optimizations for efficient agentic RL at scale.
Our amazing team at @IFM_AI built this with a lot of hard work. Thanks to @varad0309 @willis_guo @wang1999_zt @mingshanhee @DavidRBellamy @omkarenator @_matthewyang @qiancheng1231 @bhandari_jalaj and many others!
Worked on the advisor part of this, so some thoughts ->
If your teacher model can't solve a task, you don't get training data for it. That's the ceiling everyone hits. The tempting fix is to just hand the model the answer. But then you're training on trajectories from a model that was told the answer, and that's not what you want to learn from.
So we looked at where it was actually failing, and it mostly wasn't the coding. It was judgment. Calling it done too early. Trusting one passing test. Getting it right, then "improving" it into a bug. That reframes the whole problem. You don't need a smarter model. You need someone who knows the answer looking over its shoulder and saying "not yet."
That's what the advisor is. It never touches the code. The solver does all the work, it just gets stopped when it’s about to quit at the wrong moment.
That alone took us from 54% to 73% of tasks producing a usable trajectory, and more than doubled it on the hardest set. Kind of how a good mentor works, honestly. They don't grab the keyboard. They just tell you you're not done yet.
Replying to @IFM_AI
A teacher model is limited by its own abilities. But the training data we generate doesn't have to be.
Two examples:
- Long coding sessions: Replay repository PR-based tasks. The teacher handles one task at a time, but the training trajectory captures the entire session, reaching hundreds of thousands of tokens.
- Privileged advisor: Use hidden-test feedback to identify mistakes and directly edit a solver's reasoning with targeted cues.
Varad Pimpalkhute retweeted
The most detailed open LLM data release yet: K2 Horizon's pre-training data and post-training data-generation pipelines are now open!
13 trillion pre-training tokens across 5 datasets and 24 subsets. Over 10 trillion tokens of synthetic data spanning mathematics, reasoning, coding, planning, and problem-solving.
Beyond the scale, four highlights for model builders (details in the threads):
- Scaling synthetic data without losing diversity. Our generation methods produce diverse training examples at scale, nearly matching natural web text in measured diversity.
- More verification isn't always better. Strict verification can throw away useful training signal.
- Small-scale wins can mislead. Our chosen data mix actually lost in smaller experiments.
- Beyond teacher imitation. Our post-training data generation methods produce training trajectories the teacher couldn't generate on its own.
Open pre-training datasets, post-training data-generation pipelines, and processing tools for researchers to inspect, reuse, and build on.
Pre-training data: huggingface.co/collections/I…
Post-training data-generation code & tools: github.com/ifm-ai/DATA_NAVIG…
Varad Pimpalkhute retweeted
8,192 concurrent sandboxes and ~1.3 million rollouts. Really impressive engineering from @IFM_AI!
One thing I find particularly interesting about long-horizon agentic RL is that infrastructure failures can directly affect training correctness. If a sandbox crashes halfway through a trajectory and the system treats it as a failed task, we're effectively penalizing the model for something it never did wrong. At this scale, distinguishing environment failures from actual model failures becomes part of the learning algorithm itself.
RL360 also demonstrates why modular RL infrastructure matters. With Miles handling rollouts and optimization, Harbor managing agent environments, SGLang serving inference, and Megatron-LM running training, each component can evolve and scale independently.
I'm especially happy to see Miles and SGLang being used together in such a large-scale, real-world RL system. The lessons about asynchronous execution, distributed trajectory management, and sandbox reliability are incredibly valuable for the entire open-source community.
Huge congratulations to the IFM team!
github.com/ifm-ai/RL360
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
Varad Pimpalkhute retweeted
We learned a TON in the last 6 months about Agentic RL infrastructure at scale, and now we're happy to share it with everyone.
RL360 was used for the training of K2 Horizon series (artificialanalysis.ai/models…). It builds on top of miles from @radixark, adding a lot of optimizations for efficient agentic RL at scale.
Our amazing team at @IFM_AI built this with a lot of hard work. Thanks to @varad0309 @willis_guo @wang1999_zt @mingshanhee @DavidRBellamy @omkarenator @_matthewyang @qiancheng1231 @bhandari_jalaj and many others!
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
Worked on this with the goat @omkarenator and grateful for the contributions from our cracked team @rupspace @DavidRBellamy @junukim01 @mingshanhee @_matthewyang @willis_guo @bhandari_jalaj @qiancheng1231
2
72
Varad Pimpalkhute retweeted
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
Interesting work from @Harman26Singh on measuring an agent's capacity to learn through experience and self-evolution.
🤖 How well do agents learn from experience?
We study agents that amortize experience into reusable artifacts and introduce Agent Plasticity to measure how efficiently they improve on held-out environments.
Top performer ≠ most efficient learner.
2
108
Varad Pimpalkhute retweeted
🔁 𝐖𝐡𝐚𝐭 𝐢𝐟 𝐥𝐨𝐨𝐩𝐞𝐝 𝐋𝐌𝐬 𝐜𝐨𝐮𝐥𝐝 𝐮𝐬𝐞 𝟑× 𝐟𝐞𝐰𝐞𝐫 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬 𝐚𝐧𝐝 𝐚 𝟑× 𝐬𝐦𝐚𝐥𝐥𝐞𝐫 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 𝐚𝐧𝐝 𝐬𝐭𝐢𝐥𝐥 𝐛𝐞 𝐜𝐨𝐦𝐩𝐚𝐫𝐚𝐛𝐥𝐞 𝐭𝐨 𝐚 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫?
Introducing 𝐑𝐞𝐭𝐡𝐢𝐧𝐤𝐢𝐧𝐠 𝐚𝐭 𝐅𝐢𝐱𝐞𝐝 𝐏𝐨𝐢𝐧𝐭𝐬. We argue that looped LMs don't have to pay for every loop. They can scale by adding FLOPs at constant memory.
The key is the fixed point of the loop. We introduce:
– 𝐅𝐢𝐱𝐞𝐝-𝐏𝐨𝐢𝐧𝐭 𝐒𝐡𝐨𝐫𝐭𝐜𝐮𝐭𝐬: what fixed points buy a looped model in training (pre-training, post-training) and inference (prefill, decoding)
– A 𝐥𝐞𝐚𝐫𝐧𝐞𝐝 𝐝𝐞𝐩𝐭𝐡 𝐩𝐫𝐢𝐨𝐫 and 𝐨𝐫𝐭𝐡𝐨𝐠𝐨𝐧𝐚𝐥 𝐢𝐧𝐣𝐞𝐜𝐭𝐢𝐨𝐧: two ways to shape those fixed points
🧵 Long thread ahead (0/n).
This is a really nice systems optimization.
With router replay, the obvious optimization is usually to reduce how much routing state needs to be transferred. But there's a second dimension that matters just as much: when you transfer it.
If routing decisions for the full conversation are exported only after a turn/conversation boundary, that communication can land directly on the critical path and create synchronization bubbles across workers.
Streaming only the newly generated routing decisions as each turn finishes lets communication overlap with ongoing rollout execution instead.
Simple idea, but a much cleaner abstraction, definitely something worth adopting 🥹
You saw the AI & science. Let's talk about the RL infra it took to build @periodiclabs Neon.
To minimize training-inference mismatch in RL, SGLang captures inference's MoE routing decisions for each rollout and we "replay" them while training.
In agentic (multiturn tool-use) settings, SGLang exports these router decisions in response to each decoding request, i.e. after each conversation turn.
So when *any* data-parallel rank finishes a conversation turn, *all* other ranks must wait until routing data finishes exporting. This slowdown is exacerbated because we export routing decisions from the *entire* conversation rather than just the most recent turn!
When @hsu_byron @vwxyzjn discovered this in our Kimi K2 RL setup, they introduced Delta Router Replay: cache previous turns' router decisions on the training client, so you can export only the delta (most recent turn's router decisions) upon each decoding request.
Delta Router Replay significantly speeds up our long-context agentic RL runs, and @hsu_byron upstreamed it to SGLang (#24851) a few months ago.
2
134
The thing that really sold me on UNO is that you don't need to maintain a separate draft model.
That's always been one of the things holding me back from getting too excited about the usual speculative decoding regime. A separate drafter means another model to train/distill, keep in sync with the target model, serve, tune, and continuously worry about as the target model changes.
UNO basically folds the drafting mechanism back into the target model itself, which makes the whole setup feel much more practical to me. More importantly, it opens up a much larger design space for speculative decoding without having to ask "okay, but what draft model are we going to maintain for this?"
Today’s LLMs still write like typewriters: one token at a time. This sequential process creates a hard inference bottleneck.
We're introducing Uno, a diffusion-augmented LLM that delivers autoregressive quality at diffusion speed. It’s a lossless speedup method that accelerates generation without degrading response quality.
With Uno, K2-Horizon-7B outperforms state-of-the-art diffusion methods in both quality and throughput, delivering up to a 2.2× speedup with no loss in quality.
Paper: arxiv.org/abs/2609.04010
Model available at: huggingface.co/IFM/K2-Horizo…
8
162
Massive respect! 🫡
Nearly half a year of silence. We spent it studying one problem: how far RL can scale.
MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks.
Streaming the run: mimo.xiaomi.com/rl/
4
104
Varad Pimpalkhute retweeted
IFM刚放出了个神奇7B Dense小模型 K2-Horizon-7B. 神奇的是这玩意在AA Bench里面有21分, 而Qwen3.6-27B是22分, 也就是说这7B参数量快追平了27B的水平.
甚至这个模型在诸如BrowseComp测试中碾压了前几代旗舰模型(GPT-5/DeepSeek-V4). 而且工程能力比如SWEBench Verified/ Terminal Bench 分数表现也很亮眼.
所以敲打一波Qwen, 赶紧放出你们压箱底的Qwen3.8-35B-A3B, 别藏着掖着了. 要被偷家了!
这个模型现在完全是社区明星了, 它的训练数据, 怎么训练的(recipe), 训练代码以及评估方法全都是开源的. 参数上这个模型最大支持512K上下文, 但是注意, 这玩意虽然参数只有7B, 但是它是全注意力的, 所以上下文成本相当高. 目前还只有原始BF16精度, 如果要用128K上下文, 就要18G显存/统一内存. 所以还是等等8bit/4bit量化版本比较好(我刚在X上 @ unsloth 了一波, 看看哥俩会不会有时间做量化吧)
以及这个模型同样支持设置思考强度. 分为low/medium/high, 然后默认/官方强烈推荐用high.
如果现在不差显存想直接用可以使用vLLM/SGLang. 这俩已经支持了. 等 unsloth 放出量化版我给大家来一期10B以下小模型横评. MiniCPM5-2B 和其他几个小模型已经在跑了.
#K2Horizon7B #MiniCPM52B #Qwen3835BA3B
Yup, merging in RL just kind of works ~
wait DeepSeek is “model merging” during RL??
used across diff Harness/Config checkpoints to combine/init information
so we can train on N diff harnesses, take the successful ones, merge those checkpoints, and keep training?
wild that this just works well 👀
6
105
























