Martin Marek retweeted
Tinker’s Dimuon is Muon with delayed diagonal conditioning, Polar Express and a rank-dependent update norm. I recovered its update rule through the API, borrowing @mrtnm’s approach to StableReinforce. Here’s how 🧵
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback!
See our optimizers guide at tinker-docs.thinkingmachines…
Martin Marek retweeted
score centering + masked importance sampling is now the default in TRL's async GRPO Trainer!
Across various staleness and sampler quantization settings, the stability is pretty wild 👇
Martin Marek retweeted
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
Martin Marek retweeted
We reproduced this in TRL's AsyncGRPO 👇
Qwen-1.5B, vLLM in FP8 (weights + KV) and trainer in bf16:
• plain GRPO collapses after ~150 steps, reward 0.56 → 0.10
• ratio caps and masked IS alone don't save it
• score centering + MIS trains like a bf16 sampler (0.72)
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Martin Marek retweeted
We've just added score centering to slime!
You can enable sc with --use-score-centering
PR: github.com/THUDM/slime/pull/…
![[slime] Support score centering by zhuzilin · Pull Request #2405 · THUDM/slime](https://pbs.twimg.com/card_img/2115295460290113536/_kNuLYx0?format=jpg&name=800x419)
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
Martin Marek retweeted
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
Martin Marek retweeted
New paper: arxiv.org/abs/2605.26097
The main idea is that we can use an LLM to generate its own replay data to prevent forgetting, as long as we have spare capacity. Very overtrained models have to forget to learn new information.
Martin Marek retweeted
The three biggest hps for stable training in everything are lr, bs, and beta2. We’ve built up good intuitions on how to tune them over time, but this lays it all out analytically and convincingly.
this is definitely my new handbook for training big models on small gpus.
Martin Marek retweeted
Small batch size used to be crucial for training deep networks but the story was different for LLMs. This paper argues it's merely due to untuned small batch baselines and shows that actually, small batch sizes are better for LLMs too.
Martin Marek retweeted
I am recruiting Ph.D. students for my new lab at @nyuniversity! Please apply, if you want to work with me on reasoning, reinforcement learning, understanding generalization and AI for science.
Details on my website: izmailovpavel.github.io. Please spread the word!
Martin Marek retweeted
Open-source Mac app Burst Photo promises to reduce noise significantly dpreview.com/news/1008671751…
Martin Marek retweeted
I simplified your HMC implementation (by dropping features like multi-device support), vectorized the outer loop and put everything into a single 100 loc Python file. I think it makes for a simple yet fast HMC implementation 🙂 github.com/martin-marek/mini…
Martin Marek retweeted
After a lot of work, we have finished the implementation of PEP 657 🚀🎉. In Python 3.11🐍, tracebacks will annotate where exactly the error is happening in your code 🤯. No more confusion having to guess what part of the expression is wrong. Learn more at python.org/dev/peps/pep-0657…
Martin Marek retweeted
Danish news site reminds people of being skeptical of how news photographers take pictures. Here is the same place, one picture taken with a zoom lense, and the other with a wide-angle lense: nyheder.tv2.dk/samfund/2020-…
Martin Marek retweeted
The simple task of recognizing a stop sign is harder than we can imagine. @karpathy youtu.be/hx7BXih7zx8
Martin Marek retweeted
Four years ago, #AlphaGo and the legendary Lee Sedol took on an unprecedented challenge. To celebrate the landmark anniversary, we’re releasing AlphaGo The Movie for free on @YouTube. Relive the experience here: youtu.be/WXuK6gekU1Y
Martin Marek retweeted
A comprehensive study on Bayesian inference in DNNs. I guess only within Google can you conduct such careful experiments, interesting read!
Take-away: Bayesian posterior is rather poor and prior seems to be a big problem (don't scale to large nets).
arxiv.org/abs/2002.02405
Martin Marek retweeted
🚨New lecture series🚨 We've teamed up with @ai_ucl to bring you the #UCLxDeepMind Deep Learning Lecture Series: 12 lectures covering a range of topics in Deep Learning - all led by DeepMind researchers, all free, and all open to everyone. Info & tickets: eventbrite.co.uk/o/ucl-x-dee…