
Research @Microsoft - Frontier Tuning | Scalable & Efficient RL for Frontier LLMs | PhD @ KAIST | ex-@mila_quebec (Bengio Lab) | Views are my own.
Redmond, Washington, USA
Joined March 2025
- Tweets127
- Following169
- Followers282
- Likes490
Minsu Kim retweeted
“Alignment vs Capabilities” is imo a mind-virus that slows good research. Many alignment researchers block themselves from doing solid work that meaningfully improves the frontier recipe. I believe this is one of several reasons why 95% of alignment/safety research is slop. This arbitrary categorization also blocks capabilities researchers from meaningfully engaging with “alignment” methods that if engineered correctly would actually boost capabilities significantly. There is “good science” and “sloppy science”. The reason why most “capabilities” research results in less-safe models is because they monkey-patch failure modes without principled understanding.
Concrete example I’ll focus on here— reward hacking.
The “monkey-patch” greedy solution is to modify grader rubrics, task specification, and environments whenever a problem is detected. There is a place for this (obviously we don’t want hackable sandboxes) but not scalable especially when problems are subtle and hard to catch/preempt. More principled solutions (i.e. scalable oversight methods) try to bake into the RL recipe robustness against many classes of reward hacks. This is still not universal and practically imperfect but significantly more scalable and reliable. Now it’s probably not obvious to many reading this whether such methods even exist, but I’m now quite certain they do.
Minsu Kim retweeted
What is science, really, and why do we do it?
In my new post I discuss the plurality of perspectives on these questions. As we automate the scientific process with AI, it is important to hold onto the deeply human origins of this collective practice!
mj10.github.io/blog/2026/per…
Minsu Kim retweeted
Replying to @JainMoksh
Great to see!
Credit to you, @siddarthv66, and everyone else who worked on arxiv.org/abs/2503.18929 -- you all saw how KL reg could help stabilize async RL for LLMs 2+ years ago. The EMA tweak Siddarth made is a robust refinement, it works better in all my experiments too.
Totally agree with @siddarthv66. Scalable oversight isn't just about alignment. Without reliable reward signals, scaling RL may simply make models better at reward hacking.
I've got a lot of RL/post training followers who may not know what scalable oversight is, which is what im working on now at resolution. It's frequently discussed in the alignment context but really the core problem is critical for capabilities too, and may be necessary to even reach no-human-in-the-loop RSI.
Put simply: how do we train models supervised by other models, and how to scalably do RL when the AI judges/reward functions are weaker than the policy and so can be reward hacked? In other words, methods to mitigate reward hacking when RLing fuzzy tasks. If this seems interesting to you, reach out. There's some cool work we'll be doing at res.
Minsu Kim retweeted
I've got a lot of RL/post training followers who may not know what scalable oversight is, which is what im working on now at resolution. It's frequently discussed in the alignment context but really the core problem is critical for capabilities too, and may be necessary to even reach no-human-in-the-loop RSI.
Put simply: how do we train models supervised by other models, and how to scalably do RL when the AI judges/reward functions are weaker than the policy and so can be reward hacked? In other words, methods to mitigate reward hacking when RLing fuzzy tasks. If this seems interesting to you, reach out. There's some cool work we'll be doing at res.
Minsu Kim retweeted
KAIST ranks #2 in the number of papers accepted to NeurIPS 2026, ahead of MIT and Stanford!
While paper count is not the primary metric we pursue, it certainly means something. A proud moment for the KAIST AI research community!
Minsu Kim retweeted
Our AI Interaction and Learning team at @MSFTResearch is hiring research interns for next summer!
We’re looking for strong PhD students interested in LLM post-training, long-horizon environment design, user collaboration, theory, and related areas who can join us in person in the US!
Learn more about the opportunities and apply through the link below!
We're hiring PhD research interns to join the MSR AI Interaction and Learning team for Summer 2027. We advance the science of large-scale AI systems through research on behavior, collaboration, evaluation, learning, and adaptation.
Learn more and apply: apply.careers.microsoft.com/…
Minsu Kim retweeted
Are self-evolving LLM agents actually continual learners?
In our new paper, we find that current self-evolving agents remain trapped in a plasticity–stability trade-off when the task distribution drifts.
Many recent recursive self-improvement (RSI) methods rely on rewriting an agent’s own prompts, memory, and code at test time while model weights stay frozen, which works well on stationary benchmarks. In real-world deployments, however, new tasks, tools, and workflows keep arriving, requiring agents to adapt on the fly without forgetting earlier capabilities. To test whether they can learn continually, we evaluate 3 representative methods (ReasoningBank, GEPA, and Meta-Harness) across 4 non-stationary task streams built from BIRD, BFCL, and ScienceWorld (Topic, Workflow, Domain, and Compound drift).
📄 Paper: arxiv.org/abs/2610.25858
TSP is one of the cleanest RL proxies for LLM reasoning and agents: combinatorial decisions, long horizons, and sparse terminal rewards.
People trained Transformers to generate near-optimal tours for this NP-hard problem. RLOO (a popular LLM RL method) was introduced by Wouter Kool et al. on TSP in 2019.
In my past work called GFACS, off-policy GFlowNets + metaheuristic search outperformed policy-gradient baselines on TSP. I expect this to carry over to LLM agents.
Recent work on GFlowRL (applied at 235B LLM) from Microsoft Research is encouraging in this regard.
GFACS: proceedings.mlr.press/v258/k…
GFlowRL: arxiv.org/abs/2607.13394
17
1,072
Minsu Kim retweeted
Decided to revive the blog and write a short post about embracing the agents and increasing the scope of what kind of research is possible. Very much in line with the general experience but a good excuse to get back to writing blog posts (more soon!)
mj10.github.io/blog/2026/fem…
Minsu Kim retweeted
There is some very nice non-LLM literature on this topic (e.g. arxiv.org/abs/2003.06350 and arxiv.org/abs/2106.03955 by @folinoid ), but the findings need to be revisited to account for the effects of starting a strong pretrained policy
Minsu Kim retweeted
At least some settings (model-data combinations) benefit from IS on top of this kind of regression objective, but in others IS isn't needed.
(IIUC, the TBA objective below matches CORO when you set the reference to be the inference policy and drop IS).
nitter.cf/bartoldson/status/1996…
Replying to @bartoldson
🔬 Ablation 2: Reference policy resetting
Resetting prevents the stability-adding KL penalty from harming task-reward optimization. This has an effect on performance almost as large as importance sampling's!
Highly recommended read from Google DeepMind and colleagues.
There is a lot of interest in self-evolving agents, but they're hard to get right.
This work highlights how important and difficult memory and retrieval are in agents.
The big question in this paper is whether self-evolving agents keep learning without forgetting.
They tested ReasoningBank, GEPA, and Meta-Harness on four task streams that drift in topic, workflow, or domain, built from BIRD, BFCL, and ScienceWorld.
The weights stay frozen, and the agents adapt by rewriting prompts, memory, and skills.
Self-evolution improves overall scores on most streams. But more adaptation consistently means agents forget more.
Methods that rewrite a shared prompt or workspace make global edits, so they interfere with unseen tasks and overwrite older knowledge when the stream shifts.
Memory retrieval keeps updates more local, but it still pulls in memories from other domains and collides when similar queries need different workflows.
Static benchmarks hide both problems, so test self-evolving agents on a stream that changes over time.
arxiv.org/abs/2610.25858
Chat with Paper: academy.dair.ai/papers/are-s…
Minsu Kim retweeted
“All roads to off-policy learning go through path consistency”
The CORO objective used here is equivalent to the TBA objective proposed in our paper from last year. All of these are variants of PCL and in the on-policy trajectory case yields same gradient as REINFORCE
arxiv.org/abs/2503.18929
Minsu Kim retweeted
New Google DeepMind paper finds that self-improving agents forget as they learn, and the method that learned new tasks fastest broke old ones the most.
Self-evolving agents improve without retraining by editing their system prompt, memory, or skill files. In real use tasks keep shifting, and each edit made for 1 task also applies to every task that follows.
The team ran ReasoningBank, GEPA, and Meta-Harness on task streams that changed topic, workflow, or domain. They tracked how much each method learned on the current task and lost on the others.
GEPA learned the most, 6.2 points per block on average, and also lost the most elsewhere, 4.3 points. After it learned to ask users for missing details, a later block rewrote that rule and its 15.5-point gain fell to 2.5.
If your agent edits its own prompt or memory, keep task-specific rules separate and retest old tasks after every update.
Minsu Kim retweeted
The CORO objective used here is equivalent to the TBA objective proposed in our paper from last year. All of these are variants of PCL and in the on-policy trajectory case yields same gradient as REINFORCE
arxiv.org/abs/2503.18929
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
Has anyone explored value-based off-policy RL (e.g., SQL, PCL, GFlowNets) for LLM post-training at the 500B+ parameter scale?
Curious if there have been any serious attempts at this scale.
Minsu Kim retweeted
context-beyond-window.github…
If you're attending COLM, do not miss @Yoshua_Bengio's 9am talk at our workshop on Friday!





















