Armaan Abraham retweeted
In language models, turning a generative model into a critic is simple: pass in the output and prompt for a score. Robot policies can't do this - they output physical actions & there's no built-in vocabulary to predict scores.
So when it comes to RL on robot policies, we typically train a separate critic from sparse rewards - blind to everything the policy learned from the actions in the pretraining data.
We think this has it backwards - can a critic learn from those actions the way the policy does? (1/n)
Armaan Abraham retweeted
I've been thinking a lot about making robot learning scalable. To me, that means learning on the job, and using the human help still needed efficiently.
In Mulligan ⛳️, we focus each data-collection round on the initial states where the robot fails. This happens in deployment, with the robot doing the task and a person stepping in when needed.
On 3 real tasks and 2,550 blind evals, it improves on uniform collection at the same budget.
Armaan Abraham retweeted
We should stop optimizing robot policies against a single overall reward. Trajectories differ along many axes, such as speed, precision, and subtask completion, and one can be better on some while worse on others. If we collapse all of that into a single overall axis we lose this structure making the reward ambiguous and harder to optimize.
Blog: freeform-pl.github.io/fpl.we…
Paper: arxiv.org/abs/2606.32027
In RL, what if we could learn from any experience from any policy in a way that is reliable and scalable?
This would be helpful in domains like robotics where new data is expensive.
We introduce Long-horizon Q-learning (LQL) to tackle this arxiv.org/abs/2605.05812.
We scale LQL trajectory length on humanoidmaze-giant, OGBench's longest task (thousands of steps, single sparse reward):
1-step TD: 0%
n-step TD: peaks at n=4 (38.4%), drops to 6.1% at n=64
LQL (L=64): 75.7%
1
4
730
It has been great to work with @lucy_x_shi and @chelseabfinn on this!




