CS PhD @StanfordAILab, Part Time @GoogleDeepMind | Reinforcement Learning
Joined August 2015
- Tweets82
- Following155
- Followers1.8K
- Likes30
Perry Dong retweeted
In language models, turning a generative model into a critic is simple: pass in the output and prompt for a score. Robot policies can't do this - they output physical actions & there's no built-in vocabulary to predict scores.
So when it comes to RL on robot policies, we typically train a separate critic from sparse rewards - blind to everything the policy learned from the actions in the pretraining data.
We think this has it backwards - can a critic learn from those actions the way the policy does? (1/n)
Perry Dong retweeted
LLMs deployed in specialized domains, from enterprise codebases to law firms, dig through a mountain of unstructured context!
Our new paper, Librarian Agents, asks: can an agent adapt by curating its own long-context workspace, anticipating queries before they arrive? 🧵⬇️
Perry Dong retweeted
Want to know how well agents can really control robots directly?
In RoboARCade, we systematically evaluate and diagnose frontier agents controlling robots directly using tens of thousands of dollars’ worth of agent trajectories. RoboARCade contains diverse, precise, and long-horizon tasks. Paper, code, assets and every trace are out today. roboarcade.org
Perry Dong retweeted
@perryadong has some awesome thoughts on how we can bring the scale and stability of RL post-training that we see in LLMs to large robotics models! Check it out
What will be the “RLHF” moment for robotics? What will it take to get robotics to where LLMs are today and beyond?
New blog post with @chelseabfinn sharing some thoughts on the state of RL for frontier robotics models and what's missing 👇
Blog: pd-perry.github.io/posts/pos…
Perry Dong retweeted
Been meeting a lot of startups tackling the deployment layer of robotics. What does it mean to finish the last mile when verification data is expensive to generate and the environment itself can be stochastic? @perryadong has some good thoughts on what robotics RL looks like!
What will be the “RLHF” moment for robotics? What will it take to get robotics to where LLMs are today and beyond?
New blog post with @chelseabfinn sharing some thoughts on the state of RL for frontier robotics models and what's missing 👇
Blog: pd-perry.github.io/posts/pos…
Perry Dong retweeted
great piece on post-training for robotics!
What will be the “RLHF” moment for robotics? What will it take to get robotics to where LLMs are today and beyond?
New blog post with @chelseabfinn sharing some thoughts on the state of RL for frontier robotics models and what's missing 👇
Blog: pd-perry.github.io/posts/pos…
Perry Dong retweeted
New blog post with @perryadong on what we need for robots to be broadly useful in the real world.
pd-perry.github.io/posts/pos…
Reliability is the one of the biggest open challenges in AI right now. Current models work out okay if a person will be reviewing the outputs (eg drafting code), but it will become more of a bottleneck as we want systems to act with more autonomy and more trust.
What will be the “RLHF” moment for robotics? What will it take to get robotics to where LLMs are today and beyond?
New blog post with @chelseabfinn sharing some thoughts on the state of RL for frontier robotics models and what's missing 👇
Blog: pd-perry.github.io/posts/pos…
Perry Dong retweeted
RL for dynamic tasks requires handling latency.
Building on EXPO-FT, we let the base VLA operate on older images while a small edit policy operates in real time.
Outperforms RTC by a significant margin, across the board.
Paper: pd-perry.github.io/real-time…
What will be the “RLHF” moment for robotics? What will it take to get robotics to where LLMs are today and beyond?
New blog post with @chelseabfinn sharing some thoughts on the state of RL for frontier robotics models and what's missing 👇
Blog: pd-perry.github.io/posts/pos…
Introducing Real-Time EXPO-FT – Fast and Reliable RL for Real-Time VLA Policies!
Real-Time EXPO-FT unlocks π0.5 on challenging dynamic tasks, such as balancing a ball on a plate and striking a ball into the goal
(1/6)
Project with @khhung906, @DorsaSadigh, @chelseabfinn
Website: pd-perry.github.io/real-time…
Paper: arxiv.org/abs/2609.18207
Open-source code: github.com/pd-perry/expo-ft
Try it out today!
(6/6)
World models have emerged as one of the biggest directions in physical AI. At the same time, RL fine-tuning is unlocking capabilities in frontier models beyond what pretraining can achieve on its own
Can we get the best of both worlds?
We propose Q-Learning with World Models (QWM)
(1/7)
QWM scales to frontier video models
We turn a 5B-parameter video model into an action-conditioned world model: given an image + action, it predicts the next observation for tree search
QWM with frontier video models also consistently improves over the base RL method
(6/7)
QWM suggests another role for world models:
Use them with RL fine-tuning to enable test-time scaling
Project with Yueru Jia, @chelseabfinn, @DorsaSadigh
Paper: arxiv.org/abs/2608.17163
Website: pd-perry.github.io/qwm/
(7/7)
