Los Angeles, CA
Joined August 2021
- Tweets65
- Following406
- Followers147
- Likes1.9K
Pinned Tweet
Hierarchical methods for offline goal-conditioned RL (GCRL) can scale to very distant goals that stymie flat (non-hierarchical) policies — but are they really necessary?
Paper: arxiv.org/abs/2505.14975
Project page: johnlyzhou.github.io/saw/
Code: github.com/johnlyzhou/saw
Thread ↓
John Zhou retweeted
Doing proper science in robotics is hard, because human demonstrations are expensive and less controllable.
OCBench is a new benchmark providing "human-like" scripted policies. This enables scientific studies of BC and RL with virtually unlimited data!
John Zhou retweeted
Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.
Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵
John Zhou retweeted
Both offline RL and LLM RL fine-tuning can be formulated as behavior-regularized RL problems.
We propose Value Grdient Flow (VGF), a new scalable and sample-efficient paradigam that treats behavior-regularized RL as an optimal transport problem.
arxiv.org/abs/2604.14265 🧵[1/7]
John Zhou retweeted
We're moving to a future vision of fully synthetic pre-training for LLMs. Our new work explores using Neural Cellular Automata to embed reasoning before language training even begins!
I'm deeply grateful to @seungwookh, @akarshkumar0101, and @pulkitology for their mentorship, guidance, and deep insights throughout this work.
Can language models learn useful priors without ever seeing language?
We pre-pre-train transformers on neural cellular automata — fully synthetic, zero language. This improves language modeling by up to 6%, speeds up convergence by 40%, and strengthens downstream reasoning.
Surprisingly, it even beats pre-pre-training on natural text!
Blog: hanseungwook.github.io/blog/…
(1/n)
John Zhou retweeted
MolmoSpaces provides singular scale and diversity. We built a benchmark that puts that scale to use.
MolmoSpaces-Bench evaluates zero-shot policies across thousands of environments previously unseen to them under systematic variation, providing insights that go beyond a success rate %
More Below:
John Zhou retweeted
Very excited to release Contact-Anchored Policies (CAP) 🧢 today! Check out this thread for more details on that and on our in-the-loop simulation evaluations:
John Zhou retweeted
We developed a new action-chunking RL method that scales well!
tl;dr: Use *longer* action chunks for value learning (than for policy learning).
Action chunking is drawing growing interest in RL, yet its theoretical properties are still understudied.
We are excited to share some insights on when we should use action chunking in Q-learning + a new algo (DQC) to tackle hard long-horizon tasks!colinqiyangli.github.io/dqc🧵1/N
John Zhou retweeted
We built methods to handle both (1) and (2), but I’ll focus on a stupid simple trick that works particularly well: adversarial training. Adversarial training makes input gradients better behaved, in turn making gradient-based planning fast and easy. 8/11
If you’re at #NeurIPS2025 from Tuesday to Sunday and interested in any of: offline RL, offline-to-online finetuning, VLM value functions/reward models/VLAs, or RL for real-world robots, please reach out and let’s chat!
John Zhou retweeted
As a robotics researcher, I believe accurately modeling complex interactions between agents would be a big step for scaling up robot learning from unlabeled video. Looking forward to some inspiring discussion with the Cohere Labs Embodied AI community!
Don't miss our Embodied AI group's session this week on November 21st with @sshchang for a presentation on "FLAM: Scaling Latent Action World Models with Factorization."
Thanks to @nahidalam and Cole Harrison for organizing this event! ✨
Learn more: cohere.com/events/cohere-lab…
John Zhou retweeted
We scaled up an "alternative" paradigm in RL: *divide and conquer*.
Compared to Q-learning (TD learning), divide and conquer can naturally scale to much longer horizons.
Blog post: seohong.me/blog/rl-without-t…
Paper: arxiv.org/abs/2510.22512
↓
John Zhou retweeted
I had a fun chat with @chris_j_paxton and @micoolcho about the scalability of RL for robotics!
Offline reinforcement learning is crucial for robotics, but does it scale? We talk to @seohong_park , who discusses how for long-horizon manipulation problems the answer may be no — at least not yet. But there are tricks that you can use to make it work effectively.
Watch episode #38 of RoboPapers with @micoolcho and @chris_j_paxton now!
John Zhou retweeted
Introducing *dual representations*!
tl;dr: We represent a state by the "set of similarities" to all other states. This dual perspective has lots of nice properties and practical benefits in RL.
Blog post: seohong.me/blog/dual-represe…
Paper: arxiv.org/abs/2510.06714
↓
John Zhou retweeted
Congrats to @UCLA Assoc. Prof. Jonathan Kao of @ECE_UCLA @dgsomucla, on receiving the National Institutes of Health @NIH Director’s Pioneer Award, a 5-year, $5.5 million grant to fund Kao’s research into noninvasive assistive devices for the paralyzed. samueli.ucla.edu/ucla-samuel…
SAW will be presented as a spotlight @NeurIPSConf 2025! Check out more details below:
Hierarchical methods for offline goal-conditioned RL (GCRL) can scale to very distant goals that stymie flat (non-hierarchical) policies — but are they really necessary?
Paper: arxiv.org/abs/2505.14975
Project page: johnlyzhou.github.io/saw/
Code: github.com/johnlyzhou/saw
Thread ↓
John Zhou retweeted
RLZero will be presented at @NeurIPSConf 2025 . Learn more about the work in the thread below:
🤖 Introducing RL Zero 🤖: a new approach to transform language into behavior zero-shot for embodied agents without labeled datasets! RL Zero enables prompt-to-policy generation, and we believe this unlocks new capabilities in scaling up language-conditioned RL, providing an interpretable link between RL agents and humans and achieving true cross-embodiment transfer.
John Zhou retweeted
This is one of the biggest problems facing academic research in robotics of course. At smaller scales lots of things work at ~70%, how do you know what can get to ~99% when scaled up? You dont.
John Zhou retweeted
The issue in the first paragraph is real when learning without bootstrapping (e.g., with reinforce). TD learning methods can already learn along the way and figure out what went well and what didn't if the value function has a good understanding of the world. This works even if rewards are delayed by hours.
Adding planning updates to the mix allows agents to reason about actions that it did not take and could try in the future.
Scaling up RL is all the rage right now, I had a chat with a friend about it yesterday. I'm fairly certain RL will continue to yield more intermediate gains, but I also don't expect it to be the full story. RL is basically "hey this happened to go well (/poorly), let me slightly increase (/decrease) the probability of every action I took for the future". You get a lot more leverage from verifier functions than explicit supervision, this is great. But first, it looks suspicious asymptotically - once the tasks grow to be minutes/hours of interaction long, you're really going to do all that work just to learn a single scalar outcome at the very end, to directly weight the gradient? Beyond asymptotics and second, this doesn't feel like the human mechanism of improvement for majority of intelligence tasks. There's significantly more bits of supervision we extract per rollout via a review/reflect stage along the lines of "what went well? what didn't go so well? what should I try next time?" etc. and the lessons from this stage feel explicit, like a new string to be added to the system prompt for the future, optionally to be distilled into weights (/intuition) later a bit like sleep. In English, we say something becomes "second nature" via this process, and we're missing learning paradigms like this. The new Memory feature is maybe a primordial version of this in ChatGPT, though it is only used for customization not problem solving. Notice that there is no equivalent of this for e.g. Atari RL because there are no LLMs and no in-context learning in those domains.
Example algorithm: given a task, do a few rollouts, stuff them all into one context window (along with the reward in each case), use a meta-prompt to review/reflect on what went well or not to obtain string "lesson", to be added to system prompt (or more generally modify the current lessons database). Many blanks to fill in, many tweaks possible, not obvious.
Example of lesson: we know LLMs can't super easily see letters due to tokenization and can't super easily count inside the residual stream, hence 'r' in 'strawberry' being famously difficult. Claude system prompt had a "quick fix" patch - a string was added along the lines of "If the user asks you to count letters, first separate them by commas and increment an explicit counter each time and do the task like that". This string is the "lesson", explicitly instructing the model how to complete the counting task, except the question is how this might fall out from agentic practice, instead of it being hard-coded by an engineer, how can this be generalized, and how lessons can be distilled over time to not bloat context windows indefinitely.
TLDR: RL will lead to more gains because when done well, it is a lot more leveraged, bitter-lesson-pilled, and superior to SFT. It doesn't feel like the full story, especially as rollout lengths continue to expand. There are more S curves to find beyond, possibly specific to LLMs and without analogues in game/robotics-like environments, which is exciting.
John Zhou retweeted
🗓️ Mark your calendars to join us in Edmonton on Aug 5 for:
Reinforcement Learning Beyond Rewards: Ingredients for Generalist Agents
With 🔥 speakers:
@chelseabfinn , George Konidaris, @furongh,
@gio_ramponi, @MichaelD1729, Ahmed Touati
#RLC2025 #RL @RL_Conference
