
Applied Scientist @ Amazon Frontier AI & Robotics (FAR). Previously Ph.D. at @UTCompSci.
Joined April 2017
- Tweets98
- Following491
- Followers1.7K
- Likes1.1K
Pinned Tweet
How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)
Zhenyu Jiang retweeted
Super fun chatting with @micoolcho and @DJiafei!
I'll also be presenting this work at the CoRL Late-Breaking Results today from 12:30 to 1:30! Come and chat if you want to know more about it!
Coding agents powered by large language models are increasingly able to complete complex, long-horizon goals, but this capability is still largely relegated to computer tasks. Bringing agents to real-world robotics is much harder.
Enter SimEX, by @JiahengHu1, @SteveTod1998, and team. It is a simulation-based auto-research framework, wherein agents can develop skills which can be used to control real robots, developing a robust toolbox of skills which can then be used for real-world tasks like towel folding and plate manipulation.
To learn more, watch Episode 116 of RoboPapers with @micoolcho and @DJiafei!
Zhenyu Jiang retweeted
Full episode dropping soon!
Geeking out with @JiahengHu1 and @SteveTod1998 on “SimEX: Simulation-Integrated Robotics AutoResearch.”
robo-simex.github.io/
Co-hosted by @micoolcho and @DJiafei
Zhenyu Jiang retweeted
What to remember in the past to act—spatial, temporal or motion details?
We often assume what’s useful, but that’s hard to know in advance.
Meet IMPRINT 🧠: retain rich memory first, then learn to use it for control. Compact latents keep inference fast as history grows. 🧵 1/10
Zhenyu Jiang retweeted
What is the role of simulation in the era of Astra?
Traditionally, in sim2real transfer, we transfer data or policies, which requires relatively accurate simulation.
What excites me about “agentic sim2real” is that we can also transfer tools, knowledge, and procedures. A roughly correct simulator can give agents a place to test their physical intuition, run experiments, and build reusable capabilities.
That’s the idea behind SimEX, led by @JiahengHu1: develop capabilities through simulated experimentation, then use a few real-world trials to refine both the tools and the simulator.
We’ve all seen the cool “Astra for robotics” demos. But behind the scenes? Slow rollouts, hand-engineered APIs, and/or simulated behaviors that struggle on real robots.
How do we make coding agent actually work in the physical world? 🦾 Introducing SimEX: Simulation-Integrated Robotics AutoResearch 🧵
arxiv.org/abs/2609.38982
Excited to share our work, SimEX! Simulation works as a playground for coding agents to develop robot skills, while a few real-world trials help refine both the simulation envs and skills. A step toward coding agents that can act in the physical world!
We’ve all seen the cool “Astra for robotics” demos. But behind the scenes? Slow rollouts, hand-engineered APIs, and/or simulated behaviors that struggle on real robots.
How do we make coding agent actually work in the physical world? 🦾 Introducing SimEX: Simulation-Integrated Robotics AutoResearch 🧵
arxiv.org/abs/2609.38982
Zhenyu Jiang retweeted
[1/2] GPT-6 Astra might be the craziest robot policy I’ve seen. 🤯
Give it ONE demo per task, and it can control a real robot to unscrew a bottle cap, insert a plug into a power strip, clear obstacles, and even imitate a human dance.
One context. New behavior. 👇
Zhenyu Jiang retweeted
Humanoids need data. Lots and lots of data.
Introducing HumanoidMimicGen: a method that automatically generates 1000s of humanoid loco-manipulation demonstrations from a single teleoperated demonstration.
Zhenyu Jiang retweeted
We are back. After one year of quiet building.
Introducing GENE-26.5, our first robotic brain that takes a major step toward human-level capability.
For years, robotics has struggled to learn from the world’s largest and valuable data source: Humans.
Solving it means rethinking the whole stack from the ground up:
- A robotics-native foundation model.
- A 1:1 human-like robotic hand.
- A noninvasive data collection glove for motion, force, and touch.
- A simulator that turns weeks of experiments into minutes.
GENE-26.5 is trained across language, vision, proprioception, tactile, and action. We designed a set of tasks to test how far we can go with this new paradigm.
Fully autonomous, 1x speed, one model, same weights. (Enjoy with sound on)
We are approaching the endgame for robotics.
And this is just a beginning.
Zhenyu Jiang retweeted
🤖Co-training is everywhere (sim↔real[e.g. GR00T, LBM], human↔robot[e.g. PI, EgoScale], even non-robot data[e.g. PI, LBM).
But why does it work? How can we improve it further?
Taking sim-and-real imitation learning in diffusion/ flow-based models as the test bed, we performed a rigorous mechanistic analysis, drawing on theoretical insights and multi-layered experiments.
😮Key insight: it’s all about representations.
- Alignment → enables transfer
- Discernibility → enables adaptation
⚖️Both are necessary — it's better to have more aligned representations, but the model must be able to discern the domains. We term this as structured representation alignment.
⬇️Let’s take a deep dive into that:
Paper: arxiv.org/pdf/2604.13645
Website: science-of-co-training.githu…
Zhenyu Jiang retweeted
Discrete or continuous tokens? Or even tokenizer-free? The visual modeling debate rages on, but for now, let me introduce L24SQ, a provably optimal, regularizer-free quantizer with a large codebook (~200k), achieving SoTA reconstruction-compression tradeoff and generative power!
Zhenyu Jiang retweeted
Sim-to-real learning for humanoid robots is a full-stack problem. Today, Amazon FAR is releasing a full-stack solution: Holosoma.
To accelerate research, we are open-sourcing a complete codebase covering multiple simulation backends, training, retargeting, and real-world inference.
Zhenyu Jiang retweeted
Btw, the code is finally released: github.com/amazon-far/residu…
Hopefully, you'll find it helpful in your own research. Let us know if you encounter any issues or build on or extend the work!
I'm still excited about real-world RL, and residual RL as a policy-agnostic method for it.
How can we enable finetuning of humanoid manipulation policies, directly in the real world?
In our new paper, Residual Off-Policy RL for Finetuning BC Policies, we demonstrate real-world RL on a bimanual humanoid with 5-fingered hands (29 DoF) and improve pre-trained policies with ~15-75 minutes of robot interaction.
By learning residual corrections on frozen BC policies using sample-efficient off-policy RL, we achieve significant improvements in sample efficiency, enabling policy finetuning directly on the hardware — to our knowledge, one of the first examples of this on a humanoid with bimanual dexterous hands.
(If you know of other examples, let me know!)
Zhenyu Jiang retweeted
Humanoid motion tracking performance is greatly determined by retargeting quality!
Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL:
- 5 rewards,
- 4 DR terms,
- Proprio. ONLY,
- NO history/curriculum.
Ready for agile, human-like 🤖? (Best with 🎧)
🔗 omniretarget.github.io 🎥
1/9
Check out our work on residual RL for finetuning BC policies! We demonstrated real-world RL with a 29-DoF bimanual dexterous robot and improved the pre-trained base policy efficiently.
How can we enable finetuning of humanoid manipulation policies, directly in the real world?
In our new paper, Residual Off-Policy RL for Finetuning BC Policies, we demonstrate real-world RL on a bimanual humanoid with 5-fingered hands (29 DoF) and improve pre-trained policies with ~15-75 minutes of robot interaction.
By learning residual corrections on frozen BC policies using sample-efficient off-policy RL, we achieve significant improvements in sample efficiency, enabling policy finetuning directly on the hardware — to our knowledge, one of the first examples of this on a humanoid with bimanual dexterous hands.
(If you know of other examples, let me know!)
Check our work on learning humanoid manipulation from a few human video demonstrations!
Intelligent humanoids should have the ability to quickly adapt to new tasks by observing humans
Why is such adaptability important?
🌍 Real-world diversity is hard to fully capture in advance
🧠 Adaptability is central to natural intelligence
We present MimicDroid 👇
🌐 ut-austin-rpl.github.io/Mimi…
4
996
Check out Casper, a robot sidekick that actively predicts user intent and assists human interactively!
Zhenyu Jiang retweeted
(1/n) Many large-scale robot datasets have been released, but their quality varies. How can we curate them to improve policy performance? We present SCIZOR, a self-supervised framework that filters low-quality data, boosting policy performance in both simulation and real world.
Zhenyu Jiang retweeted
Ep#11 with @snasiriany @SteveTod1998 @abhirammaddukur @Lawrence_Y_Chen on Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation co-training.github.io
Co-hosted by @chris_j_paxton & @micoolcho
Zhenyu Jiang retweeted
Best poster finalist? @TairanHe99 @_wenlixiao 😂😂😂
Excited to be at Atlanta for #ICRA2025 next week and present our work DexMimicGen on Thursday! Please come check our presentation at 5:00 pm @ 411 and stop by our poster at poster board!























