Zhenyu Jiang (@SteveTod1998) header
Zhenyu Jiang (@SteveTod1998) profile picture

Applied Scientist @ Amazon Frontier AI & Robotics (FAR). Previously Ph.D. at @UTCompSci.

Joined April 2017
Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)Photo by Zhenyu Jiang (@SteveTod1998)
How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)
8
52
10
241
45,517
Zhenyu Jiang retweeted
Super fun chatting with @micoolcho and @DJiafei! I'll also be presenting this work at the CoRL Late-Breaking Results today from 12:30 to 1:30! Come and chat if you want to know more about it!
Coding agents powered by large language models are increasingly able to complete complex, long-horizon goals, but this capability is still largely relegated to computer tasks. Bringing agents to real-world robotics is much harder. Enter SimEX, by @JiahengHu1, @SteveTod1998, and team. It is a simulation-based auto-research framework, wherein agents can develop skills which can be used to control real robots, developing a robust toolbox of skills which can then be used for real-world tasks like towel folding and plate manipulation. To learn more, watch Episode 116 of RoboPapers with @micoolcho and @DJiafei!
1
3
20
812
Zhenyu Jiang retweeted
Full episode dropping soon! Geeking out with @JiahengHu1 and @SteveTod1998 on “SimEX: Simulation-Integrated Robotics AutoResearch.” robo-simex.github.io/ Co-hosted by @micoolcho and @DJiafei
7
18
2,493
What to remember in the past to act—spatial, temporal or motion details? We often assume what’s useful, but that’s hard to know in advance. Meet IMPRINT 🧠: retain rich memory first, then learn to use it for control. Compact latents keep inference fast as history grows. 🧵 1/10
5
29
11
185
22,314
Zhenyu Jiang retweeted
What is the role of simulation in the era of Astra? Traditionally, in sim2real transfer, we transfer data or policies, which requires relatively accurate simulation. What excites me about “agentic sim2real” is that we can also transfer tools, knowledge, and procedures. A roughly correct simulator can give agents a place to test their physical intuition, run experiments, and build reusable capabilities. That’s the idea behind SimEX, led by @JiahengHu1: develop capabilities through simulated experimentation, then use a few real-world trials to refine both the tools and the simulator.
We’ve all seen the cool “Astra for robotics” demos. But behind the scenes? Slow rollouts, hand-engineered APIs, and/or simulated behaviors that struggle on real robots. How do we make coding agent actually work in the physical world? 🦾 Introducing SimEX: Simulation-Integrated Robotics AutoResearch 🧵 arxiv.org/abs/2609.38982
5
12
1
131
11,991
Excited to share our work, SimEX! Simulation works as a playground for coding agents to develop robot skills, while a few real-world trials help refine both the simulation envs and skills. A step toward coding agents that can act in the physical world!
We’ve all seen the cool “Astra for robotics” demos. But behind the scenes? Slow rollouts, hand-engineered APIs, and/or simulated behaviors that struggle on real robots. How do we make coding agent actually work in the physical world? 🦾 Introducing SimEX: Simulation-Integrated Robotics AutoResearch 🧵 arxiv.org/abs/2609.38982
3
29
2,418
Zhenyu Jiang retweeted
[1/2] GPT-6 Astra might be the craziest robot policy I’ve seen. 🤯 Give it ONE demo per task, and it can control a real robot to unscrew a bottle cap, insert a plug into a power strip, clear obstacles, and even imitate a human dance. One context. New behavior. 👇
6
21
5
148
31,665
Zhenyu Jiang retweeted
Humanoids need data. Lots and lots of data. Introducing HumanoidMimicGen: a method that automatically generates 1000s of humanoid loco-manipulation demonstrations from a single teleoperated demonstration.
12
44
8
255
33,101
Zhenyu Jiang retweeted
We are back. After one year of quiet building. Introducing GENE-26.5, our first robotic brain that takes a major step toward human-level capability. For years, robotics has struggled to learn from the world’s largest and valuable data source: Humans. Solving it means rethinking the whole stack from the ground up: - A robotics-native foundation model. - A 1:1 human-like robotic hand. - A noninvasive data collection glove for motion, force, and touch. - A simulator that turns weeks of experiments into minutes. GENE-26.5 is trained across language, vision, proprioception, tactile, and action. We designed a set of tasks to test how far we can go with this new paradigm. Fully autonomous, 1x speed, one model, same weights. (Enjoy with sound on) We are approaching the endgame for robotics. And this is just a beginning.
281
1,113
425
5,784
2,753,015
Zhenyu Jiang retweeted
🤖Co-training is everywhere (sim↔real[e.g. GR00T, LBM], human↔robot[e.g. PI, EgoScale], even non-robot data[e.g. PI, LBM). But why does it work? How can we improve it further? Taking sim-and-real imitation learning in diffusion/ flow-based models as the test bed, we performed a rigorous mechanistic analysis, drawing on theoretical insights and multi-layered experiments. 😮Key insight: it’s all about representations. - Alignment → enables transfer - Discernibility → enables adaptation ⚖️Both are necessary — it's better to have more aligned representations, but the model must be able to discern the domains. We term this as structured representation alignment. ⬇️Let’s take a deep dive into that: Paper: arxiv.org/pdf/2604.13645 Website: science-of-co-training.githu…
7
64
16
394
68,795
Zhenyu Jiang retweeted
Discrete or continuous tokens? Or even tokenizer-free? The visual modeling debate rages on, but for now, let me introduce L24SQ, a provably optimal, regularizer-free quantizer with a large codebook (~200k), achieving SoTA reconstruction-compression tradeoff and generative power!
4
34
2
197
30,127
Sim-to-real learning for humanoid robots is a full-stack problem. Today, Amazon FAR is releasing a full-stack solution: Holosoma. To accelerate research, we are open-sourcing a complete codebase covering multiple simulation backends, training, retargeting, and real-world inference.
19
131
37
593
220,663
Zhenyu Jiang retweeted
Btw, the code is finally released: github.com/amazon-far/residu… Hopefully, you'll find it helpful in your own research. Let us know if you encounter any issues or build on or extend the work! I'm still excited about real-world RL, and residual RL as a policy-agnostic method for it.
How can we enable finetuning of humanoid manipulation policies, directly in the real world? In our new paper, Residual Off-Policy RL for Finetuning BC Policies, we demonstrate real-world RL on a bimanual humanoid with 5-fingered hands (29 DoF) and improve pre-trained policies with ~15-75 minutes of robot interaction. By learning residual corrections on frozen BC policies using sample-efficient off-policy RL, we achieve significant improvements in sample efficiency, enabling policy finetuning directly on the hardware — to our knowledge, one of the first examples of this on a humanoid with bimanual dexterous hands. (If you know of other examples, let me know!)
4
27
174
19,939
Zhenyu Jiang retweeted
Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 omniretarget.github.io 🎥 1/9
32
152
42
670
831,638
Check out our work on residual RL for finetuning BC policies! We demonstrated real-world RL with a 29-DoF bimanual dexterous robot and improved the pre-trained base policy efficiently.
How can we enable finetuning of humanoid manipulation policies, directly in the real world? In our new paper, Residual Off-Policy RL for Finetuning BC Policies, we demonstrate real-world RL on a bimanual humanoid with 5-fingered hands (29 DoF) and improve pre-trained policies with ~15-75 minutes of robot interaction. By learning residual corrections on frozen BC policies using sample-efficient off-policy RL, we achieve significant improvements in sample efficiency, enabling policy finetuning directly on the hardware — to our knowledge, one of the first examples of this on a humanoid with bimanual dexterous hands. (If you know of other examples, let me know!)
4
35
4,089
Check our work on learning humanoid manipulation from a few human video demonstrations!
Intelligent humanoids should have the ability to quickly adapt to new tasks by observing humans Why is such adaptability important? 🌍 Real-world diversity is hard to fully capture in advance 🧠 Adaptability is central to natural intelligence We present MimicDroid 👇 🌐 ut-austin-rpl.github.io/Mimi…
4
996
Check out Casper, a robot sidekick that actively predicts user intent and assists human interactively!
Meet Casper👻, a friendly robot sidekick who shadows your day, decodes your intents on the fly, and lends a hand while you stay in control! Instead of passively receiving commands, what if a robot actively sense what you need in the background, and step in when confident? (1/n)
1
17
1,945
Zhenyu Jiang retweeted
(1/n) Many large-scale robot datasets have been released, but their quality varies. How can we curate them to improve policy performance? We present SCIZOR, a self-supervised framework that filters low-quality data, boosting policy performance in both simulation and real world.
3
23
5
170
28,435
Zhenyu Jiang retweeted
Ep#11 with @snasiriany @SteveTod1998 @abhirammaddukur @Lawrence_Y_Chen on Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation co-training.github.io Co-hosted by @chris_j_paxton & @micoolcho
2
9
2
41
13,905
Best poster finalist? @TairanHe99 @_wenlixiao 😂😂😂
3
4
77
5,215
Excited to be at Atlanta for #ICRA2025 next week and present our work DexMimicGen on Thursday! Please come check our presentation at 5:00 pm @ 411 and stop by our poster at poster board!
How can we scale up humanoid data acquisition with minimal human effort? Introducing DexMimicGen, a large-scale automated data generation system that synthesizes trajectories from a few human demonstrations for humanoid robots with dexterous hands. (1/n)
1
7
61
4,658