CS PhD (Robotics) @ Cornell | Prev UC Berkeley
Joined March 2023
- Tweets72
- Following564
- Followers166
- Likes996
Pinned Tweet
Introducing Unified Motion-Action (UMA) Model, a robot foundation model that uses 3D object motion as a shared interface for heterogeneous robot learning. UMA treats motion and action as co-evolving variables, enabling knowledge transfer across data sources and versatile inference. 🧵 1/n
Excited to share SpatialDex! 🖐️🖐️
Human videos are everywhere, but RGB-only clips throw away the 3D structure that dexterity depends on. We reconstruct it back and pre-train a bimanual policy on 1M clips.
Details about SpatialDex will follow shortly (🧵 1/n)
SpatialDex learns 3D-grounded bimanual dexterity from 1M RGB-only human video clips. It reconstructs scene and hands in each video, aligns them at one scale, and trains the policy on both, nearly doubling the success of prior VLAs on real tasks. spatial-dex.github.io/
To investigate how each component of SpatialDex contributes to its success, we vary pre-training with fixed architecture and robot data: No human pre-training: 45.6% Hand motion: 58.9% Hand motion + scene geometry: 73.1% Per-frame instead of per-video scale: 63.8%. (🧵 8/9)
Was a lot of fun working with Xingyi He, @Calvin_Qiu2408 , Adhitya Polavaram, Cory Fan, Julius Samwer, @s_kalantari, and @KuanFang at Cornell.
Paper: arxiv.org/abs/2610.29716
Project page: spatial-dex.github.io/
Code and data will be released soon. (🧵 9/9)
Yunhao Cao retweeted
Humanoids can learn new skills by tracking high-quality reference motions, which today come from humans through motion capture or video. Routing every reference through a human proxy limits scalability and compounds reconstruction errors with the embodiment gap. KinoGen synthesizes loco-manipulation references directly for the target robot through kinematically grounded video generation. 🧵 (1/7)
I'm presenting COIL at two #IROS2026 workshops today! 🤖
🕥 10:30–11 AM · CompRobotics workshop poster (Rm 408/409)
🕒 3:15–4:15 PM · WORLDS workshop poster (Rm 301/302)
Come by and ask questions, discuss open problems with me!
Yunhao Cao retweeted
How can generalist policies learn new skills without losing their pretrained prior?
We introduce Proxy Policy Steering (PPS), which uses proxy policies to steer the frozen base policy at inference, adding task-specific behavior without touching its weights.
I definitely agree. I think Astra is the first pretrain using synthetic text data generated from robot data, so further finetuning with an action head is easier since a subspace of the text embedding would already be robot actions. We'll probably see an openai vla soon.
gpt-6 for robotics is truly a step change, but there's a reason for it and a bigger implication
very likely (duh) it was trained on robotics data, predicting end effector movement at least (e.g. "the position of the hand")
of course, it's slow, but if you're working on a task that does not require speed nor reflexes, then it is surprisingly powerful!
now if they have this data, they don't have to stop there. very likely we will see soon one of their first actual robotics models!
and hopefully quickly available in the api :)
Yunhao Cao retweeted
Behavioral cloning mystery
seohong.me/blog/behavioral-c…
I wrote a new blog post about "mysteries" in behavioral cloning that appear with real-world robot data (e.g., overfitting is "good"). I also tried to demystify them and shared my thoughts!
Yunhao Cao retweeted
Introducing KITE 🪁, a paradigm that transfers the interaction intent instead of joint-space actions across structurally different manipulators. Deploying on a new robot needs only its kinematics model (URDF), no new task demonstrations. 🧵1/8
Our analysis isolates the design choices that matter, confirms that UMA scales with both data and model size, and traces its remaining errors to execution rather than perception. 10/n
Check out our paper and project page for more details!
🌐 Website: uma-manipulation.github.io/
📝 Paper: arxiv.org/pdf/2606.16917
Huge thanks to my collaborators Shitong Liu, @chaof1234 , Meryl Zhang, @XuanchenLu , and mentors @andrewhowens , @KuanFang .