Raven Huang retweeted
Does spatial understanding translate into reliable robot action?
Introducing BEHAVIOR-Spatial benchmark: Evaluating Spatial Understanding for Robotics.
Built on BEHAVIOR-1K, our benchmark connects three levels of spatial reasoning across shared scenes, objects, and relations:
🧠 BEHAVIOR-Spatial-VQA (Understand): answer spatial questions
🎯 BEHAVIOR-Spatial-IR (Ground): point to the instructed target
🤖 BEHAVIOR-Spatial-Action (Act): execute manipulation and navigation tasks
Across 3,256 questions, 1,240 grounding queries, and 24 robot tasks, we examine when spatial perception translates into robot behavior.
Our key test: keep the training scene fixed and change the instruction. Does the robot switch targets?
Our baseline commits to the wrong target in 70% of these probes. Explicit target cues help, but reliable instruction following remains challenging.
🎬 Overview below. Benchmark data, training demonstrations, evaluation tools, and findings in the thread. 🧵
Human videos bring scale and motion priors to robot pretraining. But video mainly captures human states, leaving intended actions implicit. Applied forces help fill that gap. Introducing FACT: learning dexterous manipulation from human video and force. 🧵
7/ FACT connects human contact experience to robot execution, from force supervision during pretraining to predicting and responding to contact on the robot.
Paper: arxiv.org/abs/2610.19155
Website & demos: fact-1-0.github.io/
1
4
313
8/ Works done with @ZhiCao01756604, @yining_hong, Jiaolong Yang, Yejin Choi,@drfeifei, and @jiajunwu_cs
1
336
Raven Huang retweeted
🤖Robots need world models for visual planning, but they don't need to imagine every frame.
Introducing DreamCan: Sparse Visual Planning for Long-Horizon Robot Manipulation.
Project: dreamcan.github.io/
Code: github.com/HeegerGao/DreamCa…
Paper: arxiv.org/abs/2610.24981
Raven Huang retweeted
We gave Claude and GPT a phone video of a robot wiping. In an hour, each built a full-dynamics digital twin.
Introducing RoboSimStudio 🤖: an open and interactive toolkit for Agentic Robotics, enabling Fable, Astra and Argon AI agents to model sponges 🧽, cables 🧵 and cloth 👔.
Raven Huang retweeted
Want to know how well agents can really control robots directly?
In RoboARCade, we systematically evaluate and diagnose frontier agents controlling robots directly using tens of thousands of dollars’ worth of agent trajectories. RoboARCade contains diverse, precise, and long-horizon tasks. Paper, code, assets and every trace are out today. roboarcade.org
Real2sim has made impressive progress with coding agents. But what about sim2real, especially the visual gap? Meet RoboRender: turning simulated trajectories into photorealistic videos for policy training, achieving 71% average success in zero-shot real-world robot deployment.
6/6 Simulation supplies geometry, motions, and action labels. Video generation adds realistic appearance and diversity.
Together, they bridge the visual sim2real gap for robot learning.
Paper: arxiv.org/abs/2610.09254
Website: robo-render.github.io/
1
7
736
Works done with @wensi_ai , @ziyuchen_ , @jwang633, Zijian Du, Yang Liu, Jiaolong Yang, @drfeifei , and @jiajunwu_cs
5
607




