Robotics + AI System PhD @UCBerkeley
Berkeley, CA
Joined August 2022
- Tweets41
- Following371
- Followers467
- Likes142
Pinned Tweet
We gave Claude and GPT a phone video of a robot wiping. In an hour, each built a full-dynamics digital twin.
Introducing RoboSimStudio 🤖: an open and interactive toolkit for Agentic Robotics, enabling Fable, Astra and Argon AI agents to model sponges 🧽, cables 🧵 and cloth 👔.
Kaiyuan Eric Chen retweeted
Each AI model has different strengths. Combining and harnessing them is currently more art than science but this will evolve, esp given new results on agentic weight tuning and RSI.
Note how KEC combines Fable and Astra with RSS to rapidly tune complex simulation parameters for geometry, dynamics, deformations, materials, lighting, texture, and camera extrinsics to match a given input video. This fills a previous gap in Agentic Robotics ability to scale to new environments, objects, and tasks.
Kaiyuan Eric Chen retweeted
So impressed by foundation LLMs skill at using RSS to tune complex robot manipulation sim parameters for geometry, dynamics, deformations, materials, lighting, texture, and camera extrinsics to match a given video.
Since we launched RoboSimStudio three weeks ago:
🏆 CoRL Late Breaking Results
🎤 Oral presentation at the PhysWM Workshop (speakers: @tom_erez @erwincoumans @ylecun @YunzhuLiYZ)
🌟 300 GitHub stars
Thank you all 🙏
We recently found a new way to do agentic real2sim: using Claude Fable commanding GPT-6 Astra.
Give it an online image, or even an Amazon link, and they collaborate to recreate the world in RoboSimStudio, ready for robots under 2 hours
Since we launched RoboSimStudio three weeks ago:
🏆 CoRL Late Breaking Results
🎤 Oral presentation at the PhysWM Workshop (speakers: @tom_erez @erwincoumans @ylecun @YunzhuLiYZ)
🌟 300 GitHub stars
Thank you all 🙏
The paper, code, examples, and interactive 3D rollouts are all open 🌐 Site, 📄 Paper,
🎮 Demo: robosimstudio.github.io
💻 Code: github.com/robosimstudio/rob…
Kaiyuan Eric Chen retweeted
This is one of my favorite works in my series on antenna impedance characteristic sensing. This time, we use a commercially available device. Without any hardware modification, just with a software update. Then, your phone can sense your hand grip, worn placement, resting surfaces, and plug-and-play passive devices.
Kaiyuan Eric Chen retweeted
Let's turn everyday things into real-life Tamagotchi!
Introducing Object-Agent Twins (OATs): an open-source toolkit that allows users to easily create small, personality-bearing agents tied to real-world objects.
Video below (1/n)
Weekend project after codex reset. I took one photo of my desk, then used AI agent to turn it into a Microduck playground 💛 like Minions taking over my workspace.
Built with RoboSimStudio. Everything is open.
I took a picture with my desk with complex outdoor lighting but Real2Sim with Astra can reconstruct the scene and bring the Microducks (thanks to @pollenrobotics, @NVIDIARobotics) to life.
Below is the only input I gave to RoboSimStudio:
Everything is built with RoboSimStudio, our open-source framework for agentic robot simulation developed by @berkeley_ai and @NVIDIARobotics.
Try it yourself 👇
github.com/robosimstudio/rob…
And code for generating the video
github.com/KeplerC/microduck…
Thank you all for the interest and support!!! More features on agentic real2sim will be released in the next couple of days :)
That's genuinely insane! 🤯
An AI agent built a physics twin of a robot task from a phone video in under an hour.
This project RoboSimStudio was built on Newton by researchers from UC Berkeley and NVIDIA Robotics.
Open toolkit, every physics parameter exposed as something a human can slide or an agent can edit.
The demo is a bimanual setup, one arm holding a bar, the other wiping a marker stripe off it with a foam sponge. Input was a smartphone video.
What's more important no CAD, and no calibrated cameras. Claude and GPT-6 Astra each built a simulation of it independently. One finished in 56 minutes.
Then they ran it on the real robot and the bar support moved. In simulation it had been modelled as fixed, so the sponge kept contact. In reality it shifted and contact was lost.
The agent went back and replaced the fixed constraint with a free rigid body. The revised simulation then reproduced the tipping the original could not.
That is the loop actually closing. The real world found the modelling error and the fix went back into the twin.
🔗 Check the project here: lnkd.in/dgXJxAhw
~~
♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com
Kaiyuan Eric Chen retweeted
Follow Gemini 4 Argon's report, I implemented their robotics harness.
Gemini did a zero-shot bimanual handover. Now open source 🤖👇
Kaiyuan Eric Chen retweeted
😫Tired of spending hours teleoperating robots?
🚀What if you could collect 2.9× more demonstrations with the same human effort?
Introducing TANDEM: our new work that combines autonomous task and motion planning (TAMP) with human teleoperation only when needed. 🧵 1/n
Kaiyuan Eric Chen retweeted
Gemini-4 Argon shows a large improvement in robotics understanding on RoboVista.
It reaches 81.9%, beats GPT-6 Astra and Claude Fable 5.1 in our evaluation with lowest eval cost. Compared with Gemini 3 Flash, it improves across every robotics domain. Example VQA👇
🤖How well do today's VLMs actually understand real-world robotics? 👀
Excited to share RoboVista at #RSS2026 — a systematic evaluation and benchmark for VLMs across diverse, real-world robot applications. Website, dataset and paper: berkeleyautomation.github.io…
Developed by researchers at @UCBerkeley, @GoogleDeepMind, and @Princeton. 🧵👇
The paper, code, examples, and interactive 3D rollouts are all open
🌐 Site, 📄 Paper, 🎮 Demo: robosimstudio.github.io
💻 Code: github.com/robosimstudio/rob…
Thanks to the great team!
@eric_heiden @erwincoumans @milesmacklin #spencer_huang from @nvidia @NVIDIARobotics and
@Shuangyu0_0Xie, @UksangYoo, @XinyueZhuXY , Cael Magner, @alvinyzhu , @talking_kim, and @Ken_Goldberg from @UCBerkeley @berkeley_ai