phd student @carnegiemellon • interning @amazon far • prev @nvidia, @georgiatech • reinforcement learning, differentiable simulation, robotics
Joined July 2018
- Tweets64
- Following379
- Followers469
- Likes1.6K
Eliot Xing retweeted
When SceniX joined World Labs, we said spatial intelligence was never only about perceiving and generating virtual and physical worlds, but also interacting with them. Today, we’re sharing early results from that vision: building worlds that train robots. 🌎🤖↓
Eliot Xing retweeted
RIO, a new framework from CMU, provides a unified interface for robot control, data collection, teleoperation and AI deployment!
It gives researchers a lightweight, modular foundation for deploying robots quickly on any platform.
ri.cmu.edu/robot-io/
Eliot Xing retweeted
Newton 1.0 is now generally available. 🙌
Take robot learning to the next level with:
🤖 Stable Articulated & Complex Mechanism Simulation – accurate, reliable machine modeling.
🖐️ High-Fidelity Hydroelastic Contact Modeling – realistic soft contact and touch-based interactions.
🧵 Deformable Body Simulation – simulate cables, cloth, rubber, and other elastic materials with VBD.
⚡ Accelerated Robot Learning at Scale – seamless integration with open simulation and learning frameworks, NVIDIA Isaac Sim and Isaac Lab for scalable workflows.
Learn how to integrate this open-source physics engine into your workflow: nvda.ws/3NGTzUo
#NVIDIAGTC
Eliot Xing retweeted
How do you give a humanoid the general motion capability? Not just single motions, but all motion?
Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control.
We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames.
But tracking alone is not enough; we show how to make a useful control system out of it:
- Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game.
- VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset.
- VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically.
- VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution!
We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain.
This is the generalist "System 1" for humanoids. 🚀
Project: nvlabs.github.io/SONIC/
#Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥
Eliot Xing retweeted
What if robots could improve themselves by learning from their own failures in the real-world?
Introducing 𝗣𝗟𝗗 (𝗣𝗿𝗼𝗯𝗲, 𝗟𝗲𝗮𝗿𝗻, 𝗗𝗶𝘀𝘁𝗶𝗹𝗹) — a recipe that enables Vision-Language-Action (VLA) models to self-improve for high-precision manipulation tasks.
PLD couples real-world residual reinforcement learning with standard supervised fine-tuning — letting robots discover, recover, and distill their own data flywheel.
Quick 🧵
Eliot Xing retweeted
The laws of physics apply everywhere. ⚛️
Co-developed by NVIDIA, @GoogleDeepMind and Disney Research, Walt Disney Imagineering, the new Newton Beta, managed by the @linuxfoundation, now also runs with Isaac Lab and with MuJoCo Warp, which delivers warp speed for robot learning.
Learn how to train a quadruped and simulate multiphysics in our tech blog. 🔗 nvda.ws/4gP2woL
#CoRL2025
Eliot Xing retweeted
📣MuJoCo announcement 📣
Thrilled to share that @GoogleDeepMind has unveiled MuJoCo-Warp at @nvidia's #GTC25!
🚀 We've expanded our open-source MuJoCo simulator with MuJoCo-Warp, leveraging NVIDIA’s Warp framework for incredible acceleration. This marks a significant step in making high-performance simulation more accessible.
Eliot Xing retweeted
🚀 We've already extended MOE to various dexterity projects including dynamic pen spinning (arxiv.org/abs/2411.12734) and learning in-hand manipulation skills from demonstration (arxiv.org/abs/2503.01078). 🧵6/7
Eliot Xing retweeted
Very Cool!
Thank you for bringing us this HUGE win!!!
RL is notoriously sample inefficient. How can we scale RL on tasks much slower to simulate than rigid body physics, such as soft bodies?
In our #ICLR2025 spotlight, we introduce both a new first-order RL algorithm, SAPO, and differentiable simulation platform, Rewarped. 1/n
Eliot Xing retweeted
RL for deformables!
RL is notoriously sample inefficient. How can we scale RL on tasks much slower to simulate than rigid body physics, such as soft bodies?
In our #ICLR2025 spotlight, we introduce both a new first-order RL algorithm, SAPO, and differentiable simulation platform, Rewarped. 1/n
Eliot Xing retweeted
Congratulations @etaoxing! This is a super cool use of Warp and differentiable simulation!
RL is notoriously sample inefficient. How can we scale RL on tasks much slower to simulate than rigid body physics, such as soft bodies?
In our #ICLR2025 spotlight, we introduce both a new first-order RL algorithm, SAPO, and differentiable simulation platform, Rewarped. 1/n
Eliot Xing retweeted
Warp still holds huge potential.
RL is notoriously sample inefficient. How can we scale RL on tasks much slower to simulate than rigid body physics, such as soft bodies?
In our #ICLR2025 spotlight, we introduce both a new first-order RL algorithm, SAPO, and differentiable simulation platform, Rewarped. 1/n
Eliot Xing retweeted
The ultimate test of any physics simulator is its ability to deliver real-world results.
With MuJoCo Playground, we’ve combined the very best: MuJoCo’s rich and thriving ecosystem, massively parallel GPU-accelerated simulation, and real-world results across a diverse range of robot platforms: quadrupeds, humanoids, dexterous hands, and arms.
Best of all? You can get started today with a single command: pip install playground
playground.mujoco.org/
Eliot Xing retweeted
Everything you love about generative models — now powered by real physics!
Announcing the Genesis project — after a 24-month large-scale research collaboration involving over 20 research labs — a generative physics engine able to generate 4D dynamical worlds powered by a physics simulation platform designed for general-purpose robotics and physical AI applications.
Genesis's physics engine is developed in pure Python, while being 10-80x faster than existing GPU-accelerated stacks like Isaac Gym and MJX. It delivers a simulation speed ~430,000 faster than in real-time, and takes only 26 seconds to train a robotic locomotion policy transferrable to the real world on a single RTX4090 (see tutorial: genesis-world.readthedocs.io…).
The Genesis physics engine and simulation platform is fully open source at github.com/Genesis-Embodied-…. We'll gradually roll out access to our generative framework in the near future.
Genesis implements a unified simulation framework all from scratch, integrating a wide spectrum of state-of-the-art physics solvers, allowing simulation of the whole physical world in a virtual realm with the highest realism.
We aim to build a universal data engine that leverages an upper-level generative framework to autonomously create physical worlds, together with various modes of data, including environments, camera motions, robotic task proposals, reward functions, robot policies, character motions, fully interactive 3D scenes, open-world articulated assets, and more, aiming towards fully automated data generation for robotics, physical AI and other applications.
Open Source Code: github.com/Genesis-Embodied-…
Project webpage: genesis-embodied-ai.github.i…
Documentation: genesis-world.readthedocs.io…
1/n
Eliot Xing retweeted
Can robots make pottery🍵? Throwing a pot is a complex manipulation task of continuously deforming clay. We will present RoPotter, a robot system that uses structural priors to learn from demonstrations and make pottery @HumanoidsConf @CMU_Robotics
👇robot-pottery.github.io 1/8🧵
Eliot Xing retweeted
🤖 Humanoids are expected to be the ultimate embodiment for general-purpose robots, but reaching that potential means adapting to diverse tasks—navigation, tabletop manipulation, loco-manipulation—each needing unique control modes.
🤔 Can one controller handle it all?
✨ Introducing HOVER:
🧩 Unified Command Space: Supports kinematic position, joint angle, and root tracking in a unified versatile policy.
💪 Generalist Controller: HOVER outperforms previous specialized policies trained on specific modes (ExBody, H2O, OmniH2O, HumanPlus).
🌐 Website: hover-versatile-humanoid.git…
Eliot Xing retweeted
Want to scale RL with your shiny new GPU? 🚀
In our ICML24 Oral we find that RL algorithms hit a barrier when data is scaled up. Our new algorithm, SAPG, proposes a simple fix. It scales to 25k envs and solves hard tasks where PPO makes no progress.
sapg-rl.github.io/
1/n
Eliot Xing retweeted
I’ve been training dogs since middle school. It’s about time I train robot dogs too 😛
Introducing, UMI on Legs, an approach for scaling manipulation skills on robot dogs🐶It can toss, push heavy weights, and make your ~existing~ visuo-motor policies mobile!
Eliot Xing retweeted
Introducing DrEureka🎓, our latest effort pushing the frontier of robot learning using LLMs!
DrEureka uses LLMs to automatically design reward functions and tune physics parameters to enable sim-to-real robot learning. DrEureka can propose effective sim-to-real configurations for several robots and tasks, and we even got a bit creative with it: Let’s make a robot dog walk and balance on a yoga ball! Check out these fun videos, and follow the thread for a deep dive!
