Founder, TransGaming Inc. Now I make educational video games for robots at NVIDIA. Opinions are my own.
Toronto, Ontario
Joined November 2012
- Tweets410
- Following363
- Followers786
- Likes675
After we released the original Isaac Gym and then the Orbit framework, I took a step back from robotics to focus on 3D reconstruction and rendering - NeRFs and 3D Gaussians.
Then, in October 2024, I got a call from @JensenHuang , asking me to come back and take charge of Isaac Lab.
I was really enjoying being deep in the weeds working on technologies that were defining the future of 3D graphics. I also understood exactly what it would mean to take this on, and the level of commitment it would require.
So, I hesitated.
“It’s very flattering that you think I’m the right person for this, but I’ve been away from robotics for a couple of years, and things move fast. I don’t know our current strategy, or even who’s on the team now.”
Jensen’s response was simple, direct, and effective: “You decide.”
With that level of trust, the opportunity was hard to pass up.
Over the past two years, I’ve had the privilege of working with an extraordinary group of collaborators to define the future of robotics simulation with Newton and Isaac Lab 3.0.
I couldn’t be more proud of our incredible, hardworking team and what we’ve been able to accomplish together. Seeing everything come to fruition at GTC last week was amazing.
Thanks, Jensen!
We’re excited to announce the official release of NVIDIA Isaac Lab 3.0. 🎉
Isaac Lab 3.0 brings a new modular and lightweight architecture along with major new capabilities across the robot learning workflow: multi-physics and multi-rendering support, training at scale, and streamlined deployment on real hardware.
With Newton integration, the new OVRTX renderer, RL post-training, and LEAPP deployment, there’s a lot to explore.
Here’s a look at what’s new 🧵
(1/7)
I mentioned there was more exciting stuff to come after SGS 😀. @octi_zhang's latest work shows the kind of incredible emergent behaviour we can get from large scale RL in simulation.
I've been applying these same ideas to other tasks, and have seen impressive results already!
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
The value of scaled RL training in simulation can't be understated. Huge congratulations to @octi_zhang and everyone else who worked on the SGS paper. What comes next is even cooler - stay tuned!
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
Fantastic work by @jayjunleee and the @NVIDIARobotics Dex research group. Always great to see new foundational sim2real work leveraging Isaac Lab!
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
Gavriel State retweeted
SimFoundry is now open source. Explore it here: github.com/NVlabs/SimFoundry
Give it a try and tell us what you think!
Today, we're introducing SimFoundry, our real2sim2real framework at NVIDIA GEAR that automatically turns real-world scenes into simulation-ready worlds from a single image or video.
Website: research.nvidia.com/labs/gea…
Paper: arxiv.org/abs/2606.28276
This work marks a major step for our team toward leveraging simulations and synthetic data for foundation model training and systematic policy evaluation at scale. Code will be open-sourced soon. Stay tuned!
Gavriel State retweeted
Newton 1.5 is now available, making robot training at scale faster and more reliable.
This release delivers:
🔁 More parallel simulation, lower memory, selective resets
🤝 Consistent contact physics
🤖 Experimental batched GPU control
📦 Cleaner USD/MJCF imports
Get started: github.com/newton-physics/ne…
Gavriel State retweeted
New in @ScienceMagazine 🎉
Really excited to share that SONIC is now published in Science Robotics!
SONIC formulates super-scaling motion tracking as the foundational task for more natural and robust whole-body control. Our learned tracker and latent space further support foundation models like VLA/WAM training.
Huge thanks to the team and collaborators who made this happen.
science.org/doi/10.1126/scir…
New checkpoints coming in today!
#ScienceResearch #ScienceRoboticsResearch
Gavriel State retweeted
Closing the sim-to-real gap for humanoid robotics starts with better simulation environments.
Join us, @NianticSpatial, and @FlexionAI on Aug. 12 at 11 AM PT to see a real-to-sim workflow using 3D Gaussian splatting, Omniverse NuRec, Isaac, and OpenUSD.
📅 Add to calendar: nvda.ws/3SmCmSu
So cool! Just ordered mine!
Any chance you guys will release the Isaac Lab training setup and USD models? Even better to be able to update the model on the robot…
Mondo's Beni (mondorobotics.com/) may look cute, but there is a serious learning stack inside. It can perform flips, navigate autonomously, and track moving subjects. The secret is reinforcement learning: Beni practices at massive scale in simulation, optimizing its controller through millions of virtual trials before entering the real world.
Here is a thread on some of the research ideas and tools used by Beni: 🧵
Gavriel State retweeted
1 million downloads and counting for Isaac Lab. 🥳
Thank you to the global robotics, embodied AI, and developer community using our open-source framework to train and develop the next generation of robots.
Gavriel State retweeted
Today, we're introducing SimFoundry, our real2sim2real framework at NVIDIA GEAR that automatically turns real-world scenes into simulation-ready worlds from a single image or video.
Website: research.nvidia.com/labs/gea…
Paper: arxiv.org/abs/2606.28276
This work marks a major step for our team toward leveraging simulations and synthetic data for foundation model training and systematic policy evaluation at scale. Code will be open-sourced soon. Stay tuned!
Gavriel State retweeted
How can robots learn dexterous manipulation from human demonstrations at scale?
Excited to share CHORD: Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration.
CHORD learns from human demos by focusing not only on where contact happens, but how that contact moves the object through force and torque guidance.
This unified contact-wrench representation carries human manipulation skills across diverse behaviors, long-horizon tasks, whole-body embodiments, and real-world hardware.
We evaluated CHORD on large-scale, long-horizon, contact-rich tasks paired with human demonstrations, spanning rigid, articulated, and multi-object manipulation.
At scale:
* 82.12% average success across 1,831 tasks
* 90.77% whole-body manipulation success
* 4,739 sim-ready dexterous manipulation benchmark
* Transfer to real dexterous hands
Project page: nvidia-isaac.github.io/video…
Tech report: nvidia-isaac.github.io/video…
Code will be released soon as part of Video to Data repo github.com/nvidia-isaac/vide…, our end-to-end pipeline for converting human demonstration videos into simulation-ready assets and physics-grounded robot training data.
Huge thanks to amazing contributors: @zhu_xinghao , Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Huihua Zhao, John Welsh, @michaelv03, Wei Liu, @TingwuWang , Xingye (Dennis) Da, @zhengyiluo, Vishal Kulkarni, @sNaema, @yukez, @DrJimFan, @bowenwen_me, @danfei_xu, @SohaPouya, @Dr_YanChang.
#Robotics #PhysicalAI #DexterousManipulation #RobotLearning #NVIDIA
Gavriel State retweeted
🎉 Excited to introduce TRON, a relighting framework for 3D captures.
💡TRON pairs a neural renderer with 3D Gaussian reconstructions, achieving realistic quality, with 3D, material, & lighting control at interactive frame rates.
arxiv.org/abs/2606.11314
research.nvidia.com/labs/sil…
Gavriel State retweeted
Super excited to open source Flashdreams, acceleration library for world models! No restrictions on usage, Apache 2 license.
World models are the next frontier for video generation and paramount to Physical AI.
Flashdreams yields out of the box speed up across world models
World models are moving beyond offline generation towards interactive, real-time experiences.
Introducing ⚡FlashDreams⚡: an open-source high-performance inference and serving library built for autoregressive world models:
🔥 Up to 3.10× faster LingBot-World inference
🔥 Up to 2.12× faster Self-Forcing inference
🔥 Up to 1.40× faster Wan2.1 inference
🔥 8 integrated models
🔥 Multi-GPU, streaming, low-latency serving
🔥 Agentic skills that teach you how to use it
FlashDreams is designed for a new generation of AI systems that continuously evolve over time while responding to user interactions. It powers applications across robotics, autonomous vehicle simulation, gaming, and virtual worlds.
Github: github.com/NVIDIA/flashdream…
Docs: nvidia.github.io/flashdreams
Research page: research.nvidia.com/labs/sil…
Join the #flashdreams Discord channel at discord.gg/yTdHDqFP
FlashDreams is also the runtime backbone behind NVIDIA OmniDreams (github.com/nv-tlabs/omni-dre…)
1/n
#AI #WorldModels #FastInference #PhysicalAI #OpenSource #NVIDIA
Gavriel State retweeted
Faster-GS: github.com/nerficg-project/f…
PPISP: research.nvidia.com/labs/sil…
Btw, don't miss the PPISP oral if you're at CVPR next week.
(3/3)
Gavriel State retweeted
🚀We just released Asset Harvester, an image-to-3D model and end-to-end pipeline that extracts real object assets from autonomous driving videos!
🌐 Website: research.nvidia.com/labs/sil…
💻 Code: github.com/nvidia/asset-harv…
[1/5]
#AssetHarvester #AVSimulation #WorldModel #AutonomousDriving
Gavriel State retweeted
Robots assembling robot brain -- imagine this kind of robustness on every precision manufacturing line!
Live demo of GPU rack assembly at #NVIDIAGTC:
- end-to-end neural network (Skild Brain) finetuned with little data
- memory to perform long horizon task (placing jigs, 16 screwes, removing jigs)
- robust to disturbances and fast to set up
- no fancy sensors, just off-the-shelf arms and cameras
Nearly every system today, from energy to chips to food, is bottlenecked by scarce human capital.
We are changing that by building AI-powered industries of the future.
Check out Skild Brain robustly assembling GPU racks, a highly precise task, live at #NvidiaGTC.
Gavriel State retweeted
Excited to share the project that has surprised me the most in the last year!
Large-scale RL in simulation, no demos and no reward engineering can solve dynamic, dexterous and contact rich tasks. The learned behaviors are reactive, forceful and use the environment for recovery in ways that are extremely challenging to bake in or teleoperate!
You can play with the policies yourself to see: weirdlabuw.github.io/omnires…
And, the learned behavior transfers to real world robots from RGB camera inputs!
So what’s the trick - using simulator resets carefully! Let’s unpack (1/10)
Gavriel State retweeted
We have seen rapid progress in humanoid control — specialist robots can reliably generate agile, acrobatic, but preset motions. Our singular focus this year: putting generalist humanoids to do real work.
To progress toward this goal, we developed SONIC (nvlabs.github.io/GEAR-SONIC/), a Behavior Foundation Model for real-time, whole-body motion generation that supports teleoperation and VLA inference for loco-manipulation.
Today, we’re open-sourcing SONIC on GitHub. We are excited to see what the community builds upon SONIC and to collectively push humanoid intelligence toward real-world deployment at scale.
🌐 Paper: arxiv.org/abs/2511.07820
📃 Code: github.com/NVlabs/GR00T-Whol…
