Jimmy Wang

We introduce 3D World Simulator, an interactive world model that runs at 50+ FPS on an RTX 5090. Our model predicts in a unified 3D representation space, capable of generating: • Novel view appearance • Multi-view consistent RGB-D • 3D geometry • Spatially aligned tactile signals Play with the world model now at jwang771.github.io/3dworldsi…!

Wow, glad to see you think this is the future of robotics simulation! Since I don't have the vantage point to see what internal sim2real/r2s2r pipelines look like, I don't have the experience to make a similar claim myself. That said, I do believe in the potential of the method! There are some fun challenges I can foresee to get there: • Architecture: our method is voxel-based and it thrives when modeling dynamics in a compact workspace. However, modeling larger scenes invokes the perennial tradeoff between voxel resolution and processing speed. Maybe with good sparse voxel kernels this is ok! We're also working on an egocentric version of this attached to the robot base for room-scale simulation of mobile manipulation, which I'm also excited about! • Availability of data: I'm confident our model will work given the *right* data, but it might take a lot of effort to collect this data since our model's advantages only appear when we have more complete scene captures. Another path I see is using sparse reconstruction methods like Atlas from @theworldlabs, which could enable training from a lot of the partial data that exists. Learning a scene prior from simulation would also be interesting. • Reliability: let's say that the ingredients come together and it works. For it to be truly useful, there's still the fundamental problem of going ood and reward hacking if used for RL when visiting infeasible states or due to data bias. Perhaps since our method uses a native 3D representation, we could add/learn constraints or privileged knowledge that reflect the physical world. I do think that there's a much higher hope in 3D compared to 2D though!

Thanks Stone! I honestly do believe that this could work given the right design choices and right data. The last one is a big if though. The advantage of our method scales with the information content in the data (e.g. from extra cameras), which may make things more cumbersome. Perhaps tools like sparse reconstruction methods (Atlas?), or learning privileged priors from sim could help! However, I do wonder if this will ever get robust enough to do RL and get legitimate signal. Since its native representation is in 3D, it could be easier to enforce constraints and avoid adversarial reward hacking.