Tan Lip-Bu Professor in AI @ NTU | Director, MMLab@NTU | Generative AI, world models & embodied AI
Singapore
Joined November 2010
- Tweets1K
- Following771
- Followers3.9K
- Likes2.4K
Pinned Tweet
Meet MatAnyone-E: high-quality video matting, much faster. ✂️
Compared with MatAnyone 2 at 4K:
⚡ ~7.4× faster inference
💾 Just 7.6% of the peak GPU memory
✨ Comparable matting quality
Runs at 73 FPS at 1080p and 29 FPS at 4K on a single NVIDIA H200.
Project: shuyun-wang.github.io/projec…
Paper: arxiv.org/abs/2610.35724
Code (coming soon): github.com/Shuyun-Wang/MatAn…
Chen Change (Cavan) Loy retweeted
Wish you could change focus after taking a photo? 📸
AnyBokeh lets you refocus and adjust background blur from a single image, even when it’s already partly out of focus. Led by my student @itsmag11
Project: itsmag11.github.io/AnyBokeh/
Paper: arxiv.org/abs/2606.31959
Code: github.com/itsmag11/AnyBokeh
Wish you could change focus after taking a photo? 📸
AnyBokeh lets you refocus and adjust background blur from a single image, even when it’s already partly out of focus. Led by my student @itsmag11
Project: itsmag11.github.io/AnyBokeh/
Paper: arxiv.org/abs/2606.31959
Code: github.com/itsmag11/AnyBokeh
How does MatAnyone-E save GPU memory? By being selective about where to look—and what to look up. 🧠
Dense matching compares every current-frame feature with features stored from past frames. At 4K, those comparisons become expensive.
MatAnyone-E first takes a low-resolution look, then:
• Refines only uncertain regions.
• Searches only the most relevant memory locations.
• Reuses the coarse results elsewhere.
In this demo, just 5.9% of query locations need refinement, using 15% of memory candidates—skipping 99.1% of high-resolution matching pairs.
🎥 A long video. One coherent 3D scene.
Meet HiP3R: piecing together local 3D reconstructions to keep the scene consistent across thousands of frames. Watch the demo and explore the details 👇
Project: kkakkkka.github.io/HiP3R/
Paper: arxiv.org/abs/2610.31635
Code: github.com/HiP3Rs/HiP3R
Why is reconstructing a long video difficult?
Short sections can look good on their own, yet drift apart when stitched together. Revisiting the same place may produce misaligned geometry.
The challenge is making all these local reconstructions agree.
Has GPT-6-Astra Achieved Embodied Intelligence?
Across 12 benchmarks, access to training demonstrations and up to 10× the execution budget push several tasks close to 100% success.
How does this translate to real robots? Watch our demos across 8 robot configurations to see its capabilities and remaining challenges.
Paper: huggingface.co/papers/2610.2…
Project & demos: sensenova.ai/has-gpt-6-astra…
GitHub: github.com/OpenSenseNova/Sen…
What if you could step inside a van Gogh painting?
We used Mundo to imagine new viewpoints from a single painting.
Take a look at this short demo.
Project: mundo-world-model.github.io/
Blog: huggingface.co/blog/Luo-Yiha…
🌍 Introducing Mundo, led by my student @onelineluo: a customized world model that turns images of a place into an environment for exploration, editing, and simulation.
Explore new viewpoints, change objects and physical properties, or simulate robot navigation and manipulation — all within the captured environment.
The key is an explicit, editable world state: code specifies actions, physics computes their consequences, and a video model renders the outcome using the input images as visual context.
Project: mundo-world-model.github.io/
Blog: huggingface.co/blog/Luo-Yiha…
Code coming soon!
Chen Change (Cavan) Loy retweeted
1/6 Ropedia Super Week Day 2
How can we turn first-person human experience into structured 3D spatial intelligence?
For Day 2 of Ropedia Super Week, we're introducing two complementary research projects: HoliWorld and EgoXperience. Both explore how we can make first-person experience more useful for understanding the physical world.
Chen Change (Cavan) Loy retweeted
We developed PhysLDM, a high-quality neural simulator based on latent diffusion models (LDMs)!
Highlights:
- A spatiotemporal VAE that compresses dynamic 3D deformable meshes into a compact latent space.
- An LDM that models both global motion and local deformation.
- Fully differentiable, enabling challenging inverse problems!
We hope PhysLDM will make 3D simulation more accessible and ML-friendly!
Awesome work by Yu Zhang @yucrazing !
🌐 Project: yucrazing.github.io/physldm
📄 Paper: arxiv.org/abs/2610.07609
Chen Change (Cavan) Loy retweeted
1/5 Ropedia Super Week Day 1
How can robots learn to predict the physical world before acting in it?
Today, we're kicking off Ropedia Super Week with Ropedia W0, a world model exploring how human experience can help robots learn to predict what will happen next.
Here's what we're building.
This motivates benchmarks grounded in human needs, without in-domain demonstrations, and with a clock that keeps ticking.
Technical report: arxiv.org/abs/2610.22471
Project page: sensenova.ai/has-gpt-6-astra…
Code: github.com/OpenSenseNova/Sen…
Chen Change (Cavan) Loy retweeted
Ropedia Super Week | Oct 19–23
6 breakthroughs. 5 days. A showcase of how first-person human data can advance embodied AI, from spatial reasoning to action and control.
Here's the lineup. ↓
Mon: Ropedia W0, our latest robot world model.
Tue: HoliWorld + EgoXperience, our spatial data pipeline & full-stack data-to-model framework.
Wed: S-Agent 2.0, our evidence-driven spatial reasoning agent.
Thu: DONIC, our generalist controller for two dexterous hands.
Fri: Ropedia A0, our code-driven robot self-improvement framework.
Tomorrow, we'll take a closer look at our projects.
Follow Ropedia for a first look.
Chen Change (Cavan) Loy retweeted
DINOv2 has never seen a caption, and Qwen3 has never seen an image. We still aligned their embedding spaces without a single image-caption pair.
It even works when the images and the captions come from different datasets.
Project page: dominik-schnaus.github.io/un…
⬇️
Chen Change (Cavan) Loy retweeted
The surprising effectiveness of video diffusion models for hand motion reconstruction
Blog: huggingface.co/blog/Xingang-…
Chen Change (Cavan) Loy retweeted
Our ancestors picked up a rock. They saw a hammer, a blade, and a way to survive. Humans didn’t conquer the world with bare hands -- we did it with tools. 🛠️
Behind every tool we reach for is a tiny act of intelligence: "What do I need? What should I use? What do I do next?"
It's not just knowing what something IS. It’s imagining what it could DO. Can current AI models reason the same way?
Meet #EgoTools , collab with @ropedia_ai.
🔗 ropedia.github.io/egotools/