Xingang Pan

We developed PhysLDM, a high-quality neural simulator based on latent diffusion models (LDMs)! Highlights: - A spatiotemporal VAE that compresses dynamic 3D deformable meshes into a compact latent space. - An LDM that models both global motion and local deformation. - Fully differentiable, enabling challenging inverse problems! We hope PhysLDM will make 3D simulation more accessible and ML-friendly! Awesome work by Yu Zhang @yucrazing ! 🌐 Project: yucrazing.github.io/physldm 📄 Paper: arxiv.org/abs/2610.07609

Recent works like GenCeption show that video diffusion models (VDMs) are not just powerful generators, but also strong visual learners. Along similar lines, our work ViDiHand (NeurIPS 2026 Spotlight) shows that pretrained VDMs are surprisingly effective for 3D hand motion capture! Our insight: The ability to generate coherent videos suggests that VDMs implicitly learn motion, structure, and correspondences, which can be leveraged for perception. ✋ ViDiHand reconstructs accurate and temporally smooth 3D hand motions from monocular egocentric videos, even under severe occlusions and complex hand-object interactions. ✅ No hand detector, no motion infilling, no test-time optimization. 📈 ViDiHand substantially outperforms prior methods across three benchmarks. We believe video generative models hold great potential as foundation models for 3D perception and embodied data collection! 📄 Paper: arxiv.org/abs/2606.30308 🌐 Project & demos: vidihand.github.io/ 💻 Code (coming soon): github.com/NTUYWANG103/ViDiH…

🎬 Can video models remember the past with 85% fewer memory tokens? Introducing DeCoPrune, a training-free token pruning method for autoregressive video diffusion! 💡 Key insight: Denoising difficulty reveals what is worth remembering. We retain tokens that are hard to predict from context and prune redundant ones, with no extra model forward passes. 🚀 On LingBot World v2: 85.43% fewer historical KV tokens 4.14× faster continuation generation Long-range visual recall close to FullKV We also introduce CMBench, a benchmark for long-range visual memory in video generation. Great work by @zeqi_xiao and @qingle_liu! 📄 Paper: arxiv.org/abs/2609.39096 🌐 Project: decoprune.github.io 💻 Code: github.com/DeCoPrune/CMBench 🤗 Dataset: huggingface.co/datasets/Aora…

We just released a high-quality, penetration-free cloth simulation dataset used in ClothTransformer. 👕 🤗huggingface.co/datasets/YuCr… 493K frames of cloth simulation (2,056 trajectories, ~33GB) across collision, animated human, and robotic manipulation scenarios.

Natural images often already implicitly contain depth information — hidden in bokeh effects. Can we leverage the rich depth cue widely exist in natural images for depth estimation? We explore this in our recent project BokehDepth (ICML 2026). - Stage 1: A generative model produces calibrated bokeh stacks from the input image. - Stage 2: The bokeh stacks are integrated into a depth prediction model to estimate depth. We believe it highlights bokeh effects as an important and effective complementary cue for monocular depth estimation. 🌐 Project page: fogradio.github.io/BokehDept… 📄 arXiv: arxiv.org/abs/2512.12425 👨‍💻 Code: github.com/fogradio/BokehDep…

Transformers have succeeded in modeling phenomena traditionally associated with computer graphics, such as 3D visual effects (e.g., RayZer) and rendering processes (e.g., RenderFormer). A natural question is whether they can also tackle the challenging task of cloth simulation. We introduce 👕𝗖𝗹𝗼𝘁𝗵𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿, a Transformer-based method that reformulates cloth simulation as autoregressive next-state prediction in a learned latent space. It handles diverse scenarios under a single model, with 4-9x lower error than prior SOTAs: • Body-driven garments • Robotic manipulation • General cloth–object collisions We believe it highlights the potential of Transformer-based autoregressive models as a powerful alternative to conventional simulation approaches. This work is mainly led by my student Yu Zhang @yucrazing 🌐 Project page: yucrazing.github.io/clothtra… 📄 arXiv: arxiv.org/abs/2605.27852

I'll give a talk about Video World Models🌎 at #CVPR2026 🗓️Thu, Jun 3, 11:05 AM, Room 705/707: 1st Workshop on Video World Models videoworldmodel-workshop.git… Unfortunately, due to a visa delay (still no visa after 50 days 😅), I'll be presenting virtually this year. Nevertheless, looking forward to any discussions!

Introducing StoryMem — a memory-augmented framework for multi-shot long video storytelling. StoryMem carefully injects compact memory into the generation process with minimal overhead, enabling: • Cross-shot consistency • Smooth transitions • Narrative coherence across minutes-long videos Awesome work by Kaiwen @sze68zkw Project: kevin-thu.github.io/StoryMem… arXiv: arxiv.org/pdf/2512.19539 Code: github.com/Kevin-thu/StoryMe…

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: xizaoqu.github.io/video4spat… arXiv: arxiv.org/pdf/2512.03040

Introducing 📦𝗔𝗿𝘁𝗶𝗟𝗮𝘁𝗲𝗻𝘁🔧 (SIGGRAPH Asia 2025) — a high-quality 3D diffusion model that explicitly models object articulation, paving the way for richer, more realistic assets in embodied AI and simulation: – Generates fully articulated 3D objects – Physically plausible joints & motion – High-fidelity 3D Gaussian appearance – Supports generation from a single real image arXiv: arxiv.org/pdf/2510.21432 Project: chenhonghua.github.io/MyProj… Code (coming soon): github.com/chenhonghua/ArtiL…

Cool work that connects the idea of volume rendering with image diffusion!