Jing Tan (@jingtan_) profile picture

phd student @ mmlab, cuhk; gen AI

Hong Kong SAR
Joined June 2021
Photo by Jing Tan (@jingtan_)Photo by Jing Tan (@jingtan_)
🚀Excited to introduce 𝗜𝗺𝗮𝗴𝗶𝗻𝗲𝟯𝟲𝟬, the 1st framework that lifts standard videos into 360 videos with rich and structured motion, unlocking dynamic scene experience from full 360 degrees. 🎥🌍 - Webpage (w/ VR): ys-imtech.github.io/projects… - Arxiv: arxiv.org/abs/2412.03552
3
53
4
192
16,481
Jing Tan retweeted
Talk2Move Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes
5
16
72
10,830
Jing Tan retweeted
💡RelightVid: Temporal-Consistent Diffusion Model for Video Relighting. A temporally consistent video relighting framework. By extending IC-Light with temporal layers and multi-modal conditions, it enables coherent and flexible video relighting.🚀aleafy.github.io/relightvid
2
8
13
1,810
🎉 Excited to introduce IDArb! 🎉 Our method can predict plausible and 𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝘁 geometry and PBR material for 𝗮𝗻𝘆 𝗻𝘂𝗺𝗯𝗲𝗿📷 of input images under 𝘃𝗮𝗿𝘆𝗶𝗻𝗴 𝗶𝗹𝗹𝘂𝗺𝗶𝗻𝗮𝘁𝗶𝗼𝗻𝘀☀️ ! Webpage: lizb6626.github.io/IDArb/
2
25
75
11,419
Imagine360 is a new method that can turn any video into 360° video 👀
7
35
1
186
20,411
Thanks for sharing! 😃 Please do visit our project page for immersive VR interaction experience. Code comes in next month: github.com/YS-IMTech/Imagine…. Stay tuned!
Imagine360: Immersive 360 Video Generation from Perspective Anchor Check out the project page, highly recommended❗️ Abstract (excerpt): We introduce Imagine360, the first perspective-to-360∘ video generation framework that creates high-quality 360∘ videos with rich and diverse motion patterns from video anchors. Imagine360 learns fine-grained spherical visual and motion patterns from limited 360∘ video data with several key designs. 1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for 360∘ video generation, with motion module and spatial LoRA layers fine-tuned on extended web 360∘ videos. 2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres. 3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames. Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art 360∘ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive 360∘ video creation.
6
626
Excited to see this incredible progress in 3D panoramic scene generation! 🚀 We also have a preliminary attempt in this direction. Take a look and share your thoughts! 🙌 - LayerPano3D Paper: arxiv.org/abs/2408.13252 - Project page: ys-imtech.github.io/projects…
We’ve been busy building an AI system to generate 3D worlds from a single image. Check out some early results on our site, where you can interact with our scenes directly in the browser! worldlabs.ai/blog 1/n
2
11
1,664
🔥Text to Explorable 3D Panorama🔥 #LayerPano3D is a framework for full-view, explorable panoramic 🏝️3D scene generation🏝️ from a single text prompt. - Project: ys-imtech.github.io/projects… - Code: github.com/YS-IMTech/LayerPa… - Video: youtube.com/watch?v=dXvoFRbH… @Gradio demo coming soon :)
LayerPano3D Layered 3D Panorama for Hyper-Immersive Scene Generation discuss: huggingface.co/papers/2408.1… 3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.
2
18
2
81
14,998
Thanks a lot for sharing! We will soon open-source our code at github.com/YS-IMTech/LayerPa…. Stay tuned!
LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation Paper: arxiv.org/abs/2408.13252 Project: ys-imtech.github.io/projects… Youtube: youtube.com/watch?v=dXvoFRbH… Make sure to visit the project page. It is beautiful! Metho ⬇️ 1 | 2
2
19
2,973
Thanks for sharing our work! LayerPano3D generates 360x180 full view, explorable panoramic 3D scene from a single text prompt. More examples are available in our project page: ys-imtech.github.io/projects…
LayerPano3D Layered 3D Panorama for Hyper-Immersive Scene Generation discuss: huggingface.co/papers/2408.1… 3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.
2
10
1,515
Jing Tan retweeted
LayerPano3D Layered 3D Panorama for Hyper-Immersive Scene Generation discuss: huggingface.co/papers/2408.1… 3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.
3
26
5
173
41,913
We are excited to introduce LayerPano3D, a novel framework to generate full-view, explorable panoramic 3D scene from a single text prompt! More examples are in our project page. ✨Project: ys-imtech.github.io/projects… ✨Paper: arxiv.org/abs/2408.13252 ✨Code: github.com/YS-IMTech/LayerPa…
5
31
3
156
11,411
Want to generate camera controllable human videos like a real movie clip? Try our HumanVid dataset and a baseline model combined by AnimateAnyone and CameraCtrl. Project Page: humanvid.github.io/ Paper: arxiv.org/pdf/2407.17438 Data and code coming soon.
7
36
2
114
11,772
WIP #b3d
28
267
10
2,461
259,882
I will attempt to explain the basic idea of how diffusion models work! ... in only 15 tweets! 😲 Let's get started ↓
79
667
65
3,753
If you are desperate for space in your #NeurIPS2022 paper, eg because reviewers asked to include parts of your rebuttal into the main paper, one possibility is to enclose some $$ ... $$ equations with \textstyle{ ... }. This will typeset them as if they were inline $ ... $:
2
7
75
Read older papers (before 2000). Many important insights already appeared decades ago. Don't just read from certain groups. This tends to miss the big picture. Read from broader fields. Read statistics, robotics, physics, applied math, graphics, PL, databases, linguistics, HCI.
How to keep track with the literature? Getting started with research but overwhelmed by THOUSANDS of papers each year? How could one stay sane keeping track of the literature? 😱 Here are some tricks I found useful. 🧵
3
12
3
113
Just realized that I now have 40 (!) twitter threads! 🤩 I organized them into "Awesome Tips": github.com/jbhuang0604/aweso… Really appreciate for all your comments, critics, likes, and retweets. Hope these threads are useful! What should I write next? 🤔
3
43
3
238