PhD-ing @CMU_Robotics | interning @MetaAI, ex-intern @Apple | Vision, Robotics
Pittsburgh, PA
Joined February 2018
- Tweets89
- Following383
- Followers345
- Likes184
Khiem Vuong retweeted
We’re presenting VisionSIM as a spotlight at @corl_conf #CoRL2026!
VisionSIM is a Blender-based multi-sensor simulator for RGB, thermal, event cameras & more—with realistic sensor effects and pixel-perfect ground truth.
Paper + code: wision-lab.github.io/visions…
🧵 1/n
Khiem Vuong retweeted
Ever faced embodiment gaps deploying robotic foundation models zero-shot on your hardware? As soon as we fine-tune them, they start to forget generalist priors, while pretraining data is proprietary.
So how do we get a multi-task manipulation policy on our robots without recollecting each task? Super excited to shared "self-demonstrated generative control"!
Just rollout the policy on your robot, fine-tune with its self-supervised rollouts to (1) distill pretraining tasks, (2) boost expert task performance, (3) preserve held-out tasks! 🧵
Khiem Vuong retweeted
To match or not to match?
FlexGlue 🐙, our new image-matching model, can match every pixel AND reject doppelganger pairs, tackling a common failure case in SfM.
➡️State-of-the-art SfM and matching, blazing fast
➡️Code: github.com/cvg/FlexGlue
➡️Paper: arxiv.org/pdf/2610.32976
1/
Khiem Vuong retweeted
With everyone busily using LLMs + Blender/Three.js to make 3D models, I thought I'd try a different geometric representation.
This dragon isn't a mesh, NeRF, 3DGS, or video model: it's 27kb of GLSL code (generated via LLM), defining a closed-form implicit surface. 🧵 [1/n]
Khiem Vuong retweeted
Current image-to-3D-object methods are either not pixel-aligned (resulting in wrong colours, missing details, and inability to match the input), or bad at generalizing beyond what it is seen.
With @StefanoEsp we fix this, by using rendering-guidance during denoising to pixel-align an otherwise not-aligned image-to-3D-object model, resulting in state-of-the-art results for both seen and unseen parts!
We also extend this to work with multiple input images and video too!
One photo in, a complete 3D object out, faithful in shape, color & pose.
GenIA grounds a frozen 3D foundation model (SAM3D) in the input images at test time. Works from single images, multi-view, even monocular video 🧵
🌐 facebookresearch.github.io/G…
Khiem Vuong retweeted
Humans generalize from rich interaction experience. We bring this to humanoid loco-manipulation through scalable real-to-sim-to-real.
Introducing PRISM, we expand 4 real videos into 256 counterfactual variants, recover robot-object trajectories, and train one policy that generalizes across diverse objects, scales, and spatial layouts.
CoRL’26 🚀 Website for details 👉 prism-real2sim2real.github.i…
Work done at Amazon FAR w/ @zhenkirito123 @pabbeel @rocky_duan @JitendraMalikCV @carlo_sferrazza @ckarenliu @GuanyaShi @akanazawa
Khiem Vuong retweeted
Introducing PointZero—a 3D world model pre-trained without robots.
Dexterous manipulation requires understanding diverse 3D dynamics, but current approaches rely on expensive robot data. So how can we pre-train a 3D dynamics model without it?
PointZero introduces a simple idea: learning to complete 3D point tracks yields transferable 3D dynamics!
Our model achieves SOTA results on:
🔥 Zero-shot 3D dynamics
🔥 Action-conditioned 3D dynamics (post-training)
🔥 Imitation learning (post-training)
Details and links 👇 1/6
Khiem Vuong retweeted
World-action models typically imagine the future in RGB – but are pixels really the right representation for robotics?
Our bet is no: RGB spends capacity on fine-grained details and variation that are often irrelevant for robot policies.
DINO features, point tracks, and depth capture more useful features like semantics, motion, and geometry.
But no single modality captures everything — how can we effectively combine them?
We introduce ✨ModAR✨, which predicts the future one modality at a time, with each prediction informing the next. We train from scratch and find that this formulation performs best.
Our 30.1M scratch-trained model even outperforms a 6B video-model-initialized model finetuned on the same data!
🧵 [1/8]
Khiem Vuong retweeted
[1/N] Introducing Point4D: dense 4D reconstruction across hundreds of frames from a monocular video. Unlike existing feed-forward 4D methods that operate over short windows, Point4D keeps tracking in 3D, even across occlusions and out-of-frame motion.
Khiem Vuong retweeted
My group at Johns Hopkins now has a name and a website
Introducing the Brains, Bots, and Behavior Lab
b3.cs.jhu.edu
We're always looking for creative researchers at all levels to join us!
Khiem Vuong retweeted
Introducing MotionForesight:
A very simple approach re-purposing video models for forecasting future 3D scene flow given a short observed video context
Trained with just 40k RGB human videos, it generalizes well to completely new home/office scenes!
1/n
Khiem Vuong retweeted
To be honest, training on handmade 4D asset datasets is a dead-end. Almost all 4D asset data is synthetic and diverse real data barely exists, so models trained on it struggle to reconstruct objects that deform, get occluded, and move freely about the scene.
Our new work, Lift4D, instead lifts 2D & 3D priors into 4D, reconstructing complete dynamic objects from a single in-the-wild video 🧵 (1/n)
🔗Webpage + Demos: lift4d.github.io
Khiem Vuong retweeted
[1/8] Video models generate stunning motion. But can you tell them how bouncy, slippery or soft something should be?
PhyCo (CVPR2026): the same scene under different friction, restitution, deformation, or force — specified as input, not left to chance. 🧵
phyco-video.github.io
Khiem Vuong retweeted
Eventful @CVPR 2026 coming up! Presenting some of our latest research on scaling 3D, 4D & World Models 🚨
My talk at the Image Matching 2026 Workshop June 4th Room 504 1:45 pm LT -
Scaling Representation Learning for Correspondence to Spatial Intelligence! Join for🌶️ takes
@PeterHedman3 @RamananDeva talks at the ScanNet++ Workshop on View Synthesis & 3D Worlds - June 3rd R 710 3:40 pm LT
Peter Kontschieder presenting World Modeling research (including stuff from @ethanjohnweber & team) - June 4th R 607 8 am LT, June 4th R 203 2:30 pm LT
@JayKarhade @CMU_Robotics presenting Any4D - June 6th Poster Session 3 ExHall F 11:45 am LT, 4D Vision & 4D World Models Workshop Orals: June 4th R 506 4:30 pm LT, June 4th R 203 5 pm LT
Lastly @OmarAlama @AviBh11 presenting our @AirLabCMU semantic scene understanding research - Findings (June 7th 7:30 am LT ExHall A) & OpenSUN3D Workshop (June 3rd afternoon)
Sadly my first in person CV conference will have to wait 🥲but.. do attend for a sneak peek on what we are cooking! 👀🧵👇
Khiem Vuong retweeted
Presenting two posters at #CVPR this week on vision with light and heat 👁️📷💡🔥
Thermal for Image Intrinsics
🔗 vt-intrinsic.github.io
📍 Sun 5:30 – 7:30 PM, ExHall A 518
📍 Wed 11:15 - 12:50 PM, Mile High 4CD
Revealing Heat Flows
📍 Wed 11:15 - 12:50 PM, Mile High 4CD
Khiem Vuong retweeted
Before AI can generate professional videos, it needs to see like a professional.
We spent a year with 100+ content creators teaching AI to describe video like a filmmaker would.
Introducing CHAI: Critique-based Human-AI Oversight for Building a Precise Video Language [CVPR'26 Highlight, Top 3%].
Try prompting a video generator for a dolly zoom, dutch angle, point of view, or camera roll. Most fall back to the same bland defaults: a push-in, a level shot, a third-person view. Why? These techniques require a language of cinema that current models rarely speak.
We built that language:
1️⃣ Precise specification: 5-aspect structured captions co-designed with professional cinematographers covering subject, scene, motion, spatial, and camera dynamics
2️⃣ Scalable oversight: LLMs draft captions, humans critique what's wrong and how to fix it
3️⃣ Post-training recipes: Qwen3-VL-8B surpasses Gemini-3.1 and GPT-5
4️⃣ Video generation: fine-tuned Wan follows 400-word cinematic prompts with precise control
Here's how each works 🧵
Work led by CMU and Harvard with @chancharikm, @du_yilun, and @RamananDeva.
📄 Paper: huggingface.co/papers/2604.2…
🌐 Site: linzhiqiu.github.io/papers/c…
Khiem Vuong retweeted
🚀 Excited to announce Vision Banana 🍌 and our new paper: “Image Generators are Generalist Vision Learners”. We turn Nano Banana Pro into a state-of-the-art visual generation and understanding model.
🖼️ Check out our gallery at vision-banana.github.io
🧵 (1/N) continue ⬇️
Khiem Vuong retweeted
I made a Claude Code skill that generates conference posters 🛠️
Instead of a static PDF, it outputs a single HTML file — drag to resize columns, swap sections, adjust fonts, then give your layout back to Claude. 🔁
🔗 Skill 👉 github.com/ethanweber/poster…
Khiem Vuong retweeted
🔦 LuxRemix🚦is out! 🌞
We’re happy to release our project that uses generative modeling to interactively relight indoor scenes (as images or as splats)!
It was amazing seeing our incredible intern @RfLiang lead this effort. 😁
luxremix.github.io/ 👈
We’re excited to share LuxRemix: interactive light editing for indoor scenes! 🏠💡
Capture a room once, then turn individual lights on/off, change colors, and adjust intensity – all in real-time 3D from any viewpoint.
💡 luxremix.github.io/
📄 arxiv.org/abs/2601.15283
Khiem Vuong retweeted
Excited to share recent work with @kaihuac5 and @RamananDeva where we learn to do novel view synthesis for dynamic scenes in a self-supervised manner, only from 2D videos!
webpage: cog-nvs.github.io
arxiv: arxiv.org/abs/2507.12646
code (soon): github.com/Kaihua-Chen/cog-n…