Computer vision, SSL, Robotics PhD student at AMI Labs and INRIA Paris (Willow team) - Supervision : Yann LeCun and Jean Ponce. Ex Meta FAIR
Paris, France
Joined February 2024
- Tweets190
- Following334
- Followers1.1K
- Likes441
Pinned Tweet
Hierarchical world models, learned end-to-end from pixels!
Excited to introduce H-JEPA: stacked JEPAs, each level in its own embedding space, planning top-down.
Visual AntMaze: 3 levels lift success from 18% to 73% 👇
w/ @kevinghstz, Michael Rabbat, @ylecun, @randall_balestr
Huge work of the past years from @artemZholus, congrats 🫡
I will definitely try this unified multi-embodiment large scale robotic training setup !
Introducing RoboJEPA, the first scaling law for multiembodiment world models.
We scaled JEPA predictors up to 8B params, showed scaling laws extrapolate and correlated compute scaling with planning performance.
We release all checkpoints and the code.
robojepa.github.io
Basile Terver retweeted
Introducing H-JEPA: the first end-to-end learned hierarchical world model for long-horizon visual planning!
Built atop SIGReg/LeWM, H-JEPA provides a stable training recipe where semantic abstractions naturally emerge across levels...
Paper+Code: arxiv.org/abs/2610.06805 🧵👇
Abstract embeddings also make a better planning cost: distances grow smoothly along corridors.
Simply scoring a level-1 rollout in a higher-level space already improves AntMaze success, with no upper-level planner at all.
We score plans offline by how closely the 3D end-effector path follows the expert's (Fréchet fidelity: 0% = arm stays still, 100% = expert path).
A learned 2nd level beats flat LeWM+IDM and a shared-space hierarchy, at lower planner compute.
Everything is open: code to train, plan and reproduce every result, 93 checkpoints (4 sim envs + DROID) and the datasets.
📄 arxiv.org/abs/2610.06805
🌐 h-jepa.com
💻 github.com/kevinghst/H-JEPA
🤗 huggingface.co/jepa-world-mo…
What do higher levels learn? They drop fast, unpredictable detail and keep the slower state.
AntMaze: the leg pose fades with depth, the position stays.
FourRoom: the random distractor fades, the agent remains.
This works best when factors move at well-separated timescales.
Planning runs top-down. The top level plans toward the goal, and its predictions become subgoals for the level below. The lowest level outputs the actions.
Each planner scores its rollout against its subgoal, in the upper level's embedding space.
Our idea is to stack JEPAs. Each level encodes the one below into its own embedding space and predicts farther ahead, from learned macro-actions.
Every level keeps the simple 2-term loss (prediction + SIGReg anti-collapse), and all levels train jointly, end-to-end.
Flat JEPA world models plan in a single embedding space. That space has to serve both precise control and long-range planning.
But fine details (like the ant's leg poses) are hard to predict far ahead, and they don't tell you how close you are to the goal.
Basile Terver retweeted
[7/n] We appreciate the excellent prior work:
@yingwww_ on projection for lower-dimensional embeddings
@randall_balestr on SIGReg
@BasileTerv987 on extensive JEPA world-model ablations
@GaoyueZhou on DINO-WM
@lucasmaes_ @quentinlldc on LeWM
And of course, @ylecun — JEPA FTW!
Basile Terver retweeted
If you missed our opening remarks:
- 4th world modeling workshop: Aspen, CO, February, world models for physics, wmw-aspen.github.io
- 1st world modeling conference: Bay Area, CA, May, icwm.cc
In collaboration with @ylecun @LambdaAPI @amilabs and more TBA!
Basile Terver retweeted
Introducing LpWM: A Case for Sparse Representations in World Models
Dense Gaussian representations are a choice, not a requirement. We find that sparse representations can make latent dynamics easier to model for planning.
📄arxiv.org/abs/2608.22764
💻github.com/YilunKuang/lpworl…
Basile Terver retweeted
Chinchilla writes loss as a sum of two independent terms: model size and data never interact. Kaplan's original law coupled them.
But interaction has an impact, dropping it bends predictions exactly where you extrapolate. Skaling, our new law, puts it back with one exponent. 1/8
Basile Terver retweeted
Great work from @KrunoLehman and @webalorn !
WaiT introduces frequency-aware flow matching: generate coarse structure first, then progressively denoises high-frequency details with wavelet-specific schedules. A simple change which improves image and video generation
WaiT for the Signal: Simple Frequency-Aware Flow-Matching.
We show how to natively incorporate a fundamental property of images directly into diffusion models; setting a new pixel-space SOTA on ImageNet, while reducing compute.
📄 arxiv.org/abs/2607.28760
Full breakdown below👇
Absolutely huge work by the one and only @PierreChambon6 !
125 pages of reading for my holidays 🥰
🧵How to combine code correctness and efficiency objectives in online RL?
-> we break the correctness-efficiency Pareto frontier!
+125% relative improvement in CWM32B compared to standard RLVR (13.7 ->30.9 pass@1@top30%)
+150% with Qwen32B
Paper: arxiv.org/abs/2607.25970