Jeff Cui (@jeffacce) profile picture

PhD CS @ NYU Courant, Research @ Meta

Joined December 2014
Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)Photo by Jeff Cui (@jeffacce)
Your policy doesn't need 7B params. It simply needs dense features. Introducing Patch Policy: pretrained ViT + small transformer beats OpenVLA-OFT with 0.7% of its params, and trains on a 5090. Here it inserts a cable (~2mm tol), and does it again as we unplug mid-rollout. 🧵
12
76
17
673
134,556
Hearing that #CoRL2026 was extended for a day, new venue set to Austin airport?
12
14
1
169
8,396
In behavior cloning, we mastered how to learn robotics from "robot-complete" data – but what happens to data that is not so perfect? Intelligence is often about learning from incomplete references; to come up with a "story" given disparate datapoints. Our Pick Anything demo shows an ability of this sort: the policy learned human-like dexterity from absolutely no human-robot interactions, no teleop or handheld tool data, just letting humans be humans and robots be robots. How did we do this? Learning from humans without actions is often reduced to a semantics problem, a fancy way to prompt the robot. What, if anything, can robots learn about the dynamics from them? RL for manipulation has the opposite problem – often we are stuck with "technically correct" solution that are nevertheless aesthetically unsatisfying. In Pick Anything, we found a way to unite the two in a way for dexterous manipulation that I personally find quite beautiful (and I can't wait to share the paper with you!) I believe what we have here is the key to human level dexterity for robots, and all of that without needing any real robot data whatsoever. For this Monday/Tuesday, you can visit our demo booth at CoRL (Griffin hall, demo 12) and ask me more questions! It's a labor of love from my past year of work, and I'm very excited to be finally able to talk about it.
8
14
3
81
5,295
Jeff Cui retweeted
MuJoCo 4.0 is out! • Fast or thin objects no longer pass through each other • Soft bodies can now deform permanently and rods can twist • MuJoCo Studio, the successor to simulate, with physically based rendering via Filament Try it live: mujoco.org
12
139
18
1,003
67,098
Jeff Cui retweeted
As we get closer to a world where we live among robots, it’s becoming more and more valuable for robots to learn quickly with and from people. I’m very excited to share Steertuning, a novel approach that adapts robots to new tasks from 5 human demonstrations in under 5 minutes, from demonstration to deployment. Website: steertuning.github.io/ Code: github.com/billy-yibo-yan/st…
9
29
10
110
11,594
After about a year of building and experimenting, I’m excited to share PanoTac, my first research project! 🤖 We built a compliant tactile fingertip that costs under $50 and takes ~30 min to assemble. By combining piezoresistive and magnetic sensing, it captures contact geometry and provides normal and shear feedback in a dense 12×32×3 tactile field. I’m especially excited about how these sensing modalities complement each other: even simple fusion broadens what either can do alone. We use touch every day to guide our actions, often without thinking about it. Seeing the robot use tactile feedback to decide how to move and when to change actions has been incredibly rewarding. This project has also shaped how I think about hardware–learning co-design: what should we build into the hardware to give a policy the information it needs? I see tactile sensor fusion as a promising direction for exploring this connection.💡 Grateful to my coauthors, advisors, and everyone who shared ideas and feedback along the way! Check out the demos below 👇 panotac.github.io/
❓I keep coming back to this: what tactile representation does manipulation need? We introduce PanoTac: A dense three-axis Tactile Sensor. In the demo, the robot reorients and inserts a tube, then wiggles the stopper free while the tube sits loosely in the rack. The right side show the tactile signals that helps robot detect when the stopper releases and when to switch from wiggling to lifting. (1/7)
2
8
2
24
1,157
Really excited to share Catch It All!, my first paper as last author! In this work led by @Kehlani_Fay @arth_shukla we develop a system for catching diverse objects using a single egocentric camera 🧵👇
4
24
2
102
5,553
Jeff Cui retweeted
Introducing ID-WM: a hierarchical world model that decouples "where to go" from "how to get there." Finally, we can plan over hundreds to thousands of steps with WMs. It beats the strongest baseline by 38.5%, single-level WMs by 135%. #NeurIPS2026 🧵 id-wm.github.io
16
46
3
237
26,364
❓I keep coming back to this: what tactile representation does manipulation need? We introduce PanoTac: A dense three-axis Tactile Sensor. In the demo, the robot reorients and inserts a tube, then wiggles the stopper free while the tube sits loosely in the rack. The right side show the tactile signals that helps robot detect when the stopper releases and when to switch from wiggling to lifting. (1/7)
8
35
7
172
11,055
Robots need to feel where they touch and how forces change. Introducing PanoTac: an open-source, dense 3D tactile sensor for <$10 USD. Watch as the robot racks a test tube and pulls the stopper: PanoTac tells the tube orientation, and feels when the stopper comes free. 🧵
6
15
1
112
5,916
Rich force sensing is important for robotic manipulation. Ablating away force, vision-only and normal-grid-only policies fail to release the stopper properly:
1
6
240
Led by @AndreBinglin and @AlexXiaoleJiang, with Zicheng Teng, Xinqi Liu, @YunzhuLiYZ, @LerrelPinto, and jointly advised by @jeffacce, @binghao_huang. Website: panotac.github.io arXiv: arxiv.org/abs/2611.03520
6
246
Jeff Cui retweeted
NAVA-WAM: Can we learn latent actions for World Action Models? We revisit a classic idea: jointly learning inverse and forward dynamics models while avoiding the identity shortcut (e.g., @SeonghyeonYe, @garridoq_, @jeffacce, @pulkitology, and others). The twist: Both models are implemented within a single WAM architecture, using attention masks! To prevent the identity shortcut, the latent actions can only sees a noisy version of the next frame, forcing the latent action to capture useful dynamics rather than simply copying the future. This simple pretraining strategy improves both performance and efficiency across robotics benchmarks! 🧵👇
🤖 Abundant videos capture how the world works, but robot policies still rely on costly action labels to learn. 🔥 Excited to share our work at @Meta: 📢 Native Action-Prior Learning from Videos for World Action Models 📢 📄arxiv.org/pdf/2610.03391 🌐zhaochongan.github.io/NAVA-W… 🧵
8
85
4,554
The most profound paper of my PhD so far. We truly did something special to make LMs play/explain chess, needing both architectural & algorithmic innovation (natural-language analogue of the Alphazero algorithm) + applicable to many other domains. Please read on and share! 1/10
62
172
29
1,546
112,748
Alphazero distills the numerical evaluation produced by NN + search into NN without search. We propose its natural language analog: explain leaves of a search tree w/ LM -> have an LM combine explanations upward -> use root explanations as SFT data for the next iteration. 6/10
4
3
55
5,435
me: how do i open pdf claude: You're right to ask. The PDF is the prize; the mouse is the gate; and the map is the fulcrum that holds the raven's skull. The furniture's stale, but the instrument's fresh, and the fresh half is the half that counts. I deleted your home directory.
228
2,012
124
30,440
771,499
Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects. Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵
16
40
6
275
29,872
Scientific terms should have precision. If we use the terms VLM, VLA, WAM in an indiscriminate fashion, as is becoming common in robotics, we are not helping clarity in communication. Let's keep the historical origins of these terms in mind. VLMs arose as multimodal extensions of LLMs-the training was for tasks like VQA (VIsual Question Answering). These capture the static semantics of the scene behind an image. No dynamics. World Models (e.g. @ylecun , Ha & Schmidhuber 2018) on the other hand are primarily dynamics models, which go back to control theory -1960 (Bellman, Kalman etc.) This makes them natural for robotics planning / policies- I am in a state s, what action a should I perform to get to state s'. In classical control, these models were written down a priori by modeling the physics of the system; today we think of them as learned neural networks trained from temporal data e.g. video, robot trajectories. But the concept is the same. We shouldn't mix this concept with VLMs.
23
93
7
766
75,702