Read the full results in our first blog post: Empirical results fine-tuning π0.5 on a real manufacturing task. dream-machines.eu/blog/pi05-…

LLMs can already build a Blender scene from a room video. So is 3D vision doomed? We tried the opposite: hand the vision models to GPT-6 as tools (Pi3X for geometry, SAM3, GVHMR/Kimodo for people). It turns a room video into a better interactive sim. 👇 kevinxu02.github.io/real2sim…

In-context learning for robotics is here. - Long-horizon tasks over 10 minutes long - Never seen during pre-training - Prompted with one video, no fine-tuning We are building intelligence from the foundations up, not from the top down.

Our new work with Jayjun explores sim-first approach to pre-training and post-training. There is a lot to squeeze out of simulations and we still don't know the ceiling and this work is a step in that direction. Ideally, you want to work on problems that are compute bound or alternatively figure out a way to turn your problems into compute bound problems and training in simulations at scale is a compute bound problem with lots of opportunities. This work also explores training visuo-tactile policies in simulation. @SharpaRobotics hand has 5 tactile sensors at the finger-tips and we simulate them in IsaacLab to provide rich interactive data in the loop. We transferred policies zero-shot on two different arm-hand combinations and importantly we didn't tune any hyperparams associated with RL training or reward. Many tasks can be defined as goal reaching problems where the robot has to reach for the object, lift, transport and bring it to a desired goal and this is embodiment agnostic and in simulations, you can both generate and verify goals.

Your generalist policies might be smarter than you think! When orchestrated with a frontier reasoning model, we extract substantial performance improvements, vastly outperforming the best e2e policies or CaP. lianegalanti.github.io/Pigey… 🧵:

Kimi K3 may be an important inflection point for AI. Potentially negative for Anthropic and OpenAI while being net positive for essentially every other company in the world. I mean that very literally. Although the real “Sputnik moment” would be an open-source frontier model that was also token efficient unlike Kimi K3 which is 50-70% more expensive to run than GPT 5.6 per Artificial Analysis. Rationale:   A world where there are only 2-3 dominant frontier labs with 90% inference margins is net negative for every other layer while being awesome for those 2-3 labs. Those labs would become monopsonies for power, data centers, semiconductors and hyperscalers and would obviously vertically integrate over time into all those layers while also completely subsuming the application/software layers.    Anything that lowers margins and increases competition at the model layer is good for every other AI layer: power, semiconductors, hyperscalers, neoclouds and yes even software.   This is why Jensen is so supportive of open-source. An open-source model requires the *exact* same amount of compute to run as a closed frontier model of similar size and architecture. Kimi K3 is roughly the same price as GPT 5.6 Terra on a per token basis, which actually suggests that it is less computationally efficient as I am sure that GPT 5.6 is priced to a higher margin than K3. And given that K3 is a token wastrel, i.e. token inefficient, it is significantly more expensive per task than GPT 5.6 and Grok 4.5, which are much more token efficient. Cost per token and token efficiency (i.e. intelligence density per token) are the drivers of intelligence per unit of cost. The winning AI companies will be those that offer the most intelligence per $ over time.   Lower margin % at the model layer = more margin $ at every part of the infrastructure layer and is a godsend for software. This can happen either through open-source models like K3 at the frontier *or* having a vertically integrated model company like Meta, SpaceX or Google at the frontier. Both outcomes result in a lower margin % at the model layer as vertically integrated model companies don’t really care where the margin $ come from. This is why it was so painful for OpenAI and Anthropic when Google was right there with them from a model competitiveness perspective and why Grok 4.5 and Muse 1.1 were just as important as Kimi K3. 
The reason Kimi K3 is only *potentially* negative for Anthropic and OpenAI is 1) the @ericvishria point that the Claude and ChatGPT products and harnesses may be more important than their models today and 2) the hypothesis that they have much more advanced model checkpoints internally that are already being used for RSI. In the latter scenario, reaching RSI even a few months ahead of other labs might be enough to cement a permanent lead. Time will tell on both points. And likely fairly quickly. Caveat would be that since Kimi K3 is not token efficient and thereby actually more expensive than ChatGPT 5.6, we may need to see a more token efficient open-source model at the frontier or see Grok 5/Composer 4/Muse 2 at multiple points on the Pareto frontier for this potential risk to Anthropic and OpenAI to play out. And I am sure they will both vertically integrate as quickly as possible while continuing the product/harness strength they have shown over the last 8 months.

Video is the language of robotics. My lab at MIT is just open sourced this project that powers any robot for any task with a video model. Congratulations to Lester and the team for leading the project!

New work with @nvidia: evaluating robot policies entirely inside a world model. The policy acts, the model imagines the consequences, and the imagined evals predict real-world results. 🧵 real vs world-model rollout side by side📷

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, @jiazhi_yang2024 in RISE does RL in a compositional world model. Learn more ->

The bottleneck of frontier robotics isn’t compute, labeling, or the models themselves. It’s data collection. While language models scaled effortlessly on open internet text, robotics requires physical trajectories, motor torques, and tactile forces that cannot simply be scraped from a webpage. Every token has to be fought for. Here is a breakdown of the 7 data types shaping the industry today, each representing a trade-off between collection cost and action-label purity: 1. Real Teleoperation (AgiBot World, DROID). Collected by humans guiding hardware, it scales linearly with human hours. 2. Low-cost Capture (Mobile ALOHA, UMI handheld). It drives collection cost down while keeping real physics, though it introduces an embodiment mapping problem when transferring human hand actions to robotic joints. 3. Fleet / Deployment Data (Tesla Optimus, Figure). These are trajectories from robots already working in the field. Tesla is betting its automotive fleet infrastructure transfers to Optimus. It generates powerful, real edge cases, but requires scaled deployment. 4. Simulation (NVIDIA Isaac Sim, Genesis). While offering near-infinite scale, the sim-to-real gap still struggles to model contact-rich dynamics like slipping, twisting, friction. 5. World-Model Synthetic (NVIDIA Cosmos 3). NVIDIA just shipped Cosmos 3, which natively outputs action trajectories, not only video pixels. If a world model can accurately simulate the laws of physics natively, it reduces the need for manual teleop data drastically. 6. Egocentric video (Ego4D, Meta’s Project Aria). First-person human video captured with head-mounted rigs. Far more scalable than teleop and closer to a robot’s own viewpoint. Still carries no robot action signals on its own. 7. Internet video (Youtube, TikTok). Maximum scale, lowest cost, effectively free. It captures the widest range of objects, tasks and physical situations, but with zero action labels and (mostly) a third-person viewpoint. Collecting data is only the step one. The next great execution challenge is engineering a coherent training recipe that can blend these heterogeneous data sources into a single model.