Ivan Skorokhodov

It's more or less obvious that LLMs can control robots perfectly well if you give them enough time, but "enough time" is exactly the problem. Imagine a human controlling a robot arm via text commands "move left 0.1 cm; rotate the gripper's yaw angle 3 degrees" vs directly tele-operating it with their own hand: the former is fundamentally limited because the high-bandwidth continuous control stream (e.g., humans can generate thousands of "micro-actions" per second) is being squeezed through a low-bandwidth text interface. So there is no way of speeding up this approach and the stamp "20.7x + thinking pauses removed" would stay there for quite a while until some breakthroughs in LLM acceleration would happen — and that should be something bigger than whatever we've seen in the past ~5 years. Otherwise, this would result in economically unviable execution in practice and inability to perform low-latency tasks at all (e.g., cooking, humanoid walking, etc.): very soon you will find yourself distilling such an approach into some action-prediction model (which is actually quite interesting) or replacing the text tokens with some hidden representations — and this will be getting closer and closer to a vanilla robotics pipeline.

I accidentally learnt that I was an Outstanding Area Chair at CVPR. I didn’t even know that it’s a thing, turns out those conference booklets are actually useful

Oh gosh, they've just raised $47B 3 months ago. VCs are now subsidizing tech bros token maxxing

If you're going to CVPR this year — we're hosting a small Rhoda AI evening party on Thursday, June 4th. Come say hi if you're into robotics or generative models (or just good food): luma.com/xqrmps2b

Meta Superintelligence Labs was able to achieve 90% MFU on the H100/H200 GPU architecture for end-to-end distributed Transformer training. It is ~2x faster than any existing crazily-optimized large-scale training workflow of any bigtech lab, and something that Nvidia itself was unable to achieve even for pure Hopper GEMM kernels in 4+ years of relentless tuning. This basically closes the gap between theoretical and practical GPU FLOPs utilization* and is pure gains: Llama-4 would have cost ~$250M instead of ~$0.5B if only the discovered optimization techniques (using torch.compile and FP8 matmuls) were known earlier. Some people may find it hard to believe, but it's surprising how often we overlook simple ways to move the needle. *you can't go above 90-95% MFU against NVidia spec-sheet peak performance without various benchmark trickery like slowing down clocks, tuning data patterns, layout tweaking, etc.

After almost 3 years at Snap, several dozens of research and engineering projects, countless wild moments lived through together with the incredible team, I've decided to start a new adventure in robotics and join @rhodaai to bring the technological singularity a little closer

The recent Drifting Models paper from Kaiming's group got very hyped over the past few days as a new generative modeling paradigm, but in fact, it can actually be seen as a scaled-up/generalized version of the good old GMMN from 2015 (and the authors themselves acknowledge this in the paper in Appendix C.2, noting that GMMN can be seen as Drifting Models for a particular choice of the kernel). Also, I am very skeptical about its scalability (for higher diversity / higher resolution datasets, larger models, and videos). The way Drifting Models work is actually very simple: - 1. Sample random noise z ~ N(0, I) - 2. Feed it to the generator and get a fake sample x' = G(z) - 3. For each fake sample x', compute its similarity (in the feature space of some encoder) to each of the real samples x_i from the current batch. - 4. Push it closer toward these real samples using the similarities as weights (i.e. so that we push to the nearest ones the most). - 5. To make sure that we don't have any sort of mode collapse, repel each fake sample from other fake samples via the same scheme. - 6. Profit Now, GMMN follows exactly the same scheme, with the only difference being that it uses a different (unnormalized) function in the "distance computation" and doesn't allow for cleanly plugging in normalization/scaling in the similarity scores or CFG. Why didn't GMMN take off and why am I skeptical about Drifting Models? The issue is that it makes it much harder to compute any meaningful similarity when your dataset gets more diverse (happens when you switch to foundational T2I/T2V model training), or the batch size gets smaller (happens when your model size or training resolution increases), or your feature encoder produces less comparable representations (happens for videos or more diverse datasets). You can sure get informative similarities for 4096 batch size on the object-centric, limited diversity ImageNet with ResNet-50 feature encoder, but for smth like video generation, we train on hundreds of millions of videos or, at high resolutions + larger model sizes, with a batch size of 1 per GPU (not sure if will be fast to do inter-GPU distance computations). From the theoretical perspective, even though the final objective and the practical training scheme are the same, the mathematical machinery to formulate the framework is very different and enables direct access to the drifting field (e.g., to easily enable CFG which the authors already did). But I guess what I like the most about this paper is that Kaiming's group is boldly pushing against the mainstream ideas of the community, and hopefully it will inspire others to also take a look at the fundamentals and stop cargo-culting diffusion models.