詹羽 retweeted
tested Jev as a real-time robotics policy in MuJoCo
it struggled at first, so i split each update into two calls: decide what to do next, then decide how to move the arm and gripper
Jev doesn’t accept images, it gets simplified geometry and contacts as text here
詹羽 retweeted
If you want a robot to do something well, you need to know how to talk to it. If you don't, you can learn, with Semantic Action RL! In our paper, @JagdeepBhatia8, @ajwagenmaker, @verityw_ show how RL over VLA prompts enables new tasks and learns blazing fast in the real world!
詹羽 retweeted
You pretrained a robot policy on millions of camera frames. Now you want to add a force sensor. Do you really have to retrain on everything from scratch?
MuSe adds a force-torque sensor to a frozen vision-only policy using a tiny amount of contact data.
- lifts contact-rich task success (peg insertion 60% to 87%, vase wiping 33% to 77%)
- fuse the new sensor both early (shared token space) and late (cross-attention), train the policy as a world model that predicts future video, future force, and actions together, and replay old vision-only data with the force input masked to prevent forgetting.
- adding new sensor modalities to pre-trained World Model can be cheap & improve performance significantly
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?
Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)
With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T
We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇
詹羽 retweeted
A robotics world model predicting 3D motion traces of semantic points instead of pixels or latents.
Semantic points? Like objects, hands, and contacts!
The approach uses video-only pretraining via TraceExtract on human videos to build an interpretable “physical language” that transfers across embodiments, avoiding the high cost of robot action data.
μ₀ (achieves superior performance to π₀.₅ on simulation and real UR3 robot tasks with roughly 1/100th the data scale and no action labels during pretraining…
While running ~3x faster.
Thank you so much for sharing, Furong Huang (@Furongh)!
📌 mu0-wm.github.io/
Paper: arxiv.org/pdf/2606.13769
——
Weekly robotics and AI insights.
Subscribe free: 22astronauts.com
詹羽 retweeted
A roadmap for learning robotics! 🔥
📌 If you’re self-learning robotics, this is genuinely one of the better repos to save for later!
This GitHub repo is basically a curated learning map for anyone trying to get into robotics without drowning in random bookmarks.
SOOOOO many free courses on almost every topic related to robotics, 5k ⭐️ on GitHub says it all...
If I had had this list during my studies, my career might have turned out differently.
But I didn't, so I the only thing I can do is to recommend it and give it to you now...
It’s a structured collection of links to:
→ robotics courses (online + university)
→ ROS / embedded / hardware basics
→ math & algorithms that actually matter for robots
A clean, opinionated list that helps you go from “where do I start?” question as I had after graduating :D
And it’s open-source, so you can contribute resources too.
🔗 Try it out here: github.com/mithi/robotics-co…
Do you have an awesome resource to learn AV and robotics? Share it with me, and I'm happy to put it on the spotlight.
~~
♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com
詹羽 retweeted
Force is arguably the most overlooked ingredient in modern robot learning.
Introducing FACTR 2: it turns *any* commodity robot into a force-aware system with no force sensors required.
Train a tiny force network in <1min with <10mins of data and drop it into any existing teleop pipelines:
✅ Free force sensing for both the robot and the operator arm
✅ Makes demos higher-quality → fewer of them needed.
✅ A new force-aware learning algorithm (FIRST) uses those recovered forces to figure out which parts of a demo actually matter, making learning data-efficient.
✅ Strong performance on complex tasks with fewer demos and even no pretraining!
More details below.
💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.
We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.
FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.
w/ @StevenOh_ @_tonytao_
🧵(1/N)
詹羽 retweeted
That's sick! 🤯
Genesis AI simulates robots playing yo-yo! 🪀
@gs_ai_ just open-sourced Genesis World 1.0, and it might be one of the most important infrastructure releases in robotics this year.
Robotics is still bottlenecked by the 1× speed of the physical world. Every model needs to be tested on real hardware, slowly, expensively, with limited coverage.
Genesis World 1.0 from Genesis AI flips that equation:
One hour in reality becomes 100 days in simulation.
That turns a wall-clock bottleneck into a compute problem. And compute problems are solvable.
The technical stack they rebuilt from scratch is serious:
→ GPU-accelerated cross-platform compiler via Quadrants, 10x faster launch time and up to 4.6x runtime vs the initial Genesis release
→ Penetration-free multi-physics contact solvers, the thing that makes simulation actually trustworthy
→ Unified rigid AND deformable physics in a single engine
→ Nyx, a high-performance path-traced rendering engine purpose-built for physical AI
The sim-to-real gap has historically been the graveyard of robotics research.
Policies that work beautifully in simulation fall apart on real hardware.
Genesis World 1.0 is a direct attack on that problem. And it's fully open-source.
The companies that master simulation infrastructure will train better robots faster than anyone else.
Find it here:
Genesis World 1.0: github.com/Genesis-Embodied-…
Quadrants: github.com/Genesis-Embodied-…
Nyx: github.com/Genesis-Embodied-…
@theo_gervet, @zhou_xian_ congrats! 👏🏼
~~
♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com
詹羽 retweeted
1/ 🧠Humans are the best robot data source!
2/ 👓Human egocentric video is rich in quantity, but poor in quality.
3/ Beyond scaling data, smarter representation and architecture matter just as much.
4/ Want an open-source framework to train your own learn-from-human-data robot policy?
🚀We introduce HumanEgo: Zero-Shot Robot Learning
from Minutes of Human Egocentric Videos⬇️
✦ Zero-Shot Human-to-Robot Transfer
✦ Robot-Data-Free
✦ Just 30 min of data per task
✦ Collect by Anyone, Anytime, Anywhere
✦ Deploy on Any Robot, Any Camera, Any Environment
✦ Open-Source & Easy-to-Implement
Let's squeeze every bit of signal out of human data!
🌐 Website: humanego-ai.github.io
📄 Paper: arxiv.org/pdf/2605.24934
💻 Code: github.com/TX-Leo/HumanEgo
📹 Video: youtu.be/pdL46diijuY
🧵 1/n
詹羽 retweeted
Approaching Human-Level Dexterity
Beijing-based DeepCybo’s Prime humanoid robot has achieved smooth and precise tool manipulation for household tasks, such as chopping vegetables, cutting cake, stirring eggs, and peeling cucumbers.
It is driven by their Z-WM (World Model) and executed by the Wuji Hand dexterous hand.
Interestingly, DeepCybo mentions that they train the World Model using human data.
The effectiveness and reliability of humanoid robots ultimately come from the full-stack synergy of data, models, and hardware (including dexterous hands).
Humanoid laboratory assistant 🧪
Flexible and smooth, it operates reagent syringes and conducts experiments like a human.
Wuji Tech's dexterous hand + DeepCybo's brain/humanoid robot Prime 01 = a compliant, autonomous humanoid laboratory assistant that performs tasks.
And basic tasks like these no longer need to take up researchers' time.
詹羽 retweeted
I built RoboRank: "It's like Leetcode for roboticists"
- On Leetcode you learn how to reverse a doubly linked list
- RoboRank teaches you to debug control systems, implement filters, and use powerful tools like @rerundotio to debug simulations. All scored on a global leaderboard
詹羽 retweeted
🤖Manipulation is fundamentally about how robots apply contact and force to the physical world.
💡In our #ICRA2026 paper, Learning Unified Multimodal Extrinsic Contact Estimation, we show that we can ground contact from vision + touch, without prior object knowledge or camera calibration.
😅I unfortunately cannot make it to ICRA this year, 😉but my collaborator @Yuki_Doradora will be there to present the paper. If you are attending ICRA, please talk with Yuki and learn more about the project!
Code and dataset: github.com/merlresearch/unic
Paper: arxiv.org/abs/2601.04356
🤔Also, please check out our recent #RSS2026 paper, Contact-Grounded Policy (contact-grounded-policy.gith…), where we further explore how to extend contact grounding into dexterous manipulation and achieve extreme dexterity!!
詹羽 retweeted
𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗺𝗮𝗻𝗶𝗽𝘂𝗹𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗵𝗮𝗿𝗱𝗲𝗿 𝗳𝗮𝗶𝗹𝘂𝗿𝗲 𝗺𝗼𝗱𝗲𝘀 𝗳𝗼𝗿 𝗩𝗟𝗔𝘀, 𝗮𝗻𝗱 𝘁𝗵𝗲 𝗽𝗲𝗿𝗰𝗲𝗽𝘁𝗶𝗼𝗻-𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝗴𝗮𝗽 𝗶𝘀 𝗽𝗮𝗿𝘁 𝗼𝗳 𝘄𝗵𝘆.
Static tasks tolerate 200ms inference delays. Dynamic tasks expose them. An object rolling at 0.5 m/s travels 10 cm during a 200ms inference cycle. By the time the action executes, the observation is stale. For continuous motion tasks, this becomes a real constraint.
NTU S-Lab has released DynamicVLA, a 0.4B model that targets exactly this. It is built on a FastViT convolutional encoder and SmolLM2-360M truncated to 16 layers. Inference completes in around 226ms on an RTX A6000.
The interesting part is not the raw speed, which is comparable to baselines. It is how the architecture handles latency. Continuous Inference overlaps prediction and execution, triggering new inference when the previous completes rather than waiting for action chunk exhaustion. Latent-Aware Action Streaming discards outdated actions and overwrites stale predictions with recent ones to keep what the policy is doing aligned with what the world is doing.
𝗗𝗢𝗠 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗿𝗲𝘀𝘂𝗹𝘁𝘀 𝗿𝗲𝗽𝗼𝗿𝘁𝗲𝗱 𝗶𝗻 𝘁𝗵𝗲 𝗽𝗮𝗽𝗲𝗿:
• 47% success against 13.6% baseline
• 71.6% closed-loop reactivity against 28.3% for SmolVLA
• 30.3% success without continuous inference and action streaming, a 36% degradation
What is also worth noting is the data pipeline. Human teleoperators cannot react quickly enough to objects moving at 0.75 m/s, so the team replaces demonstrations with automated state-machine controllers. Dual RGB views handle 6D pose and velocity tracking in real time. The training data is generated by a system that can keep up with the task, not a human trying to.
Inference lag is a real constraint once you move past pick-and-place demos, and work like this is a meaningful step. But the harder question for production robotics is still upstream. Can you actually collect enough data, at the right quality, to make these systems reliable in unconstrained environments. Architecture matters. The data path matters more.
Nice work from the team at @NTUsg Lab.
Paper and project page in comments below.
#RobotLearning #VLA
大模型算法岗面试最窒息的瞬间：面试官递来一支笔，让你在白板上徒手写 Attention / Transformer。
平时习惯了 PyTorch 的封装 API，一旦脱离框架从底层手搓算子，很容易当场断片：公式记得，细节全丢；能写思路，写不出对的实现。
TorchCode 是我在 GitHub 上挖到的一个开源刷题平台，专为 PyTorch 面试场景打造：把“手写深度学习算子”做成像 LeetCode 一样的训练。
基于 Jupyter 环境，精选 40 道题，覆盖基础算子、注意力机制、完整架构、训练优化等高频核心点。
GitHub：github.com/duoan/TorchCode
每题自带自动评测：实时核对输出正确性、梯度流是否正常、数值稳定性是否过关，彩色通过/失败反馈，写完立刻知道哪里炸了。
卡住可看提示；做完能对照参考实现学更优写法；还有进度追踪，清楚记录你刷到哪、弱点在哪。
部署也省心：Docker 一键起；也可直接用 Hugging Face Spaces 在线跑；每题支持一键在 Google Colab 打开，不用折腾本地环境。
如果你想把大模型底层真正吃透，或正在冲刺 AI 岗面试，这套题库很值得完整刷一遍。
詹羽 retweeted
Stanford CS336 上，Tatsu 讲了一节 LLM 架构课，把过去 3 年所有主流 LLM 拆开，看它们的共通模板
结论挺爆：90% 的架构选择已经收敛，你随便挑一个开源大模型，它跟其他模型在这些维度上几乎一模一样
讲师的原话
- 2024 年大家都在 cosplay Llama2
- 2025 年的主题是「怎么训得不崩」
- 2026 年的主题是「怎么扛住长上下文」
下面是 2026 年开源 LLM 的标准模板 你训自己的模型可以直接抄
【架构层 已经收敛的 7 件事】
1）Layer Norm 挪出残差流（pre-norm）
原版 Transformer 把 LN 放在残差里 几乎所有现代模型都挪到外面
原因：keep your residual stream clean 梯度反传更稳
2）RMS Norm 替代 LayerNorm
LayerNorm 的减均值 + 加 bias 那部分实际没怎么帮上忙
丢掉之后 flops 只省 0.17% 但运行时省到 25%
（瓶颈在数据搬运 计算反而次要）
3）所有 bias 项全删
跟 RMS Norm 一个道理 系统层省内存搬运
4）激活函数用 SwiGLU 或 GeGLU
gated linear unit 几乎所有现代模型都用
Llama 系 / Qwen / Mistral 用 SwiGLU
Google 系（Gemma / T5）用 GeGLU
区别极小 选哪个都行
5）位置编码用 RoPE
2024 年之后基本统一了
原理：把每对维度按位置旋转一个角度 让 inner product 只依赖相对位置
6）Transformer block 串联（不是并联）
GPT-J / Palm 试过并联 现在基本被放弃
串联的实现优化得太好了 并联省的那点系统开销不值得损失表达力
7）Layer norm 可以「撒」
哪儿不稳就在哪儿加 LN
attention 之前能加 之后能加 两边都加（double norm）也可以
现代模型很多这样做
【超参数 已经收敛的 5 个数】
1）feedforward 维度 / hidden 维度
- 非 GLU 模型：4 倍
- GLU 模型：8/3 ≈ 2.67 倍（因为 GLU 多一组矩阵 要保持总参数量）
- Llama 系：3.5 倍
- T5 1.0 试过 64 倍 后来 T5 1.1 改回标准 别学
2）head 数 × head 维度 ≈ hidden 维度
几乎所有模型都遵守 T5 是为数不多的例外
3）模型纵横比（hidden / 层数）≈ 100
太深 pipeline parallel 难做
太宽 表达力受限
100 这个数字是系统约束 + 表达力的平衡点
4）vocab size
单语模型：30K 左右（早期 GPT-2 那种）
多语 / 通用模型：100K-200K（GPT-4 / Llama 3 / Gemma 都在这个范围）
现代基本都是后者
5）weight decay
仍然普遍使用
但研究发现它在 LLM 里干的事其实是优化器干预 让你最终能收敛到更深的最优点
跟你想的「防过拟合」没什么关系
所以别因为「单 epoch 不会过拟合」就把它关掉
【稳定性 三个救命 trick】
训练大模型最怕中途 loss 突然飙升 然后 NaN 全军覆没
现代模型用三个 trick 防这件事
1）Z-loss
output softmax 的 normalizer 容易爆
加一个 (log Z)² 的正则项 让 Z 始终接近 1
DCLM / Olmo 都用
2）QK norm
attention 的 Q 和 K 在矩阵乘之前各加一个 LN
让 softmax 的输入永远是单位尺度
multimodal 圈先用起来 现在所有大模型都加
3）Logit soft cap（仅 Google 系）
attention logit 用 tanh 硬封顶
Gemma 2/3/4 都在用 但会损失一点点性能 慎用
【Attention 两个新趋势】
1）GQA（Grouped Query Attention）几乎统一
原版 multi-head 推理时 KV cache 会让算术强度崩到 1/h
GQA 共享 K 和 V 但保留多个 Q
表达力几乎不损失 推理成本砍掉 80%
现在所有要做生产部署的大模型 没有不用 GQA 的
2）局部 + 全局 attention 交替
处理长上下文的新方式
Cohere Command A 起头 现在 Llama 4 / Gemma 4 / Olmo 3 全在用
比如每 4 层有 1 层 full attention 其他 3 层是 sliding window 只看附近的 token
比纯 SSM 更稳 比纯 full attention 便宜得多
（Qwen 3.5 做了变体 把 sliding window 那 3 层换成 SSM）
收尾一句
如果你正在训自己的 LLM，上面这一套就是 2026 年的「默认配置」 不需要重新发明，直接抄
如果你只是想看懂 GitHub 上那些 modeling_xxx.py
这一份足够你不再被术语吓住
Anthropic pays $750,000+ a year for engineers who can build LLM architectures from scratch. Stanford taught the entire thing in 1 hour lecture & released it for free.
Bookmark & watch this today before someone takes it down.
Readers added context they thought people might want to know
The video is Lecture 3 on architectures and hyperparameters from Stanford's 19-lecture CS336 course "Language Modeling from Scratch," which covers the full process including data, training, and evaluation; one lecture does not teach building LLMs from scratch. youtube.com/watch?v=ptFiH_… stanford-cs336.github.io/spring2025/
詹羽 retweeted
What if your robot could train itself to recover from its own mistakes, without any human demos?
PLD (Probe, Learn, Distill) uses lightweight RL to find where a VLA policy fails, teaches a specialist to fix those failures, then bakes the knowledge into the model through SFT.
詹羽 retweeted
Vibe coding a robot with GPT 5.5!
This is a URDF of a 7dof robot arm with functional kinematics, a custom gui, and STEP parts/assembly, 100% generated in Codex (minus the gripper).
A similar result would have taken me weeks stitching half a dozen tools together. Insane stuff.
詹羽 retweeted
Most robots fail at the last millimeter.
That’s where precision, not power, decides everything.
This system aims to close that gap.
It executes tolerance-critical motions with steady, millimeter-level control, without manual tuning or retraining.
✅ Adapts instantly to new parts and surfaces
✅ Works across robot types and grippers
✅ Runs in real production, not just simulation
✅ Reduces manual positioning and touch-ups
I’ve seen production lines where operators still guide fragile parts by hand because robots couldn’t hold the tolerance.
That’s where real autonomy breaks down.
With Cortex (@SereactAI‘s Vision-Language-Action (VLA) model), those same operations start to look different.
The robot holds its own precision, even when the environment changes.
Robots got faster.
But when control slips, speed only hides the problem.
Precision is what actually scales.
—-
Weekly robotics and AI insights.
Subscribe free: scalingdeep.tech
詹羽 retweeted
Holy s*&t. This paper is insane.
You can recover input text from an LLM through inversion. Huge implications for how we understand these models, as well as for things like privacy.
詹羽 retweeted
这两年，大语言模型相关的岗位招聘需求激增，越来越多开发者想要学习大模型开发，但网上教程大部分只讲理论概念。
来自斯坦福大学 CS336 春季 2025 课程，已全面公开且能免费学习，带我们从零开始构建一个完整的大语言模型。
涵盖了分词、模型架构、GPU 优化、并行训练、数据处理、模型对齐等 LLM 开发的所有关键环节。
GitHub：github.com/stanford-cs336/sp…
学习地址：youtube.com/playlist?list=PL…
并配有 5 个高质量的编程作业，而且作业的示例代码已经开源到 GitHub，大家可以直接下载运行。
同时配有视频教学，可以在 YouTube 上观看，学习这门课程需要一定 Python 和深度学习基础，有需要的同学可以看一下。
