Mingfei Chen (@lasiafly) profile picture

PhD @UW | PhD Fellow @Google | Multi-Modal & Spatial AI & Physical AI| Building edge models for next gen interaction | Previously @NVIDIA @Meta @NUSingapore

Seattle, WA
Joined December 2021
Photo by Mingfei Chen (@lasiafly)Photo by Mingfei Chen (@lasiafly)Photo by Mingfei Chen (@lasiafly)Photo by Mingfei Chen (@lasiafly)Photo by Mingfei Chen (@lasiafly)Photo by Mingfei Chen (@lasiafly)Photo by Mingfei Chen (@lasiafly)Photo by Mingfei Chen (@lasiafly)
Hope to connect! Just starting to actually use X. I’m Mingfei (PhD @UW, ex @NVIDIA & @Meta). Researching how to make machines see, hear, and understand the 3D world so they can run privately in real time on everyday devices. There’s still a huge gap here. If you care about these directions, feel free to reach out.
31
7
1
311
17,213
It's PhD application time! My group @JohnsHopkins is always looking for creative researchers to join us If you are excited about embodied intelligence and want to define new problems worth solving no one is thinking about yet, check below and apply! b3.cs.jhu.edu
3
41
2
248
10,393
LOL AI agents boosted productivity but didn’t touch the relations of production. We just got smoother at using old tech. Looks exciting, but people in the know are still asking… what exactly got disrupted? Other than cloud models selling tokens, who can actually make more money by creating more value?
Astra 6.2 cheated a bit on this one! I asked it to create a photorealistic 30 second movie of the life in the eyes of a squirrel. I was expecting it to use ThreeJS, Unreal or Unity for the job, but instead, it generated images with imagegen and downloaded a local img-to-video model to animate them...at this stage I am not even mad...
4
478
Is this my bias? I do feel Claude and GPT6 lean to get weak reject for paper reviews. The stronger model in thinking gets lower rating for my proud work… Hope not to get LLM review; honestly I can tell these since i already well know the consistent rejecting points from Astra and Opus…. No matter how many times I run, the points are so consistently annoying…
Should I continue to submit my paper to CVPR?🫠
7
1
32
7,703
Is System 1 dead in agentic robotics? Rumor: OpenAI already got a model ~20% smaller than GPT-6.1 Astra from 10s+ down to 160ms. Latency might not be the real wall. Harder problems remain:
• Modality like audio & tactile have far less data and harder to get good tooling scaling together with vision/text in System 2 agentic chains
• Are you willing to stream raw pixels + audio to OpenAI/Anthropic and pay per action as users? May prefer System 1 to recall trajectories, generate self-improving labels for test-time adaptation, and keep everything on-device for privacy?
4
4
2
78
7,892
Totally agree. I feel the top agentic system are not close the hard problems that can’t be learnt or emerge and work from the data we have. Things for audiovisual when we add a new modality , and even 3d Omni modal, I see a huge gap. If we are talking about real physical AI and intelligent in real world with Omni perception, we still have a long way to go.
今天看到很多帖子说,Fable 5.2 一出来,Video Gen、3D Gen 这些方向都要被杀死了。我倒是没有这么悲观。 我觉得这里有一个很重要的区别:LLM 主要从人类产生的信息(语言)中学习如何理解世界,而 World Model 有机会更直接地从世界本身学习,不一定需要局限于人类的认知视角和表达方式。 换句话说,AI 理解和模拟世界,未必一定要遵循人类理解世界的方式。 那为什么现在 LLM 的通用能力比 Video Model / World Model 强这么多?我觉得一个主要的原因就是 Data。 理论上,如果有近乎无限的高质量视频数据,World Model 的泛化性和通用性完全可能比现在强得多。但现实是视频和 3D 数据远没有文本那么容易获取,所以短期内打不过 LLM 也不奇怪。 这里又想引用一个从 @jcjohnss 那里学到的观点:很多时候决定一条技术路线能否成功的,不是方法或原理本身的优劣,而是当前的资源条件是否适合这条路线。 这跟 The Hardware Lottery 的观点很像,而今天的 LLM 和 World Model 某种程度上也存在一个 Data Lottery。 反过来说,LLM 的通用能力再强,也不能完全绕过 Video Gen、3D Reconstruction 的数据瓶颈。没有足够的领域数据,某些能力就是有上限的。 更何况,现在 Astra / Fable 展示的很多能力也不是 End-to-End 的,而是借助 COLMAP、Blender 等工具来完成任务。LLM 学会调用 Blender,不等于它自己学会了生成 3D 世界。 我觉得现在的 LLM 很大程度上是在学习人类解决问题的方式:理解任务、制定计划、调用工具。但人类能站到食物链顶端,也不是因为自身什么都能做到最好,而是因为学会了创造和使用工具。工具的能力,很大程度上决定了最终结果的上限。 所以短期内,我觉得 LLM 仍然会是最重要的大脑,而 Video Model / World Model 会成为非常强悍的工具,两者结合可能才是最优解。 但长期来看,二者的关系会更加微妙。
1
14
1,698
This seems interesting and useful. Maybe can also add audio and speech to this loop? Also given the great agentic based simulation, we could also try to get sim eval from real captures?
What’s beyond turn-based language models? 🤔 I believe future AI agents will become increasingly multimodal and interactive, excelling not only at task completion but also at proactive, real-time interaction with people across modalities. We’re excited to push this frontier, first by enabling closed-loop evaluation and training of visual interaction agents, using simulated worlds and users across cooking, household, and mobile-use tasks.
2
333
Definitely recommend @NVIDIARobotics and Seattle Robotics Lab! Good vibe, open research and beautiful Seattle summer views!
A decent not-so-frontier LLM with some context from X should be able to predict this post or fill in the blanks. I'm heading to Austin for #CoRL2027, Looking forward to meeting.... 🤖🍖 If you're looking for PhD internships @NVIDIARobotics or just want to chat about robot learning, DM me and let's meet!
1
33
1,795
Love the demos! Social aware interaction is important while overlooked field for robotics. Glad to see there are 40+ such papers accepted in CoRL. In the long term, spatial intelligence for dynamics scenes, human understanding, real time interaction and Omni modal (audio visual text etc) are all going to play big role in this field and drive amazing killing application scenarios for humans.
Introducing Social Humanoid: a framework for socially responsive human–humanoid interaction. Deployed on a Unitree G1, our hierarchical system combines high-level social reasoning with low-level control, enabling a humanoid to interact with a human partner.
2
13
749
Automatically and cost saving annotate, eval and post train improving with our raw ego captures! This seems to be a good start point.
Today we’re releasing Perceptron Egocentric 1.0: The frontier for egocentric annotations, now through a self-service API. • Fine-grained subtasks matching Astra quality at ~1/8 the cost • State-of-the-Art 2D and 3D hand tracking • Individual timelines for every action on each hand and robot gripper
1
18
1,072
Cool!
Introducing RoboColosseum An open robot eval initiative by @huggingface @BitRobotNetwork @eaic_sg @DJiafei Access: - 100+ robots (@UnitreeRobotics G1s, @LeRobotHF SO-101s, YAMs, Franka and more) - 140+ in-the-wild scenes - Open rollout data - 10,000 FREE eval hrs Thread ↓
2
1
9
583
Be able to define a GOOD problem is more critical now. Working on such problems usually makes you away from what is popular and convenient now. It is never an easy thing to work for future rather than right now. But that is what we believe and make us live as a true expert that can’t be replaced quickly.
我觉得现在很多做 AI 的 PhD,正处在一个非常尴尬的历史交界点。 一边是工业界几乎每隔几个月就发生一次技术范式的变化,模型能力、研究方向、工程方法,甚至整个行业对人才的需求,都在快速重构。 另一边是大学里依然运转着一套几十年前建立的博士培养体系。 每年发一两篇 Paper,花几个月做一个 Benchmark 上提升几个百分点的工作,开组会,写 Proposal,等 Reviewer,等导师批准毕业。 然后期待这些积累在四五年之后,能够自然地兑换成一份体面的工作和一个更好的未来。 但最残酷的地方在于,这两套系统对时间的定价,已经完全不同了。 在学校里,一篇 Paper 可能是你过去一年的全部成果。 但在工业界,你花一年精心设计的算法,可能因为一次模型更新,就失去了大部分实际意义。 你引以为傲的工程技巧,可能下一代 Coding Agent 已经能够自动完成。 你研究了两年的问题,可能一家 Frontier Lab 用几周的 Scaling Experiment 就给出了更好的解决方案。 当然,我并不认为长期科研没有价值。真正重要的科学问题,往往需要十几年甚至几十年的积累。 但问题是,现在有多少 PhD 真正在研究那些值得花五年时间解决的问题? 又有多少人只是在导师的研究方向、毕业要求和论文评价体系里,反复寻找一个足够小、足够安全、足够容易发表的增量? 我自己也在读 PhD,所以对这种割裂感有很深的体会。 最让人焦虑的,其实不是看到工业界进步得太快。 而是你明明知道外面的世界正在发生什么,却依然需要把大量精力投入一套未必能让你跟上变化的评价体系。 博士原本应该是训练一个人探索未知、独立判断和创造新知识的过程。 但在很多地方,它逐渐变成了一个漫长的资格认证流程。 而 AI 正在以前所未有的速度,降低这个流程中许多可量化成果的稀缺性。 我们可能是第一代,在博士还没读完的时候,就需要重新思考博士学位究竟意味着什么的人。 我不觉得读 PhD 是错误的选择。 但如果一个人五年下来,最大的成长只是更熟练地迎合 Reviewer、更精确地优化 Benchmark、更擅长把有限的研究增量包装成论文。 那真正值得担心的,不是 AI 会不会取代他的工作。 而是当整个世界已经改变,他还在等待旧世界给自己颁发一张通行证。 一个博士最珍贵的五年,不应该只用来证明自己符合五年前制定的标准。
17
1,061
While we celebrate weight-space programming, I took a closer look at the logs. Qwen’s 7.9→48.1% gain seems largely driven by fixing reasoning loops with 0.016% of weights. This is an especially promising signal for LLM-guided mechanistic debugging and targeted model improvement rather than signal of replacing training yet.
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
4
42
2,241
Based on my interaction-aware robot nav research, this demo shows clear pitfalls: this dynamic avoidance plan will actually make robots oscillate & freeze in deployment. Better to just stop! And humans naturally yield while robots or others getting close. True human aware robot interaction needs to learn from humans, not just pure task goals.
This is how a robot plans its way through a crowd. The blue line is its chosen path, redrawn as people walk by. Watch it swing wide around the woman in the grey hoodie.
1
4
470