Senior Research Scientist @Google working LLM agents, post-training and distillation. NLP+ML Ph.D. from @USC_ISI. Ex-@AWS AI, ex-@federalreserve.
Sunnyvale, CA
Joined May 2019
- Tweets90
- Following234
- Followers577
- Likes97
Great to see PLE featured across multiple key platforms:
☁️ Google Cloud: cloud.google.com/blog/produc…
🤝 Mercor: mercor.com/blog/introducing-…
🤗 Hugging Face (1.3K downloads and counting!) huggingface.co/datasets/goog…
Submit your agent to our leaderboard: ple-bench.com!
Thrilled to see our PLE benchmark is now offical on Huggingface!
Can AI agents do real engineering work?
Physical-Engineering's Last Exam (PLE), from @GoogleResearch & @GoogleCloud: 85 long-horizon tasks built by licensed engineers across mechanical, electrical, civil and structural engineering.
Even the best model + harness scores ~50%. 🧵
Rujun Han retweeted
New Google paper finds that the best AI agent scores only 50.6% on physical engineering jobs, often by missing a hard safety or feasibility rule.
Engineering benchmarks for LLMs mostly test textbook questions or toy problems that top agents already handle well. Real engineers work from piles of PDFs, spreadsheets, and CAD files and must issue designs that meet code and stay cheap.
They built PLE, 85 electrical, civil, structural, and mechanical jobs written or checked by practising engineers. Agents must read about 61 project files per task and deliver schedules, reports, or 2D and 3D drawings.
Claude Opus 5.5 in Claude Code topped 14 models at 50.6%, using 27M tokens and 63 minutes per task. On design tasks its outputs met 91% of constraints, but only 41.3% met every hard constraint.
If you build engineering agents, gate every design on hard safety and code limits, and spend effort on model choice before tuning the agent scaffold.
At #EMNLP! Please come by our FinanceHarness poster on Monday 11–12:30, Hall H.
Also happy to chat about intern & full-time roles at Google Cloud AI Research. Topics incude physical engineering agents (nitter.cf/HanRujun/status/211346…), agentic data synthesis & RL, harness, RSI, etc.
Rujun Han retweeted
Banger paper from Google Cloud AI Research on agents doing professional engineering work.
(bookmark it)
For professional engineering deliverables, the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task.
They introduce a benchmark called PLE (Physical-Engineering's Last Exam). It has 85 tasks across electrical, civil, structural, and mechanical engineering. Solving a task means producing a billable deliverable, such as a design package a licensed reviewer would sign off on, from a multi-format project corpus with physics and building-code requirements.
Some tasks are written end to end by practicing engineers. Others start from an engineer's problem statement, and an adversarial Creator-Solver agent pipeline builds the project environment, grader, and reference deliverables before experts validate them and make them harder.
Frontier models were tested under seven public agent harnesses. Most existing engineering benchmarks are textbook QA or toy design problems that leading agents already saturate.
arxiv.org/abs/2610.30537
Chat with Paper: academy.dair.ai/papers/ple-p…
Think your harness can beat 48.3%?
Join the PLE Open Competition:
• Bring your harness (hidden test set)
• 13 Google-sponsored models run FREE
• Prizes coming soon
📄 Paper: arxiv.org/abs/2610.30537
🌐 Leaderboard: ple-bench.com
💻 Code: github.com/google-research/p…
PLE Team: @HanRujun, @yufan_zhuang, @ARRay693, Han Yu, Zoey CuiZhu, George Lee, @ming5_alvin, Binz Roy, Varun Sakalkar, Vishy Tirumalashetty, Burak Gokturk, @tomaspfister, @chl260
Rujun Han retweeted
Banger paper from Google.
It's standard practice to sample several agent rollouts and trust the answers they agree on.
This Google paper shows that agreement can hide shared errors, while disagreement often points to the correct alternative.
VeriHarness turns the same base model into an agentic verifier with two jobs.
One resolves claims where rollouts disagree by checking workspace evidence.
The other challenges claims that every rollout agrees on and looks for requirements they all missed.
Across five long-horizon benchmarks, it gives the best selection scores among the baselines tested. With evidence-backed revision, it adds 6.2 points over a single rollout with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8.
The authors also release about 26,000 rollouts.
Paper: arxiv.org/abs/2610.00972
Chat with Paper: academy.dair.ai/papers/verih…
Thanks all coauthors for the contributions! @caiqizh, @ZifengWang315, Zoey Cuizhu Nigel Collier, @tomaspfister, @chl260
📄 Paper: arxiv.org/pdf/2610.00972v1
🌐 Website: veriharness.com
💻 Code: github.com/google-research/v…
Recursive Self-Improvement (RSI) is transforming how AI agents evolve, but have we considered that it needs regularization too?
🚀 Excited to share our latest work: RRSI (Regularized Recursive Self-Improvement of Agent Harnesses) arxiv.org/abs/2609.24972!
📄 Paper: arxiv.org/abs/2609.24972
💻 Code: github.com/google-research/r…
🌐 Website: regularized-rsi.com
Thanks all coauthors: @richardxp888, @ZifengWang315, @anmourchen, @yufan_zhuang, @yoonholeee, @ChengsongH31219, Han Yu, Zoey Cuizhu, @ming5_alvin, @HuaxiuYaoML, Burak Gokturk, @tomaspfister, @chl260