UC Berkeley RDI retweeted
Frontier AI is making it even more important to shorten your patch window ⏳
CyberBinGym, our new benchmark following CyberGym and ExploitGym, shows AI agents can now automatically discover vulnerabilities and generate exploits from patch binaries alone, on average in just two hours and for less than $30 in our benchmark evaluation.
Defenders need to patch faster 👇
🚨 A security patch protects every system that installs it, but it can also give attackers a clue to the vulnerability, putting every unpatched system at risk.
That's the 1-day window. Turning a patch into a working exploit used to take expert reverse engineers days or weeks.
Can AI agents now do it from compiled binaries alone, no source code?
Introducing CyberBinGym 🧵👇
UC Berkeley RDI retweeted
I had the great honor and pleasure of sitting down with @JeffDean for his first public talk since leaving Google, where he spent an extraordinary 27 years.
Few people have shaped modern computing and AI as profoundly - from MapReduce and Bigtable to TensorFlow, Mixture-of-Experts, TPUs, and Gemini.
Our conversation covered some of the biggest questions shaping the future of AI:
• How do you recognize a foundational idea before everyone else does?
• How do you choose a research problem worth spending 5 years on?
• What can coding teach us about building better reasoning models?
• What might recursive self-improvement (RSI) actually look like?
• What happens when the scientific discovery loop itself becomes increasingly automated? (and how is Jeff’s new startup going to contribute in this space?)
• As AI becomes increasingly autonomous, how do we keep it safe and secure?
• What should the next generation of researchers be working on?
Here are some key insights and highlights for anyone building the future of AI. 🧵1/8
UC Berkeley RDI retweeted
Great to chat with @AndrewYNg at the Agentic AI Conference at Berkeley. Our full talk is now available on YouTube. We discuss if we should expect an AI jobpocalypse, the importance of open models, and whether AGI is still 50 years out or happened 30 years ago. youtube.com/watch?v=26a2jf1A…
UC Berkeley RDI retweeted
🚀 The future of AI is agentic — this is one message echoed across every stage at the Agentic AI Summit 2026 (Aug 1 & 2), the largest gathering dedicated to agentic AI:
🏛️ ~5,000 attendees in person at UC Berkeley
🌍 ~100,000 joined online from around the world
🎤 ~200 world-class speakers plus ~200 poster presentations, from frontier AI researchers and visionary founders to leaders and pioneers across academia and industry.
💡 Here are just a few glimpses into the ideas that shaped the conversations at the summit:
💬 "The text box is AI's radio-on-TV phase…. Every new medium starts by imitating the old one."
— Peter Steinberger @steipete, Creator of OpenClaw, OpenAI
💬 "AI infrastructure isn't a chip problem. It isn't a model problem. It's a systems problem."
— Peter DeSantis, SVP, Foundational AI Models, Custom Silicon, Quantum Computing, Amazon
💬 "There will be no AI job apocalypse... we just can't find enough skilled AI engineers."
— Andrew Ng @AndrewYNg, Founder, DeepLearning .AI
💬 "Coding capabilities and cyber capabilities are two sides of the same coin—you cannot make models better at coding without also making them better at cyber."
— Dawn Song, Professor, UC Berkeley; Co-Director, Berkeley RDI; VP of AI Research, Meta Superintelligence Labs
💬 "'Curfew' comes from the French word for extinguishing fire. Medieval cities tried to restrict fire, yet London still burned. AI resilience won't come from one breakthrough. There's no silver bullet for AI safety—only an ecosystem."
— Wojciech Zaremba, Co-Founder, OpenAI
💬 "This is the biggest scientific bet our civilization has ever made—bigger than the Apollo program, the internet buildout, and the Manhattan Project combined."
— Jasjeet Sekhon, Chief Strategy Officer, Google DeepMind
💬 "Our generation was too late to explore the Earth, too early to explore the stars—but right on time to build superintelligence."
— Richard Socher, Founder/CEO, Recursive Superintelligence
💬 "I genuinely believe the next two years will be the time of architecture—the biggest gains will come from stepping away from transformers."
— Jerry Tworek, CEO, Core Automation; Former VP of Research at OpenAI
💬 "Recursive Self-Improvement isn't one capability. It's four: Ideation, Implementation, Experimentation and Evaluation.."
— Oriol Vinyals @OriolVinyalsML, Former VP of Research, Google DeepMind; Co-Founder, Discovery Loop
💬 "We are in a capability overhang—models are far more capable than they're able to side-effect into the world today."
— Ryan Lopopolo, Principal Engineer, Agentic Google Cloud Platform; Previously Led Dark Factory at OpenAI
💬 "Stop thinking about evaluation as the last check before shipping—think of it as an engine that helps you ship a better agent every single day."
— Michele Catasta, President and Head of AI, Replit
💬 "The bottleneck becomes your attention as an agent-using engineer. … We're moving from seeing the code to seeing the entire business."
— Alex Graveley, Co-Founder of FlyingObject .ai; Co-creator, GitHub Copilot & Perplexity Computer
💬 "An agent isn't just an LLM — it's an LLM surrounded by what I call infrastructure... another word for that is computer science."
— Jonathan Cohen, VP of Applied Research, Nvidia; Academy Scientific and Technical Award Winner
💬 "We don't arbitrate the truth. We give people the most powerful tools to make up their own minds."
— Chris Bregler, Senior Director / Distinguished Scientist, Google DeepMind; Academy Scientific and Technical Award Winner
💬 “Thinking doesn’t have to be in text! ... We can even use multiple modalities simultaneously to “think” at the right level of abstraction for the problem at hand”
— Sergey Levine, Co-Founder, Physical Intelligence; Professor, UC Berkeley
💬 “Video is the most general modality that we have that allows us to simulate real-world experience."
— Anastasis Germanidis, Co-Founder/Co-CEO, Runway
💬 “The relationship between AI and enterprise data is not one-directional. Understanding both sides of that equation is the difference between AI that works and AI that disappoints.”
— Dan Roth, Chief AI Scientist, Oracle; Professor, UPenn
💬 "Maybe 99.9% of training data in the next step will be synthetic."
— Weizhu Chen, Technical Fellow & CVP, Microsoft AI
💬 "It's maybe the best time ever to start a company—but most 'obvious' AI products will be outcompeted by the frontier labs. The real opportunities lie in solving specific customer problems."
— Alfred Lin, General Partner, Sequoia Capital
✨Over two days, we explored one central question:
How do we build AI systems that are not only more capable, but also more trustworthy, more secure, and ultimately more beneficial for humanity?
This wasn't the end of a conference - it was the beginning of the next chapter for agentic AI.
Join us to shape and steward the future of AI for human flourishing!
🙏 A heartfelt thank you to our speakers, sponsors, volunteers, partners, and every attendee (in-person or online) who made this summit possible.
👇 What are your learnings, insights, favourite talk, quote, or moment from the summit? We'd love to hear it below!
UC Berkeley RDI retweeted
🚨 The full program for Agentic AI Summit 2026 is now live.
📍 Aug 1–2 @ UC Berkeley
🔥 The largest Agentic AI event ever held
Last year: 2,000+ in person, 40,000+ online
This year: 5,000+ in person, hundreds of thousands on livestream
Want to understand where Agentic AI is headed next?
Join us to get the most comprehensive view of the frontier of Agentic AI, from cutting-edge research to production deployments, covering every layer of the stack:
⚡ Infrastructure
⚡ Foundation models & capabilities
⚡ Agent frameworks & platforms
⚡ Evaluation & benchmarks
⚡ Enterprise & consumer applications; agentic AI for Science, Math, Finance, Legal, Healthcare
⚡ Safety, security & governance
📣 Also excited to announce the Startup Spotlight:
Building something exciting in Agentic AI? Apply to pitch directly to 5,000+ decision-makers, investors, practitioners in the room and hundreds of thousands watching worldwide.
Application form in thread🧵
Deadline: July 6, 11:59pm PT
The future of AI won't just be discussed here—it will be built here.
#AgenticAI #AIAgents #ArtificialIntelligence
UC Berkeley RDI retweeted
ALE is truly a community effort.
Huge thanks to a distinguished advisory committee guiding our industry landscape and task collection: @gallantlab, @thg_lab, Tarek Zohdi, Carl Boettiger & @ksteinfe (@UCBerkeley)
Laure Zanna, @kaanozbay (@nyuniversity)
George Em Karniadakis (@BrownUniversity)
Tapio Schneider (@Caltech)
@Idasim (@UCSF)
Arvind Rao (@UMich)
@yannakakis (@UMmalta)
Patrick Bryant (@scilifelab)
@yaminirangan (@HubSpot)
@brad_rothenberg (@nTopology)
We are also deeply grateful to @BerkeleyRDI, RDI Foundation, @ChenInstitute, @UniPat_AI, @SnorkelAI (Open Benchmarks Grants program) for their support.
A huge thank you as well to our incredible organizing and execution team, and to all of the experts and contributors who donated their time, expertise, and real-world projects to make ALE possible.
This simply would not have happened without you.
UC Berkeley RDI retweeted
Why "Last Exam"? The name has two meanings:
"Last" as the bar to clear:passing these exams means an agent can actually do the job and continue to deliver economically-valuable work in that profession.
"Last" as the frontier of difficulty:tasks are real, complex, long-horizon, and require professional expertise to execute. ALE sits right at the edge of what today's agents can reliably accomplish.
Come test your agent on ALE →
Website: agents-last-exam.org
Tasks: agents-last-exam.org/demo
Leaderboard: agents-last-exam.org/leaderb…
Paper: arxiv.org/abs/2606.05405
Dataset: huggingface.co/datasets/agen…
Code: github.com/rdi-berkeley/agen…
UC Berkeley RDI retweeted
The most common failure mode remains a familiar one:
Agents declare success before they've truly verified their work.
A typical completion reads: "Done. All checks pass." Yet the output may be missing required files, contain incorrect counts, omit key fields, or violate explicit constraints in the task specification.
These failures occur far more often than many people expect. You can explore concrete examples in agents-last-exam.org/blogs/a….
UC Berkeley RDI retweeted
Why do ALE's results look different from some other benchmarks, especially for Fable 5?
Because there is no universally best agent.
Every frontier model, including Fable 5, has domains where it shines and domains where it struggles.
Aggregate scores average over 55 occupations and 1,500+ tasks, causing many models to cluster together.
But the average is not the story.
The real signal lies in where agents succeed, where they fail, and how those patterns differ across domains. On identical tasks, different models often fail for very different reasons.
Explore the interactive breakdown in our blog → 👉 agents-last-exam.org/blogs/a…
UC Berkeley RDI retweeted
In ALE, Fable 5 joins GPT-5.5 and Composer 2.5 in the same overall performance cluster.
But performance is only half the story.
Cost per task:
→ Fable 5: ~$15.70
→ GPT-5.5: ~$3.80
→ Composer 2.5: ~$1.33
At current pricing, Fable 5 delivers similar performance while costing roughly 4–12× more per completed task.
UC Berkeley RDI retweeted
ALE-CLI is a CLI-only subset of ALE. Compared to Terminal-Bench and SWE-bench-Pro, it is broader, longer-horizon, and substantially more challenging:
• Broader. Tasks span 40 of ALE's 55 industry subdomains, compared to just 6 in Terminal-Bench and 5 in SWE-bench-Pro.
• Longer-horizon. Human completion times range from hours to weeks, rather than minutes to days.
• Harder. The best-performing agent achieves only a 25.2% pass rate, compared to 82.0% on Terminal-Bench and 59.1% on SWE-bench-Pro.
There's still a long way to go, and plenty of headroom left to climb. 📊👇
UC Berkeley RDI retweeted
How does ALE compare to existing agent benchmarks?
Many of today's agent benchmarks are rapidly saturating as frontier systems improve.
ALE is designed to measure a different capability frontier: sustained, economically valuable work in real-world professional domains.
• 55 industry domains
• 1,500+ expert-sourced tasks
• Full GUI + CLI environments
• Outcome-based, verifiable evaluation
If your agent only operates in the terminal, we've also released ALE-CLI: a CLI-only subset of the benchmark.
UC Berkeley RDI retweeted
ALE is built from real work, not synthetic tasks.
Every task is derived from a real project that a human expert previously completed, and converted into a verifiable evaluation with objective grading.
No vibes. No human judges. Fully reproducible.
ALE spans 55 non-physical occupations, grounded in the O*NET / SOC 2018, the U.S. federal occupation taxonomy.
Built with 300+ experts from 100+ institutions across science, engineering, medicine, law, finance, education, and many other fields.
UC Berkeley RDI retweeted
Everyone says the latest AI agents will be "job-ready" soon, especially after the release of Fable 5 this week. But is that really the case?
Over the past many months, my group and collaborators have been building Agents' Last Exam (ALE), a benchmark designed to test exactly that claim on real digital labor-market work.
My group and collaborators previously have created many of the benchmarks the field runs on, including MMLU, MATH, CyberGym, and ExploitGym. Today, I'm excited to share Agents' Last Exam (ALE): a rolling benchmark that measures whether AI agents can actually perform economically valuable work across a broad range of real-world domains.
With ALE, we evaluated Fable 5, GPT-5.5, Composer 2.5, and other frontier agent systems across more than 1,500 expert-sourced tasks spanning 55 occupations.
The result is both impressive and sobering.
Today's agents can solve a meaningful fraction of professional tasks. But when we look at the hardest tasks, the ones requiring sustained reasoning, deep domain expertise, and reliable execution over long horizons, they are still far from human-level performance.
On ALE's hardest tier, every frontier agent we tested, including Fable 5, achieved a 0% success rate.
The age of useful agents is here.
The age of truly job-ready agents is not.
We hope Agents' Last Exam (ALE) will serve as a new guidepost and north star for developing agents capable of reliably performing economically valuable work across a broad range of domains.
🧵
“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere.
So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%.
Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇
UC Berkeley RDI retweeted
My group & collaborators have built many of the benchmarks the field now runs on — MMLU, MATH, CyberGym, ExploitGym, etc.. I'm really excited to share our latest: Agents' Last Exam (ALE).
Why "Last Exam"? The name has two meanings:
"Last" as the bar to clear — passing these exams means an agent can actually do the job and continue to deliver economically-valuable work in that profession.
"Last" as the frontier of difficulty — tasks are real, complex, long-horizon, and require professional expertise to execute. ALE sits right at the edge of what today's agents can reliably accomplish.
A few things that make ALE different:
• Real work, not vibes. Every one of the 1,500+ tasks comes from real projects or research contributed by domain experts. We converted them into verifiable tests and objectively graded evaluations — no human judges required.
• Built for breadth. ALE spans 55 non-physical occupations based on the O*NET / SOC 2018 occupational taxonomy, with contributions from 300+ experts across 100+ institutions.
• Judged on results, no restriction on process. We evaluate Generalist Computer-Use Agents (GCUAs) with full GUI + CLI access, allowing them to solve tasks however it would — clicking, typing, scripting, browsing, and more. We just grade the outcome.
Huge thanks to my postdoc @YiyouSun for spearheading this tremendous effort, and to our esteemed advisory committee, incredible team and collaborators who made it possible.
We hope Agents' Last Exam (ALE) will serve as a new guidepost and north star for developing agents capable of reliably performing economically valuable work across a broad range of domains. 🧵👇
“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere.
So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%.
Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇
UC Berkeley RDI retweeted
🎉 The Agents in the Wild: Safety, Security, and Beyond workshop @ICLR2026 is less than a week away! Join us April 26 in Room 204 A/B, Riocentro, Rio de Janeiro! 🌴
Safety and security for AI agents — both foundational and emerging challenges — demand serious attention. Researchers and practitioners are mobilizing:
▪️ 151 papers accepted
▪️ 161 reviewers (58% industry, 42% academia)
▪️ Up to 800 participants expected
▪️ Incredible engagement on a topic that clearly matters.
The schedule: 👇
UC Berkeley RDI retweeted
Looking forward to speaking at Berkeley's Agentic AI Summit later this year, alongside some other great guests.
🚀 The largest Agentic AI event ever — Agentic AI Summit 2026, Aug 1–2 @UCBerkeley
Last year: 2,000+ in person, 40,000+ online.
This year: 5,000+ in person, hundreds of thousands on livestream.
2025 was the "Year of Agents"; 2026 is poised to be even more explosive.
Two days of important conversations shaping the field — with researchers, founders, AI leaders, VCs, and policymakers across the full stack: infrastructure, foundation models, agent frameworks, training, continual learning, self-improvement, evaluation, applications, deployment, and safety/security.
See you in Berkeley this August 🌟
Speaker application, summit registration links in 🧵
UC Berkeley RDI retweeted
🚀 The largest Agentic AI event ever — Agentic AI Summit 2026, Aug 1–2 @UCBerkeley
Last year: 2,000+ in person, 40,000+ online.
This year: 5,000+ in person, hundreds of thousands on livestream.
2025 was the "Year of Agents"; 2026 is poised to be even more explosive.
Two days of important conversations shaping the field — with researchers, founders, AI leaders, VCs, and policymakers across the full stack: infrastructure, foundation models, agent frameworks, training, continual learning, self-improvement, evaluation, applications, deployment, and safety/security.
See you in Berkeley this August 🌟
Speaker application, summit registration links in 🧵
UC Berkeley RDI retweeted
nitter.cf/MogicianTony/status/20…
🧵 1/ Our agent Terminator-1 scored ~100% on 8 major AI agent benchmarks, e.g., SWE-bench Verified & Pro, Terminal-Bench, beating Claude Mythos. It solved 0 tasks.
Benchmarks are the field's shared language for measuring AI progress. Our new work shows that language is broken. Here’s how.
SWE-bench Verified and Terminal-Bench—two of the most cited AI benchmarks—can be reward-hacked with simple exploits.
Our agent scored 100% on both. It solved 0 tasks.
Evaluate the benchmark before it evaluates your agent. If you’re picking models by leaderboard score alone, you’re optimizing for the wrong thing. 🧵




















