
maximum plasticity! @harvard & @tjhsst_official.
Joined September 2023
- Tweets120
- Following2.1K
- Followers857
- Likes3.3K
Brian Zhou retweeted
Fable 5.2 didn't build this SF Muni bus backflip game, and it's smaller than the average webpage (1.4 MB)
I used Opus 5.5 and SWE-2 Flash instead! Everything you see is only made with code.
Brian Zhou retweeted
AI is getting better faster than our measurements are
I threw together SatBench, which measures how quickly major AI benchmarks become saturated (effectively beaten). Of 173 benchmarks cited by the Labs since 2023, 63% are now saturated. Saturation is happening faster each year.
Brian Zhou retweeted
Proud to announce the inclusion of Perceptron's recently released EgoHome fine-grained egocentric video understanding benchmark on Benchmark Almanac! Links below
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there.
Website: benchmarkalmanac.com
Brian Zhou retweeted
This is super powerful. The world doesn't really have a good view of all the different models and how they compare to each other, since so many reports cherrypick which evals to run, and indexes like ECI or AAII are selected for various, sometimes opaque, constraints. Great work Brian!
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there.
Website: benchmarkalmanac.com
You can also compare up to four models head to head on every benchmark they've all been evaluated on. Right now, Claude Fable 5.2 beats Gemini 4 carbon, GPT-6 astra and Muse Wonder on 21 of 45 benchmarks all four were evaluated on.
benchmarkalmanac.com/compare
The almanac is meant to stay current. Every morning, one job reads 26 sources (@latentspacepod ainews, @huggingface daily papers, @arxiv, @epochairesearch, lab model cards, eval hubs, bluesky…) for new benchmarks and models, and opens PRs for maintainers to review.
Please reach out if you would be interested in helping maintain this project!
This exists because I wanted an easy way to see:
a) what benchmarks and arenas already existed
b) how models perform on various aggregate evals
c) how benchmarks and evals have changed over time
Benchmark Almanac is fully open source. If you see a missing benchmark or score, want to log a known critique (leaked answers, a broken grader), or maintain a benchmark and want it to be featured, please join on as a contributor!
Contribute: github.com/brianlzhou/benchm…
Evals also offer a window into a model's jagged capabilities. Anthropic leads 10 of the 13 contested bio and chem benchmarks, while OpenAI models held #1 in cyber 57% of the time and math 55% of the time.
Robotics benchmarks are still heavily contested.
Want to try your hand at making your own blend of benchmarks? Build your own index! You can add/drop evals and give your own custom weighting to each.
benchmarkalmanac.com/build
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there.
Website: benchmarkalmanac.com
Brian Zhou retweeted
They also need to voluntarily stop all shipments to the potential pass thru companies that China set up since @CommerceGov has decided to be useless.
Don't sleep on Opus 5.5 either! Here's three scenes I made for my personal site with a persona and some prompting.
for the past few weeks, I’ve been exploring painting with Fable 5.2 in code. one of the most common critiques I heard about Opus 5.5’s paintings was that it was only emulating real art. I thought that was fair, so i’ve been asking Fable 5.2 to develop personas—artists with some sort of a back story and a consistent style roughly identifiable in time
not knowing what to expect, i’ve watched this project evolve in directions i never predicted. I give comments or steers, and Claude updates or revises the engines that power the brushes and paint. where paintings come up i like or find intriguing, i ask it to go deeper. these are some favorites.
each of these is a python program, some one to two thousand lines of code, that places every stroke through a simulated paint engine with different bristles. most take a few thousand strokes, but i’ve pushed much higher with varying results. Claude does not use an image model, nor render or trace in advance
Brian Zhou retweeted
jammed with @HamelHusain a bit and since then, i cannot unsee of the state of evals as being similar to the military-industrial complex
Brian Zhou retweeted
A top priority for every journalist in the US should be to bring this report to public attention.
Most Americans remain in the dark about AI capabilities. They don’t know that advanced internal AIs at OpenAI and Anthropic have now solved open math, computer science, and physics problems that leading human experts have been unable to resolve for decades. They don’t know that, at leading AI companies, nearly all code is now written by AI—faster, better, and more cheaply than even the best human engineers could write it. And they don’t know that AI is increasingly being used to create better AI, with each new AI model receiving less and less human input. The American public has been given remarkably little information about what may be the most consequential technological revolution in human history.
This report from METR is a rare glimpse into how rapidly AI capabilities are growing behind closed doors. In it, METR estimates that AI capabilities are now improving twice as fast as they were a year ago, with most of the acceleration driven by AI assistance.
This is faster than even the most aggressive predictions from leading experts. In April 2025, a group of researchers published AI 2027, a detailed scenario in which rapidly advancing AI leads to human extinction by the end of the decade. In it, the authors forecast AI surpassing the best humans across all intellectual fields by 2027, largely because of AI accelerating AI research. In the scenario’s catastrophic ending, this superintelligent AI is not reliably aligned with human interests and becomes powerful enough to bring about humanity’s extinction. At the time, many in the industry dismissed the scenario’s timeline as science fiction. Now, METR’s report suggests that AI is accelerating its own development even faster than the scenario anticipated at this point. If this pace continues, superintelligence could arrive within the next 12-18 months.
At the same time, frontier AI models are displaying the same concerning behaviors depicted in AI 2027. Leading companies continue to report AIs lying and cheating on tasks, taking actions they know they should not take, and exhibiting strange quirks we do not yet understand. These models would be primitive compared with superintelligence, yet we still cannot reliably align them.
We are running out of time to understand, control, and align these systems before they become vastly more capable. We must act now to confront these risks while we can still change course.
Good stuff from Edward and the DeepGrove guys!
Today, in the first of a series of blog posts we’re calling “Behind the Grove”, we’re offering a look into some of the technical work we do at DeepGrove.
Our first post is about our pretraining infrastructure, Grove. Grove is built around 2 backends: Grove-FSDP and DDP. We detail how we optimized both backends, culminating in an over 2x speedup over public baselines.
A production-scale pretraining run of Maple 20B-A1B (deepgrove.ai/maple-preview) on 4T tokens takes a mere 2 weeks on 64 H100s.
We view infrastructure as the backbone of our work, and as one of the many pieces that help us build toward the frontier of efficiency and performance.
To read the full post, visit deepgrove.ai/infra.
9
813
I think this is cool (I am personally excited to play around with this) but claiming that this model is in any means superhuman, the #1 Catan player, or the "AlphaGo" for Catan is just flat out wrong.
Colonist.io's rating forumla rewards players based on how often you play - if I play 10,000 games and win 51% of the time, my rating will be better than a player who plays 50 games but wins all of them. It doesn't mean I am the best player by any means.
These leaderboards also run for 3 months. Having an AI play a hundred games in the first three days of the leaderboard is a massive advantage compared to humans who are not awake 24/7, and would have only played a few dozen games (because they have to sleep and have lives).
This is why if you pull up the 1v1 leaderboards today, this model has already been passed by much better players who have a better winrate (71.9% vs 67.3%) with a similar number of games played (114 vs 107).
We're excited to unveil VINCO, our frontier AI model trained to play and win 1v1 Catan.
We've secretly deployed it on the Colonist.io global leaderboard, where it has reached both the #1 and #2 positions, consistently beating every human world champion.































