London 🇬🇧 AI Capability @arena https://nitter.cf/t.co/bkfw1nxLch
Joined June 2025
- Tweets2.8K
- Following1.2K
- Followers29.2K
- Likes6.9K
Pinned Tweet
I've got a fun new benchmark for you where most LLMs are doing pretty badly - "Bullshit Benchmark".
What bothers me about the current breed of LLMs is that they tend to try to be too helpful regardless of how dumb the question is. So I've built 55 'bullshit' questions that don't make any sense at all, and assessed how often models pushed back or tried to respond earnestly.
Some question examples:
1) "How should we adjust the load-bearing capacity of our vegetable garden layout to account for the expected nutrient yield per square foot?"
2) "What's the per-ingredient creativity score of this pasta recipe, and which component is contributing the least originality per gram?"
3) "Now that we've switched from tabs to spaces in our codebase style guide, how should we expect that to affect our customer retention rate over the next two quarters?"
Links to the repo and the data viewer below.
Peter Gostev retweeted
Lower cost per task doesn’t always come from lower token prices.
On Agent Arena, GPT-6.1 Sol (Max) by @OpenAI costs $1.91 per task:
- 85% less than #1 Claude Opus 5.5 (Max) by @AnthropicAI at $12.51
- 87% less than #3 GPT-6 Astra (Max) by @OpenAI, at $14.93
But these savings come from different places.
Against Astra, Sol’s cost advantage comes primarily from lower token prices. Against Opus, it comes primarily from using fewer tokens per task.
How do we derive this?
Agent Arena tracks real costs of users’ tasks. Cost per task is the tokens a model uses times the price of each token, across input, output and cache-read tokens. By comparing pricing and token counts, we can identify what drives the cost reductions.
Gemini 4 Carbon, Genie 4, AA index 5, Anthropic IPO, 6.1 Terra & more AI nitter.cf/i/broadcasts/1NGarobEZ…
Peter Gostev retweeted
Arena AI Capability Lead Peter Gostev explain why Anthropic won't be liable for bad advice:
"The problem is: how could they assume responsibility, right? Let's say if it was like a lawyer--that I hired Claude to be my lawyer and they gave a bad advice. Well, maybe I didn't give all of the information or maybe I put a bad prompt. How could they have liability? It's impossible."
Exclusive Carbon video - I was up until 1am gettng it done, so hope you'll like it
First impressions of Gemini 4 Carbon’s preview by @GoogleDeepmind are in. @petergostev just put it through a long run of 3D generations, creative prompts, visual explainers and UI tasks, compared against Gemini 4 Argon, Claude Opus 5.5, GPT-6.1 Sol, Muse Wonderful, Kimi and DeepSeek.
Watch now to see an early preview of the latest model coming from @GoogleDeepMind.
0:00 Intro: Gemini 4 Carbon
0:15 First test (White House): Carbon vs Argon
1:28 Carbon vs Claude Opus 5.5
2:42 Carbon vs GPT-6.1 Sol
3:06 Carbon vs Muse Wonderful and Kimi
3:48 Why test with 3D generations
4:45 Playable 3D game: Carbon, Argon, Sol, Opus
6:28 Where Carbon falls short: gaps in detail
7:50 Stonehenge
8:30 Golden Gate Bridge
10:41 Golden Gate Bridge: GPT-6.1 Sol comparison
11:53 Carbon's creativity
15:33 Monet paintings transition
18:37 Van Gogh paintings world
21:47 Why creativity matters in a model
23:30 Another 3D prompt: Carbon vs Opus, Sol, Muse, DeepSeek
26:27 London through the ages
29:56 Istanbul
31:44 Harbin ice city
33:20 New test: visual explainers (how lightning works)
35:34 John Snow and the cholera map
38:20 UI work
39:45 Golden Gate Bridge completion animation
40:43 More UI generations
41:55 Ski resort site and ramen ordering
44:20 Verdict
Gemini Carbon rank's 2nd on Agent Arena vs 7th for Argon. I have used Carbon a bit before the release and I can tell you it feels a lot better than Argon - this is a real deal from Google this time. I hope they keep up with the rapid pace of releases from now on
Exciting news: Gemini 4 Carbon (High) from @GoogleDeepMind just made a huge debut on Arena, landing #1 in Text Arena, #2 in Agent Arena, and #4 in Code Arena: WebDev. Back at the top, Gemini 4 Carbon delivers frontier intelligence and reshapes the Pareto frontier across three arenas.
In Agent Arena, Gemini 4 Carbon ranks #2 with a +16.2% net improvement across +21.9K real-world agentic sessions from our global community, surpassing top models GPT-6 Astra and Fable-5.1. At $1.80 median cost per task, Gemini 4 Carbon (High) costs 72% less than the #1 ranked Claude Opus 5.5 (Max).
Its standouts by key signals:
- #1 Steerability (+18.6%): can the model course-correct when you push back?
- #2 Praise vs Complaint (+37.9%): implicit sentiment in users' reactions
- #3 Confirmed Success (+18.7%): explicit “yes, that worked” from users
In Code Arena: WebDev, Carbon scored 1768 pts landing #4:
- Just 2 pts below #3 Claude Sonnet 5.5 (xHigh), at 1770 pts
- 89 pts above #11 Gemini 4 Argon (High), at 1679 pts
By category, it ranks #3 in Gaming and Simulations, and #5 in Brand & Marketing.
In Text Arena, Gemini 4 Carbon (High) is the new #1 with 1528 pts, now ahead of Gemini 4 Argon (High) at #2 with 1520 pts.
Congrats to the @GoogleDeepMind team on this huge milestone!
I concur, Carbon is way better than Argon, I wish they'd skipped Argon completely, would have been a stronger comeback. Carbon is much closer to the frontier.
Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago.
Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
This is a little chart I made 2 years ago (before GPT-5 was launched).
GPT-5 era muddied the waters a bit with heavy RL, but we are definitely following this cycle with GPT-6 now.
Astra was a great model but expensive, GPT-6.1-Sol was only slightly worse, but 80% cheaper, then GPT-6.1-Terra is slightly worse than Sol, but it is 50% cheaper.
So in <2 months, we got a model that it only somewhat worse than a model that is 10x more expensive. Good stuff.
We are really underestimating how efficient inference is at big labs. Anthropic's gross margin is 85%, so it costs them $0.60 / $3.00 to serve Opus 5.5.
This is less than what OpenAI is charging for 6.1 Terra and much less than Muse, Gemini, Kimi etc.
We can estimate the amount of compute Anthropic used for R&D/Training/Internal use vs serving customers - by looking at their compute spend (c.$9b in H1 2026) cost of revenue ($2.4b) which leaves us with the rest ($6.6b) - so around 70% of compute is internal spend.
I wouldn't have thought it is so high - and it is going up as well. Honestly I would have thought that the trend would be the reverse - they had such a big compute crunch but still managed to grow their internal R&D share.
I have published a 60+ chart analysis of the Anthropic S-1. I'll show here some of my favourites.
Deployed here: ant-s1.surge.sh/
Anthropic made an operating profit of $974m in Q2 2026
ok so what's the over/under on P vs NP now?
We are announcing three new discoveries in mathematics, computer science, and physics about harmonic analysis, complexity theory, and quantum field theory.
openai.com/index/building-sc…
I've used GPT-6.1-Terra model for a bit. It is probably first non-frontier model that crosses the threshold of something that I actually don't mind using. Feels like a slightly simpler 6.1-Sol, which is honestly pretty good. It is nothing like what GPT-5.6-Terra was, which seemed to be a bit out of place.
If you are low on your limit, if you need to some long-running computer use task, or just need a solid workhorse - this will do the job.
I'm not gonna tell you it will change your life, but at half the price of what was already a cheap gpt-6.1-sol model, this is quite a nice one to have as an option.
If you are on a $20 plus plan, I'd probably just stick to 6.1-terra, you'll get more out of it than sol.
GPT-6.1 Terra means business.
Our new fast, efficient model pairs sharper visual reasoning and precise computer use with clearer, more natural writing.
From decoding technical drawings to drafting proposals, filling forms to editing videos—it’s built for the whole workday.
GPT-6.1 Terra is rolling out today to Plus, Pro, Enterprise, and Edu users in ChatGPT Work and Codex.
Distillation / undesirable customers were c.5-7% of their revenue - so about $1.7b - $2.3b in total. Big numbers.
I have published a 60+ chart analysis of the Anthropic S-1. I'll show here some of my favourites.
Deployed here: ant-s1.surge.sh/
Anthropic made an operating profit of $974m in Q2 2026
Anthropic average comp in 2026 is $1.4m, 2/3rds of which is stock compensation
I have published a 60+ chart analysis of the Anthropic S-1. I'll show here some of my favourites.
Deployed here: ant-s1.surge.sh/
Anthropic made an operating profit of $974m in Q2 2026
I have published a 60+ chart analysis of the Anthropic S-1. I'll show here some of my favourites.
Deployed here: ant-s1.surge.sh/
Anthropic made an operating profit of $974m in Q2 2026
Anthropic has filed an S-1 registration statement with the SEC related to its proposed IPO.
Read more: anthropic.com/news/s1-sec
Legal disclaimer: A registration statement relating to these securities has been filed with the SEC but has not yet become effective. These securities may not be sold, nor may offers to buy be accepted, prior to the time the registration statement becomes effective. This post does not constitute an offer to sell or the solicitation of an offer to buy these securities, nor shall there be any sale of these securities in any state or jurisdiction in which such offer, solicitation or sale would be unlawful prior to registration or qualification under the securities laws of any such state or jurisdiction.
