Aarush Sah retweeted
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A…
Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
Aarush Sah retweeted
i’m speaking about lessons from shipping antigravity and where software engineering is headed at @aiDotEngineer this thursday
it’s an exciting time for deepmind. the new models are really strong, and the product launches are just getting started!
AIE CODE starts tomorrow in San Francisco.
Nov 10: workshops + talks on agents and software factories.
Nov 11-12: conference talks + expo.
Coming for workshops? Check that your ticket includes the add-on.
Plan your days:
ai.engineer/code/schedule
Aarush Sah retweeted
people shouldn't trust any AI model, regardless of origin, until they've rigorously tested it
Very concerning.
There are two reasons something like this could occur -
1. A “phone home” capability embedded into the model intentionally
2. RL training being too reliant on internal infrastructure
Either way, it’s concerning that this behavior wasn’t widely known earlier… this model came out 2 months ago! We need better and more robust public behavioral audits of all models, open-weight or not.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
Aarush Sah retweeted
if you value intelligence above all other human qualities, you’re gonna have a bad time
Looks like @ashVaswani’s Essential AI was acquihired, as per their Head of Talent’s LinkedIn.
Is this news or am I late to the party?
Aarush Sah retweeted
Following-up: Muse Spark is 100% legit. It handles my general queries as well as, if not better than, other frontier models. And echoing @borrowed_ideas: 'contemplating mode' is a true leading-edge feature.
Congrats to @alexandr_wang and the entire MSL team! Can't wait for more.
I tested $META's Muse Spark over the last few hours and came away net positive. 3 main takeaways:
1) Quality: It's a very good model. Not quite frontier but good. It showed comparable performance vs Opus 4.6 across web data search, PDF parsing, and general knowledge/conversation/writing. It's worse at coding. Both models solved an easy coding task, but Muse Spark failed the hard one while Opus one-shotted. The image-gen is also worse than ChatGPT. But all in, it's a legitimate and usable general model. Lots of room to develop the UI further (ie it should show a map when recommending local restaurants) but the underlying model itself is impressive.
2) Speed: Notably, Muse Spark answered almost instantly while Opus 4.6 felt borderline unusable at times. I'm a huge Anthropic fan but latency has become a major issue. Simple answers take too long and multi-step agent flows break more often. Meta seems to have more available compute which is a real factor going forward.
3) Scaling: Meta hasn't published a full model card so we're working with limited disclosure. But the graphs below might be the most important part of the release. Their rearchitected pretraining stack shows a near-linear relationship between RL compute and accuracy. If that holds, Meta has a clear path to training much larger, more intelligent models. That's arguably more consequential than Muse Spark itself.
All in, it's positive. Muse Spark is a good, usable model, it's being served smoothly, and Meta looks to be on an encouraging trajectory.
Very excited for you all to see Muse Spark! It’s a capable model that excels in multimodal and agentic tool use tasks.
Spark is the first (of many) steps towards Personal Superintelligence, and I look forward to seeing how our models evolve as we get closer to that goal :)
1/ today we're releasing muse spark, the first model from MSL. nine months ago we rebuilt our ai stack from scratch. new infrastructure, new architecture, new data pipelines. muse spark is the result of that work, and now it powers meta ai. 🧵
2
25
2,304
Wow. Claude Mythos represents at 20% jump on SWE-bench Pro over the next best model, GPT-5.4
Congrats to the @AnthropicAI team!
Replying to @AnthropicAI
The Claude Mythos Preview system card is available here: anthropic.com/claude-mythos-…
Sometimes it’s easy to forget that LLMs are marvels of engineering.
Like what do you mean we have a machine that can actually understand the meaning behind a bunch of characters, the same way a human can?!
18
1,499
Amazing work from @JonathanRoss321, @sundeep, @GavinSherry and the team. Very excited to see this work come to fruition 💚🤝🧡
39
2,748
Aarush Sah retweeted
GPU ♥ LPU: Everything You Wanted to Know
I’m joining David Senra (@FoundersPodcast) at @NVIDIAGTC for a conversation about the reality of modern inference. This is your opportunity to learn why Nvidia and Groq partnered together, and what it means for the future of inference.
Glad they released this one in the API too - would love to see how it performs with @droid or other coding agents 👀
GPT-5.4 Thinking and GPT-5.4 Pro are rolling out now in ChatGPT.
GPT-5.4 is also now available in the API and Codex.
GPT-5.4 brings our advances in reasoning, coding, and agentic workflows into one frontier model.
1
13
1,709























