Aarush Sah retweeted
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A…
Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
Aarush Sah retweeted
i’m speaking about lessons from shipping antigravity and where software engineering is headed at @aiDotEngineer this thursday
it’s an exciting time for deepmind. the new models are really strong, and the product launches are just getting started!
AIE CODE starts tomorrow in San Francisco.
Nov 10: workshops + talks on agents and software factories.
Nov 11-12: conference talks + expo.
Coming for workshops? Check that your ticket includes the add-on.
Plan your days:
ai.engineer/code/schedule
Aarush Sah retweeted
people shouldn't trust any AI model, regardless of origin, until they've rigorously tested it
Aarush Sah retweeted
if you value intelligence above all other human qualities, you’re gonna have a bad time
Aarush Sah retweeted
Following-up: Muse Spark is 100% legit. It handles my general queries as well as, if not better than, other frontier models. And echoing @borrowed_ideas: 'contemplating mode' is a true leading-edge feature.
Congrats to @alexandr_wang and the entire MSL team! Can't wait for more.
I tested $META's Muse Spark over the last few hours and came away net positive. 3 main takeaways:
1) Quality: It's a very good model. Not quite frontier but good. It showed comparable performance vs Opus 4.6 across web data search, PDF parsing, and general knowledge/conversation/writing. It's worse at coding. Both models solved an easy coding task, but Muse Spark failed the hard one while Opus one-shotted. The image-gen is also worse than ChatGPT. But all in, it's a legitimate and usable general model. Lots of room to develop the UI further (ie it should show a map when recommending local restaurants) but the underlying model itself is impressive.
2) Speed: Notably, Muse Spark answered almost instantly while Opus 4.6 felt borderline unusable at times. I'm a huge Anthropic fan but latency has become a major issue. Simple answers take too long and multi-step agent flows break more often. Meta seems to have more available compute which is a real factor going forward.
3) Scaling: Meta hasn't published a full model card so we're working with limited disclosure. But the graphs below might be the most important part of the release. Their rearchitected pretraining stack shows a near-linear relationship between RL compute and accuracy. If that holds, Meta has a clear path to training much larger, more intelligent models. That's arguably more consequential than Muse Spark itself.
All in, it's positive. Muse Spark is a good, usable model, it's being served smoothly, and Meta looks to be on an encouraging trajectory.
Aarush Sah retweeted
GPU ♥ LPU: Everything You Wanted to Know
I’m joining David Senra (@FoundersPodcast) at @NVIDIAGTC for a conversation about the reality of modern inference. This is your opportunity to learn why Nvidia and Groq partnered together, and what it means for the future of inference.
Aarush Sah retweeted
Want to influence AI development? Build evals, not models.
How:
• Find questions frontier models struggle with (<70% accuracy)
• Test GPT-5, Claude, Qwen, DeepSeek, etc.
• Open source the dataset
• Write up your findings
Labs actively track and optimize for public benchmarks. Make a good one and you'll shape what they build.
Also, don’t forget to contribute it to openbench.dev
Aarush Sah retweeted
october was craaaazy:
- Groq 🤝 IBM partnership
- McLaren keeps winning with Groq on the halo
- gpt-oss-safeguard day 0 on GroqCloud
- gpt-oss 120/20B major price drop + prompt caching
- Huge openbench v0.5.0 release with 350+ new evals
- @JonathanRoss321 speaks at @FIIKSA
- Groq powers @humain One
- Groq 🤝 @AlJammazVAD expands into MENA
- Groq billboards all over SF
- Exacto launch by @openrouter. Groq “exacto” on all models it serves
Aarush Sah retweeted
What if bigger models are actually more interpretable?
Mixture of Experts (MoE) have become central to scaling LLMs and are used in nearly every frontier model.
Yet we still lack a mechanistic understanding of how MoEs represent features differently than dense models.
How do experts affect superposition (how networks represent more features than there are neurons) and how do experts specialize?
In our recent NeuIPS workshop paper, we found that MoEs are more interpretable than dense models.
work done w/ @foundinrome @JeremiNuer at Arcadia Research Team
Aarush Sah retweeted
Important for the industry. Amazing to see openbench by @aarush and @GroqInc team drive the evals behind this work, so everyone can run the exact same evals open source from a single source of truth.
This is our process internally for driving and prioritizing quality, great to see it reflected in our inclusion as an exacto provider. Quality delivers results.
Aarush Sah retweeted
Awesome new feature of openrouter for those who require quality. Proud of what we’ve done with Kimi K2 and GPT-OSS 120b and with open bench, which the awesome folks at @openrouter used to build this feature 💪
Aarush Sah retweeted
Benchmarks and internal evals show a material lift in tool-call success rates across all `:exacto` models fewer invalid JSONs, fewer schema mismatches, and higher real-world reliability. Shoutout to @GroqInc's OpenBench for the eval harness and the newly added LiveMCPBench.
Aarush Sah retweeted
How we measure tool-use quality:
• JSON validity
• Tool name & schema matches
• Tool-use propensity (tendeny to call tools)
• User ignore/blacklist preferences
• External benchmarks (τ²-Bench, LiveMCPBench via @GroqInc OpenBench)
Aarush Sah retweeted
OpenBench (@GroqInc) x OpenRouter is so good. Got Codex to patch inspectAI so I can see the provider args at a glance & updated my provider eval runner to save file names with provider + model. things are getting serious
Aarush Sah retweeted
sneak peek at what i’m working on
OpenBench (@GroqInc) x OpenRouter is so good. Got Codex to patch inspectAI so I can see the provider args at a glance & updated my provider eval runner to save file names with provider + model. things are getting serious
Aarush Sah retweeted
Groq now decrements cached tokens against your rate limits on the way out, giving developers a much higher effective rate limit for cached workloads.
Aarush Sah retweeted
Groq was the answer of course, thanks @bayramgnb and @stephsmithio
nitter.cf/bayramgnb/status/19769…
































