Aarush Sah (@aarush) header
Aarush Sah (@aarush) profile picture

20. Staff SWE @Meta Superintelligence Labs, @NVIDIA LPU, @GroqLLC Head of Evals Created openbench

Menlo Park, CA
Joined September 2022
Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)Photo by Aarush Sah (@aarush)
Aarush Sah retweeted
Replying to @benklieger
Evals are the new software engineering @alexgshaw
1
2
34
1,623
Aarush Sah retweeted
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A… Some observations: - HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption - Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark) - Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor - Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔 - In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA) - DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it - Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
84
81
30
688
71,037
Aarush Sah retweeted
i’m speaking about lessons from shipping antigravity and where software engineering is headed at @aiDotEngineer this thursday it’s an exciting time for deepmind. the new models are really strong, and the product launches are just getting started!
AIE CODE starts tomorrow in San Francisco. Nov 10: workshops + talks on agents and software factories. Nov 11-12: conference talks + expo. Coming for workshops? Check that your ticket includes the add-on. Plan your days: ai.engineer/code/schedule
6
8
2
72
6,887
Replying to @irl_danB @xeophon
people shouldn't trust any AI model, regardless of origin, until they've rigorously tested it
4
4
41
4,219
if you value intelligence above all other human qualities, you’re gonna have a bad time
898
3,523
1,180
24,370
9,970,796
Following-up: Muse Spark is 100% legit. It handles my general queries as well as, if not better than, other frontier models. And echoing @borrowed_ideas: 'contemplating mode' is a true leading-edge feature. Congrats to @alexandr_wang and the entire MSL team! Can't wait for more.
I tested $META's Muse Spark over the last few hours and came away net positive. 3 main takeaways: 1) Quality: It's a very good model. Not quite frontier but good. It showed comparable performance vs Opus 4.6 across web data search, PDF parsing, and general knowledge/conversation/writing. It's worse at coding. Both models solved an easy coding task, but Muse Spark failed the hard one while Opus one-shotted. The image-gen is also worse than ChatGPT. But all in, it's a legitimate and usable general model. Lots of room to develop the UI further (ie it should show a map when recommending local restaurants) but the underlying model itself is impressive. 2) Speed: Notably, Muse Spark answered almost instantly while Opus 4.6 felt borderline unusable at times. I'm a huge Anthropic fan but latency has become a major issue. Simple answers take too long and multi-step agent flows break more often. Meta seems to have more available compute which is a real factor going forward. 3) Scaling: Meta hasn't published a full model card so we're working with limited disclosure. But the graphs below might be the most important part of the release. Their rearchitected pretraining stack shows a near-linear relationship between RL compute and accuracy. If that holds, Meta has a clear path to training much larger, more intelligent models. That's arguably more consequential than Muse Spark itself. All in, it's positive. Muse Spark is a good, usable model, it's being served smoothly, and Meta looks to be on an encouraging trajectory.
1
12
1
119
22,916
LPU in Ian Buck's hand, sitting in the audience at GTC
9
22
318
17,817
GPU ♥ LPU: Everything You Wanted to Know I’m joining David Senra (@FoundersPodcast) at @NVIDIAGTC for a conversation about the reality of modern inference. This is your opportunity to learn why Nvidia and Groq partnered together, and what it means for the future of inference.
16
40
11
440
69,587
Aarush Sah retweeted
you can just do things
1,479
2,908
755
24,563
3,002,571
Want to influence AI development? Build evals, not models. How: • Find questions frontier models struggle with (<70% accuracy) • Test GPT-5, Claude, Qwen, DeepSeek, etc. • Open source the dataset • Write up your findings Labs actively track and optimize for public benchmarks. Make a good one and you'll shape what they build. Also, don’t forget to contribute it to openbench.dev
1
1
7
941
Aarush Sah retweeted
october was craaaazy: - Groq 🤝 IBM partnership - McLaren keeps winning with Groq on the halo - gpt-oss-safeguard day 0 on GroqCloud - gpt-oss 120/20B major price drop + prompt caching - Huge openbench v0.5.0 release with 350+ new evals - @JonathanRoss321 speaks at @FIIKSA - Groq powers @humain One - Groq 🤝 @AlJammazVAD expands into MENA - Groq billboards all over SF - Exacto launch by @openrouter. Groq “exacto” on all models it serves
8
9
3
134
12,321
Aarush Sah retweeted
What if bigger models are actually more interpretable? Mixture of Experts (MoE) have become central to scaling LLMs and are used in nearly every frontier model. Yet we still lack a mechanistic understanding of how MoEs represent features differently than dense models. How do experts affect superposition (how networks represent more features than there are neurons) and how do experts specialize? In our recent NeuIPS workshop paper, we found that MoEs are more interpretable than dense models. work done w/ @foundinrome @JeremiNuer at Arcadia Research Team
11
16
4
56
15,175
Important for the industry. Amazing to see openbench by @aarush and @GroqInc team drive the evals behind this work, so everyone can run the exact same evals open source from a single source of truth. This is our process internally for driving and prioritizing quality, great to see it reflected in our inclusion as an exacto provider. Quality delivers results.
Introducing `:exacto`, Precision Tool-Calling Endpoints OpenRouter now offers a curated subset of providers on top open source models with measurably higher tool calling accuracy delivering more reliable tool use.
3
8
31
9,462
Aarush Sah retweeted
Awesome new feature of openrouter for those who require quality. Proud of what we’ve done with Kimi K2 and GPT-OSS 120b and with open bench, which the awesome folks at @openrouter used to build this feature 💪
Introducing `:exacto`, Precision Tool-Calling Endpoints OpenRouter now offers a curated subset of providers on top open source models with measurably higher tool calling accuracy delivering more reliable tool use.
2
6
1
30
9,910
Benchmarks and internal evals show a material lift in tool-call success rates across all `:exacto` models fewer invalid JSONs, fewer schema mismatches, and higher real-world reliability. Shoutout to @GroqInc's OpenBench for the eval harness and the newly added LiveMCPBench.
1
2
22
3,185
How we measure tool-use quality: • JSON validity • Tool name & schema matches • Tool-use propensity (tendeny to call tools) • User ignore/blacklist preferences • External benchmarks (τ²-Bench, LiveMCPBench via @GroqInc OpenBench)
2
4
35
3,437
OpenBench (@GroqInc) x OpenRouter is so good. Got Codex to patch inspectAI so I can see the provider args at a glance & updated my provider eval runner to save file names with provider + model. things are getting serious
4
5
1
50
14,477
sneak peek at what i’m working on
OpenBench (@GroqInc) x OpenRouter is so good. Got Codex to patch inspectAI so I can see the provider args at a glance & updated my provider eval runner to save file names with provider + model. things are getting serious
5
3
1
34
6,471
Aarush Sah retweeted
Groq now decrements cached tokens against your rate limits on the way out, giving developers a much higher effective rate limit for cached workloads.
3
3
1
20
2,496
Aarush Sah retweeted
Groq was the answer of course, thanks @bayramgnb and @stephsmithio nitter.cf/bayramgnb/status/19769…
Replying to @levelsio
Have you tried @GroqInc models? It’s insanely fast and cheap. There’s plenty of models like different versions Llama, Kimi, and gpt-oss You don’t even need to add credit card to try them
9
6
62
46,293