DevOnX retweeted
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
DevOnX retweeted
I often get emails from junior engineers asking me what to do.
Here's what I just wrote to one of them
DevOnX retweeted
CLAUDE FOUND THE SEEDS
anthropic.com/research/ecdsa…
DevOnX retweeted
Hi, I'm Felix. I'm an eng lead working on Claude. I love watching people take flight and do things they never thought possible.
We often describe that as "being more ambitious" when using AI, which is tricky advice to follow. I wrote a tutorial:
felixrieseberg.com/how-to-be…
Playing with this skill is fun 😎
Introducing Anatomy
Describe a machine in one line. Claude brings it to life.
- Interactive isometric wireframes you can rotate, inspect and explore
- WebGL shaders for fire, water, plasma and more
- Real 3D motion, with individual parts that rotate and move
Install it as an agent skill. Fully open source.
DevOnX retweeted
We're open-sourcing Rebalancer, the assignment-problem solver Meta has used for resource allocation across our infrastructure for over nine years.
Given a set of objects and a set of bins, Rebalancer assigns one to the other to optimize your objectives under your constraints, including hardware placement, service and task placement, traffic routing, and more. It separates how a problem is specified from how it's stored, solved, and debugged, with both an optimal MIP solver and a parallelized local search solver.
At Meta it solves ~40 million assignment problems a day across 30+ problem formulations. P99 solve time is 12 seconds on a problem with 265k objects and 3.2k bins.
🔗 Read more on our engineering blog: engineering.fb.com/2026/09/2…
Another tips & tricks from @karpathy
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
DevOnX retweeted
I’ve been reading the sandboxing arguments between infosec people and AI alignment folks, and I tried to summarize and referee them a bit in this post. blog.cryptographyengineering…
DevOnX retweeted
What is effort really? When do you change it it and why not just use max effort for everything?
I dove deep into this problem, looking into evals and doing my own tests and I was quite surprised by the results.
DevOnX retweeted
listen to me, imagine when Jev has vision/multimodal capabilities, the world will be exploded
Put Jev before the LLM:
Search -> Jev -> enough evidence?
NO -> search again
YES -> generate
Then put Jev after the LLM:
grounded? relevant? conflicting? confident?
If confidence is low, don’t return the answer.
Search more.
So in summary, this could be the standard:
RAG = KNOW
LLM = THINK
Jev = DECIDE
Tools = ACT
DevOnX retweeted
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
DevOnX retweeted
Habitat is OpenAI's online storage platform that powers everything from ChatGPT to Codex.
It has grown over 10x year over year. Before the Rust rewrite, its Python service handled more than 20 million requests per second at peak.
