PhD student, Center for High Energy Physics, IISc Bangalore | Non-perturbative QFT • Tensor Networks • Cosmology | AI x Science
Joined June 2019
- Tweets1.3K
- Following577
- Followers1.2K
- Likes16.1K
One way to check AI alignment might be to test for semantic drift in LEAN. As models get better, the scores will tell us more about which ones are honest and which ones are not (in tightly controlled experiments).
Idk, might be interesting.
General lesson from the last few days: Writing large prompts where you really pour out everything you have (both what you understand and feel) really pays off.
CritPt v2 has finally been released in the recent major overhaul of the @ArtificialAnlys Intelligence Index v5.
The older version had many errors which were first reported in the Fable 5.1 system card.
more observations below:
Apologies - I missed this discussion in the Fable 5.1 system docs!
Section 8.9, “CritPT-Corrected” in Anthropic’s Claude Fable 5.1 & Claude Mythos 5.1 System Card (Sept 1, 2026). Pages 168–171.
The revised benchmark cleans pervasive ambiguities and errors found in the original problems to evaluate advanced postgraduate scientific reasoning more accurately.
gpt-6.1 Terra and gpt-6.1 Sol on the new CritPt v2 (hard). waiting for Astra now to fill this plot.
The non-trivial jump from non-reasoning to reasoning in gpt-6.1-Terra is explained here:
The labels on the AA chart are rounded to the nearest thousand, so “2k” and “2k” could be quite different numbers, say 1.6k against 2.4k. But even with exact numbers, the jump would make sense, because the way the tokens are spent changes between the two settings.
In the non-reasoning bar, the whole 2k is answer tokens: the model writes its setup, code and final expression straight away, committing to an approach in the first few lines. The low-effort bar is split between a short reasoning segment and the answer. On research-level problems, the expensive mistake is usually early and conceptual: the wrong limit, the wrong boundary condition, a sign convention, missing a symmetry. A few hundred tokens of planning before committing can catch that, and once the approach is right the rest is mostly mechanical. So a small reasoning budget can buy a lot, while the same number of tokens spent writing out an answer does not. The gap is 16 points, about 6 of the 40 questions, so it is not noise.
There is also a second effect. “Non-reasoning” and “low” are probably not two settings of one dial. They are likely different modes of the model, trained differently, with the reasoning mode trained specifically to get answers right. Comparing them by token count assumes the model is the same and only the budget changes, and that assumption may not hold here.
"higher model capability is associated with greater reasoning faithfulness, with several strongest models approaching near-perfect faithfulness"
This is one of the most important metrics to test for right now and I'm glad the results are positive here.
My student, Gongrui Cheng, wrote a benchmark paper with his friends! arxiv.org/abs/2610.33134
Chayanka_42 retweeted
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
Chayanka_42 retweeted
I see how some people can watch flashy robot demos on the timeline and think with a shiver that the robot apocalypse is around the corner, but reality check:
There is no robot in the world that can change the batteries in a jack-o-lantern.
If you have a robot friendly door handle it might be able to walk outside, pick it up, and carry it back inside. The robot could open the drawer with your batteries in it, but then things get tough.
Opening the package of AA batteries?
Extracting a precision screwdriver from its case?
Unscrewing four tiny screws that require a hard press and twist?
Popping batteries out of the case?
Yes, you could build and train a specialist robot to do that today, but there is a very long way to go to replace the full range of human capabilities. Near term doomer scenarios should stop at “secretly controlling humanity” instead of “wiping humanity out” 😀
The Yang-Mills existence proof is 1147 pages... Holy 🤯
Statement of Yang-Mills Working Group, a scientific collaboration to interpret AI result related to the existence of quantum chromodynamics & completions of the Standard Model, including a subproblem proving the existence of the Quantum Yang-Mills theory. buff.ly/BwV89Tk
