Introducing the Artificial Analysis Decision Index, benchmarking decision models across classification, tool calling, knowledge and reasoning. Jev leads among decision models; language models such as Claude Haiku 5.5 still deliver higher accuracy, but at much higher cost and latency Decision models deliver structured answers or classifications with associated probabilities, offering faster responses and lower costs than using language models. The Artificial Analysis Decision Index v1.0 compares the capability, cost, and speed of decision models and also compares to language models. Benchmarks in the Artificial Analysis Decision Index v1.0: ➤ AA-Omniscience-Decision: A private dataset of 500 questions adapted from the proprietary AA-Omniscience benchmark, measuring decision models’ knowledge and tendency to hallucinate ➤ HLE-MultiChoice: 513 text-only multiple-choice questions from Humanity’s Last Exam, testing decision models’ reasoning capabilities ➤ When2Call: A tool-calling benchmark from NVIDIA, testing whether models can correctly decide to call a tool, request clarification, decline to answer, or respond directly, evaluating capabilities relevant to agentic workflows and tool routing ➤ WildGuardTest: A safety benchmark evaluating models’ classification capabilities across harmful user requests, assistant refusals, and harmful responses. We use a subset of 835 of the original 1,725 test rows where all three human annotators agreed on all three labels, after excluding questions that consistently trigger model refusals Key results: ➤ Jev 1.13 leads the Decision Index, and sits on the Capability vs. Cost Pareto frontier: Jev 1.13 from @typesafeai scores 58 on the Artificial Analysis Decision Index, ahead of GLiDE (54) from @fastinoAI and Microsoft-Decision-1 (54) from @Microsoft. Decider v1.1 27B from @perplexity_ai has the lowest Cost per 1M Tasks at $17, with Microsoft-Decision-1 ($31) and Jev 1.13 ($42) also on the Capability vs. Cost Pareto frontier. Celeris-1 Decision has the fastest Time per Task at 0.06 seconds, more than 2x faster than Jev 1.13 (0.13 seconds per task) ➤ Decision models are poorly calibrated on knowledge and reasoning tasks: Across almost all probability levels for chosen answers, accuracy falls below that probability for all decision models. This gap is largest on AA-Omniscience-Decision and HLE-MultiChoice, while most models are relatively well-calibrated on WildGuardTest (classification) and When2Call (tool calling) ➤ Decision models offer significant speed and cost advantages over language models, but trail on knowledge and reasoning: On classification and tool-calling tasks, leading decision models perform comparably to language models while being significantly faster and cheaper. However, language models substantially outperform on knowledge and reasoning. Claude Haiku 5.5 (Low) scores 61 on the Decision Index, 3 points ahead of Jev 1.13, but takes 1.4 seconds per task vs. 0.13 seconds (11x slower) and costs $184 per 1M Tasks vs. $42 for Jev 1.13 (4.4x more expensive) We will continue to develop the Artificial Analysis Decision Index to help developers and companies better understand when and how to use decision models.

We're launching our leaderboard for FrontierCode 1.1, a private evaluation from Cognition that tests models on completing coding tasks in a way that would actually get merged by maintainers of real-world repositories @cognition announced FrontierCode in June 2026 and updated it to version 1.1 in July. Cognition built the dataset with open source maintainers who spent more than 40 hours per task. Each task requires the agent to make a change to a real open source repository, then checks both unit tests and qualitative tests for what maintainers would require to merge the change. We run FrontierCode 1.1 Main, the primary 100-task set of the benchmark's 150 tasks, using our Stirrup open source reference harness for apples-to-apples comparisons across models. All tasks are private to avoid contamination. Each attempt is graded with both blocking and non-blocking criteria. An attempt that fails any blocking check scores zero, and one that passes them all scores between 70% and 100%. Unlike other coding evaluations, FrontierCode has a more holistic view of success based on real usage. Models need to pass unit tests, but are also assessed on behaviors like making excessive changes and large diffs, modifying files outside the relevant scope, or adding unhelpful tests. We think this approach is valuable in the current context: models are now very capable at coding tasks, but their changes can still require substantial cleanup before a maintainer would merge them. Key takeaways: ➤ Leading models still fail more than 40% of the time: Claude Opus 5.5 (max with fallback) leads at a score of 52% after passing blockers in 58% of its attempts. Nine of the 100 tasks were not passed in any attempt by any of the models we evaluated for launch ➤ The top of the leaderboard is tightly packed: nine models from six labs score between 43% and 52%. MiMo-V2.6-Pro is the leading open weights model at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44% ➤ GPT-6.1 Sol (max) offers the strongest performance for its cost: At $0.81 per task it scores 49%, matching Claude Fable 5.1 (max with fallback). This means it achieves similar performance to leading Anthropic models, at less than a tenth of the cost per task Thank you to @silasalberti, @ybenpan, @cl571128, and the rest of the team at Cognition for building FrontierCode and collaborating with us to bring it to Artificial Analysis - we’re working to add FrontierCode to our coding agent leaderboard very soon.

We gave four frontier image editing models the same photo and 30 edits in a row: Ideogram 4.5 keeps most of the room intact, while the others drift, GPT Image 2.5 Sunburst most visibly. We've seen some interesting demos of multi-turn editing consistency from the latest image editing models, so we ran our own test: GPT Image 2.5 Sunburst, #1 on our Image Editing leaderboard, against Ideogram 4.5, FLUX 3 and Nano Banana 2.1. Our leaderboard scores single edits; this tests what happens when 30 consecutive changes stack up, with each model editing its own previous output through a 30-step real estate staging sequence: light the fire, add a sofa, repaint the walls, swap day for twilight, and more. Why are the final results so different? @ideogram_ai's Ideogram 4.5 and @bfl_ml's FLUX 3 edit locally. On small edits like adding a vase of tulips, we measured that they left 95% or more of the image essentially untouched. GPT Image 2.5 (Sunburst) re-renders most of the entire image on every edit, leaving only about a fifth of the image unchanged, so small shifts in colour and detail compound over turns. Nano Banana 2.1 sits in between: its edits stay local, but the rest of the image shifts slightly and gradually darkens.