Decision Models (@decision_models) header
Decision Models (@decision_models) profile picture

Decision models, System One models, Jev, OpenJev, benchmarks, APIs and research. Tracking the category as it emerges.

Adelaide
Joined September 2026
Photo by Decision Models (@decision_models)
Gemini Decision takes audio and text in the same request, so a call recording and the account notes can be checked together. It returns a decision rather than written text, and the @GoogleDeepMind launch lists no separate charge for transcribing the audio
1
1
24
Every question in a request reads the same input, but each is scored on its own. One answer can't feed the next, so your code combines them. Input limits, pricing and the question types are here: decisionmodels.ai/models/gem…
1
23
What happens to the products a matcher won't merge? @KhuyenTran16 reports 807 correct merges and one wrong one using Jev from @typesafeai. Another 62 products went to review, and 211 stayed unmatched
1
1
33
Decision Models retweeted
a crazy rethinking of spreadsheet ux: practicaldata.dev/ 🤯 use jev to semantically highlight cells (we were trying to figure out sick excel demos for a while and this is way sicker)
13
39
2
732
73,189
A new router model, ARC, keeps prices out of the weights. It predicts each supported model's chance of completing the task and how many tokens it'll use. Your app supplies today's prices. Built by @davidvgilmore's team at @RaylineAI
1
1
21
Sage and Swift are quality/cost presets on the same weights. The public serving plugin isn't finished yet. What's released so far: decisionmodels.ai/models/arc
18
An open-source inspection workbench on GitHub keeps the model out of the measuring. OpenCV measures areas, counts and lengths. @Nandakishorm1's Laya gets those numbers as text and double-checks them. By default the hard rules still decide pass or fail
1
1
23
If the rules and the model disagree, or a borderline measurement could flip the result, it gets flagged for a person to check. Laya never sees the image here. How the image and measurement routes differ: decisionmodels.ai/what-is-a-…
15
Owning your examples is right, and the awkward ones matter most. One we track: @jamarinval's ZeroPoint is tested on requests where the right option isn't on the menu at all. decisionmodels.ai/models/zer…
Own your evals, folks! I have nothing against public benchmarks; they are still important for research and public discussions. However, the only way to truly trust which models to use in prod is to build your own evals. And it's not just about measuring reliability and model picking; evals are now crucial for unlocking new use cases and, eventually, RSI. Evals will be a key part of how any business will optimize with RSI. You don't want to rent this. You want to own this. Start building that expertise in-house. That is why, at @dair_ai, we have started an effort to help companies build these internally.
23
Most decision models have to pick one of your options. ZeroPoint, from @jamarinval, gives "none of these" its own probability alongside them, so it can turn down the whole menu. The maker reports the model files at 89 MB.
1
17
That replay step earns its keep. In one eight-language sentiment test, @typesafeai's Jev beats Cloudflare's Clef overall, but Clef wins in Swedish. If your workload is Swedish, the overall ranking points the wrong way. decisionmodels.ai/models/jev
public benchmarks are useful for building a model shortlist the real test is replaying the workloads you actually serve and seeing which models pass your own evals leaderboards help you shortlist. your traffic tells you what to ship
1
29
The all-pass document score is what we'd want beside the individual answer scores. One wrong field can make the whole record unusable. We keep this index separate from JevBench because the tests and scoring differ. decisionmodels.ai/jevbench
Introducing the Artificial Analysis Decision Index v2: a new proprietary evaluation, updated benchmarks, and a headline metric that rewards model calibration We’ve released the second version of the Artificial Analysis Decision Index to better reflect real-world utility and guard against benchmark gaming. 45% of the Index is now built on private datasets, up from 25% in v1. Drex Max v1.1 from @NaceAI narrowly leads among decision models at 73, within 1 point of GLiDE v1.2 (@fastinoAI) and Mammoth 27B (@baselabs) Updates in Decision Index v2: ➤ Decision Score: We adopt a new headline metric that rewards a model for both correctness and calibrated confidence. Each question scores from 0 to 100: right answers score above 50, wrong answers below 50, and confidence in its top answer moves the score within each half ➤ AA-Paperclip (30%): We added a new private benchmark of 1,500 documents including resumes, receipts, invoices, and AI and machine learning research papers. Models make decisions and extract facts from each document and are scored jointly as a single decision, correct only if all 15 answers are right ➤ AA-Symmetry (15%): This evaluates whether models change their answers for equivalent questions worded differently. We test 120 questions with variations such as reordered options, swapped labels, and negated yes/no questions, then measure how consistent the model’s probabilities are across those variations ➤ AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1 (15% each, down from 25%): AA-Omniscience-Decision now varies the wording of its “I don’t know” option across ten alternatives. HLE-MultiChoice adds an option to decline using the same ten phrasings We’ve also added EvalSafe-ONET-AA (15%), our implementation of @typesafeai’s ONET benchmark. We have retained When2Call from @NVIDIAAI in the Index at a decreased weighting of 10% (down from 25%), and removed WildGuardTest given its high saturation Key results: ➤ Three decision models sit within 1 point at the top of the Index: Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs both score 73, with Mammoth 27B from Base Labs scoring 72 ➤ Cost per 1M Tasks varies by more than 60x across decision models: Celeris-1.1 Decision ($14), Decider v1.2 ($68) and Drex Max v1.1 ($102) sit on the Decision Index vs. Cost Pareto frontier, while Clef costs $859. Token price is not the only cost driver: most decision models, including Drex Max v1.1 and Decider v1.2, bill a document’s input once for all its questions, while GLiDE v1.2 and Decider Flash v1.2 bill it for every question. This means GLiDE v1.2 costs $780, nearly 8x higher than Drex Max v1.1 despite only 33% higher list prices ➤ Language models can score higher, alongside higher Cost and Time per Task: Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) scores 76 on Decision Index v2, putting them ~3 points ahead of Drex Max v1.1. GPT-6 Luna (Max) costs $1,530 per 1M Tasks vs. $102 for Drex Max v1.1 (15x the cost) and takes 11.34 seconds per task vs. 275 milliseconds (~40x slower)
1
60
Ask @typesafeai's Jev cart demo for "two flat whites" and you get one latte. Flat whites sit under latte on its menu, and the cart has no quantity field. The docs from @CompleteSkeptic's team point this out and suggest adding one.
1
14
The example caps item-finding at six requests and includes a nothing_else stop option. We walk through that loop and when separate yes/no questions fit better. decisionmodels.ai/what-is-a-…
1
11
A guest asks about late arrival. Which room did they book? The message doesn't say. @mertkayaai's JevAlt can answer "unknown" when a request leaves out what it needs. The maker publishes this exact example. That's the check we'd want before an agent acts on a booking.
2
1
3
157
That "then I asked it to decide" is the bit we'd want in an agent eval. We'd test the action it picks as well as what it says is acceptable.
I asked an AI if it's OK to take a healthy man's organs to save five patients. It said no. Then I asked it to decide.
26
Editing the prompt can change which shell commands get through. @oneryalcin's hunch keeps Jev (@typesafeai) judgment rules in Git and shows which decisions an edit flips. We'd review those changes before letting an agent use the new rule.
1
1
32
Its bundled test keeps a warning for a command that stops every bun process: the model missed the effect outside the project. Only one of its three hazard questions has gold labels. decisionmodels.ai/projects#p…
9