On this page
- What does DecideBench measure, and why can a frontier LLM solve it?
- What do the 450 decisions look like?
- Why do contrastive pairs matter?
- How are cost and latency measured?
- What is a Pareto frontier, and which models sit on it?
- What changed from v1 to v2?
- What are DecideBench's limitations?
- How does DecideBench compare with other decision model benchmarks?
- How do you reproduce DecideBench or add a model?
- FAQ
- Sources
DecideBench v2 is a System One model benchmark published on 9 November 2026, with open data (CC BY 4.0) and open code (MIT). It measures accuracy, cost per decision and latency for 26 System One models on 450 decisions written in production input format. Instead of one ranking, it reports Pareto frontiers: the models no other model beats on both price and accuracy, or on both speed and accuracy.
A System One model, or decision model, reads an input, a question and a short list of options, and returns one option key. TypeSafe coined the term when it launched Jev, its own decision model, on 15 September 2026, and what is a System One model covers the category. For the picks the results lead to, read the best System One models of 2026.
- What: 450 multiple-choice decisions in 225 contrastive pairs, 9 task families, 3 to 10 options each.
- Who made it: Cho Yin Yong at XY Space. XY Space also makes the Yev models and runs AI Space.
- Reference: Claude Opus 4.8 scores 99.3% at $26,542.70 per million decisions and 1,336 ms.
- Most accurate decision model: Drex Max (Nace.AI), 94.4% at $91.00 per million and 258 ms.
- Where to check: the GitHub repo, the leaderboard and the dataset.
What does DecideBench measure, and why can a frontier LLM solve it?
DecideBench measures how close a cheap decision model gets to a frontier LLM on decisions the LLM already gets right. Claude Opus 4.8 answers 447 of the 450 items correctly, so the open question is price.
Building the test around a model that already passes it puts price at the centre. Opus 4.8 costs about $26,500 per million decisions and takes 1.3 seconds each. The benchmark asks how close far cheaper models get to it.
Drex Max, the closest, costs about 1/290 as much per decision. It answers in about a fifth of the time. It gets 94 of every 100 right, where Opus 4.8 gets 99.
Every entry gets four numbers:
| Metric | What it is |
|---|---|
| Accuracy | Share of the 450 items answered correctly |
| Pair accuracy | Share of the 225 pairs with both halves right |
| Cost per 1M tasks | What a million decisions cost at the measured rate |
| Latency p50 | Median time per request, 4 requests in flight |
Claude Opus 4.8 is the reference the items were built against, not a decision model, and it does not compete in the decision-model results.
What do the 450 decisions look like?
Each item is a decision a piece of software would face in production, written in the format it would arrive in. There are 50 items in each of nine families:
| Family | Typical input |
|---|---|
| Approving an AI agent's proposed action | The agent's proposed tool call as JSON |
| Routing an analytics agent to a tool | The conversation so far and the data on hand |
| Checking a drafted claim against its sources | A retrieval result with its sources |
| Moderating a post | The post and the written policy |
| Ruling on a return | Order dates, amounts and the returns policy |
| Gating a staged software release | A release controller's metrics and rules |
| Reading a customer's intent | A customer message |
| Triaging an inbound email | A raw email with headers |
| Triaging a bug report | A bug report |
The inputs are what the model would see at runtime: a JSON payload, a raw email with its headers, a release controller's metrics, a retrieval result with its sources. Where the decision depends on a policy, the full written policy sits in the question. The median input runs to 893 characters, and items offer between 3 and 10 options.
The two halves of each pair differ by one small, realistic edit that changes the right answer. In one routing pair, the data the agent needs is a file attached to the chat in one half and a database table in the other, so the right tool changes. A model that answers from topic words picks the same tool for both.
Every entry that accepts worked examples sees one solved example per option before each item, drawn from a separate pool of 382 examples. Three small encoder models (Laya, CLM and Julia-1) have no place to take examples, so they run zero-shot, meaning with no examples at all. TEV, Together AI's decision model, and OpenAI's Decisions API were also run zero-shot, reported in their own column.
Why do contrastive pairs matter?
Pair accuracy is the stricter score: a pair counts only when the model gets both halves right. It separates models that read the detail from models that recognise the topic.
On plain accuracy, Drex Max leads Jev by 8.2 points. On pair accuracy the gap doubles to 16.4 points.
| Accuracy | Pair accuracy | |
|---|---|---|
| Drex Max | 94.4% | 89.3% |
| REAX 1 27B | 92.4% | 87.1% |
| pplx-decider v1.2 27B (self-hosted) | 92.2% | 86.2% |
| Jev | 86.2% | 72.9% |
| GPT-6 Luna | 84% | 72% |
Source: DecideBench v2, measured 9 November 2026
Drex Max loses 5.1 points going from accuracy to pair accuracy. Jev loses 13.3. That gap means Jev more often gets one half of a pair right and the other wrong, which is what reading the topic and missing the edit looks like. The method behind this is the same one we describe in how to test a classifier: build the hard twin of each case, and score the pair.
How are cost and latency measured?
Hosted entries pay for the tokens each API billed, at the provider's list price on a recorded date. Self-hosted entries pay for GPU time.
| Hosted (API) | Self-hosted (open weights) | |
|---|---|---|
| Cost basis | Billed tokens at dated list price | Wall-clock time for all 450 items at 4 in flight, on one NVIDIA RTX PRO 6000 Blackwell (AWS g7e.2xlarge, $3.36/h on demand) |
| Latency | Median per request, timed from one client machine, network included | Median per request, timed on the GPU box, no network |
| Where it is recorded | results/v2/meta/<entry>.json: model served, endpoint, price, price source, date | Same file, plus the GPU and hourly rate |
Two consequences follow. Self-hosted latency has no network in it, so those entries stay off the latency frontier rather than beating hosted models on an unfair clock. And a self-hosted model on a busier server would cost less per task than on a dedicated GPU at 4 requests in flight.
The meta files also catch quiet changes. Drex Max's file records that Nace.AI served drex-max-v1.1 when drex-max was requested, at $0.06 per million input tokens as of 19 October 2026. Jev's records typesafe-jev-1.13.0, called through AI Space, XY Space's platform, with a gateway hop on the clock.
What is a Pareto frontier, and which models sit on it?
A model is on a frontier when no other model is at least as good on both axes and better on one. On the cost frontier, nothing is both cheaper and more accurate. On the latency frontier, nothing is both faster and more accurate.
A frontier answers "what is the best I can get for this price" for every price at once. A single ranking hides that: the third most accurate model can be the right buy if it costs a third as much.
| Frontier | Entries on it (accuracy, cost per 1M, median latency) |
|---|---|
| Cost (all entries) | Drex Max (94.4%, $91.00), Jev (86.2%, $75.04), s1-fast Blink v0.3 (82.0%, $29.82), Laya typed-decisions 421M (39.1%, $13.96), Julia-1 144M (26.9%, $9.13) |
| Latency (hosted) | Drex Max (94.4%, 258 ms), GPT-6 Luna (84.0%, 196 ms) |
| Self-hosted cost | pplx-decider v1.2 27B (92.2%, $386.34), Clef 27B (85.3%, $216.42), yev1-4b (79.3%, $71.15), OpenJev Flash 9B (75.3%, $41.33), Laya (39.1%, $13.96), Julia-1 (26.9%, $9.13) |
The Opus 4.8 reference and the random baseline sit at the two ends of the cost and latency frontiers and are left out above. yev1-4b is ours, made by XY Space. Laya and Julia-1 are on the cost frontiers only because they are the cheapest entries. Their accuracy sits close to random guessing, which scores 25.1%.
What changed from v1 to v2?
v2 is a new, harder test, and its scores are not comparable with v1's. v1 had 400 items in 8 families with short summarised inputs, and most entries scored above 90% on it. That ceiling is why v2 exists.
| v1 (29 September 2026) | v2 (9 November 2026) | |
|---|---|---|
| Items | 400 (200 pairs) | 450 (225 pairs) |
| Families | 8 | 9: adds triaging an inbound email and gating a staged release; drops review sentiment |
| Inputs | Short summaries, median 279 characters | Production format, median 893 characters |
| Worked examples | 297 | 382 |
| Self-hosted GPU | NVIDIA L4 | NVIDIA RTX PRO 6000 |
To show the size of the change: Drex Max scored 98.8% on v1.1 and 94.4% on v2. That is a different test, not a regression in the model. Other posts on this site quote v1 numbers; read them as results on the easier test.
Gating a staged release is the family that pulls the field apart. The model has to compute a crash-free rate or an error ratio and check it against an ordered list of rules. Opus 4.8 gets all 50 right; no decision model gets more than 37 (Drex Max, 74%). Ruling on a return, with dates and amounts under a written policy, is the next widest gap.
What are DecideBench's limitations?
The main ones are who wrote the items, how many there are, and the fact that they are public.
- A language model wrote the items. Claude wrote every item, example and policy; none comes from a public dataset. Labels were written from each item's policy and checked by Claude Opus, Sonnet and Haiku. A careful reader could dispute a few. Of Opus 4.8's three misses, the changelog argues two are label-correct and one is arguable. Using models to check labels has the same strengths and blind spots as LLM-as-a-judge.
- 450 items is not many. Each family has 50, so a one-item swing moves a family score by 2 points. Family numbers have wide intervals.
- The data is public. A model trained on it later may score higher. Every data line carries a canary string so training pipelines can exclude it.
- Some models are not in it. v2 did not test Liquid AI d1-ultra, Microsoft-Decision-1, Strands Decider 2B, Upstage Solar Decide or meraGPT Decider 1.
- Some models could not be run. Drex 1.5 is missing because Nace.AI now serves Drex Max under that name (all 450 answers matched). Quyet-1.0-Large was retired on 3 November 2026. Jev without examples is missing because AI Space had no gateway credits for Jev during the runs.
- Two servers are wrappers written for the benchmark. Perplexity ships pplx-decider as a Python library, which DecideBench batches behind a Jev-style API.
How does DecideBench compare with other decision model benchmarks?
The other public decision model leaderboards measure different things, so their numbers do not line up with DecideBench's.
| Benchmark | Publisher | What it scores | Cost and latency |
|---|---|---|---|
| DecideBench v2 | XY Space (Cho Yin Yong), 9 Nov 2026 | 450 production-format decisions across 9 business families, against a frontier LLM reference | Measured per entry, shown as frontiers |
| Artificial Analysis Decision Index | Artificial Analysis, v1.0 on 13 Oct 2026 | Weighted average of four evaluations: knowledge with an "I don't know" option, multiple-choice reasoning from Humanity's Last Exam, NVIDIA's When2Call and Ai2's WildGuardTest | Cost per 1M tasks and median time reported beside the index |
| AIM System One Index | AIMultiple, v1.2 updated 3 Nov 2026 | z-scored average of browser tasks, a 1,655-question classification set and a bias test, across 64 models | Shown for reference, not counted |
Artificial Analysis and AIMultiple cover more models, including several DecideBench has not tested. DecideBench covers fewer models on decisions shaped like a production workflow. If your decisions look like its nine families, its numbers are the closer guide; for knowledge or reasoning questions, the other two are.
How do you reproduce DecideBench or add a model?
Clone the repo and run the published numbers back from the stored predictions:
uv sync
uv run python -m decidebench.dataset # validate items, example pool and canary
uv run python -m decidebench.report # rebuild the tables from results/v1 and results/v2
uv run python tools/v2/breakdown.py # rebuild the per-option breakdown
uv run pytest # regression tests pin every published numberTo re-run a hosted entry, put its API key in .env and run uv run python tools/v2/run_v2.py --entry <name>. Self-hosted entries need an NVIDIA GPU machine reachable over SSH, through tools/selfhosted. MAINTAINING.md in the repo covers adding a model: write an adapter, give it worked examples, register it, run it, then regenerate the tables, re-pin the tests and add a dated changelog line.
DecideBench is useful to XY Space because we build agentic workflows that call decision models, and AI Space hosts Jev and open models on Western infrastructure. The benchmark is how we choose which model sits behind which decision. For what those choices cost at volume, see System One model pricing; for the leader against the original, Drex Max vs Jev.
FAQ
Is DecideBench independent?
No. DecideBench is built by Cho Yin Yong at XY Space, which also makes the Yev models (yev0-4b and yev1-4b) and runs AI Space, the platform Jev was called through in v2. What it offers instead of independence is a public method: the items, every model's raw predictions, each run's price source and date, and regression tests that pin every published number. Anyone can rerun it and check.
Why is Claude Opus 4.8 the reference and not Claude Opus 5.5?
Claude Opus 5.5 was run. Anthropic's cyber safeguard declined 7 of the 50 items on routing an analytics agent to a tool before the model could answer. Opus 5.5 got 442 of the other 443 right, but a reference needs an answer on every item, so Opus 4.8, at 447 of 450, is the reference. The Opus 5.5 raw run is in the repo.
How often is DecideBench updated?
When models launch, change price or retire. v1.0 shipped on 29 September 2026, with dated updates through October (for example Drex Max on 19 October and Decision Models' four models on 25 October). v2.0 and v1.1 both landed on 9 November 2026. Every change is a dated line in the changelog.
Can I submit a model to DecideBench?
Yes, by adding it yourself. MAINTAINING.md explains how: write an adapter for your model's API or server, register it, run it, and publish the regenerated results with a changelog line. Hosted entries are priced at a dated list price, so the run records where the price came from. Self-hosted models run on the same RTX PRO 6000 setup, so their cost is measured on the GPU it is priced by.
What is the best System One model on DecideBench?
The best System One model on DecideBench depends on what you trade. Drex Max is the most accurate decision model at 94.4%. s1-fast Blink v0.3 gets 82.0% at $29.82 per million decisions. Self-hosted, pplx-decider v1.2 27B gets 92.2%. The pillar post gives a pick per reader, and open source System One models covers the self-hosted options.
Sources
- DecideBench, README and v2 results, Cho Yin Yong (XY Space), 9 November 2026
- DecideBench, CHANGELOG, v2.0 entry, 9 November 2026
- DecideBench, leaderboard and dataset, Hugging Face, 9 November 2026
- TypeSafe, Introducing System One Models & Jev, 15 September 2026
- Nace.AI, Drex Max guide, price as of 19 October 2026
- Artificial Analysis, Artificial Analysis Decision Index, 13 October 2026
- AIMultiple, AIM System One Index, updated 3 November 2026
Written by
Cho Yin Yong
Principal AI Solutions Engineer, XY Space
Principal AI Solutions Engineer at XY Space. University of Toronto lecturer for five years, co-author of two patents, winner of two competitive AI awards, and nine years of regulated engineering leadership.
More from Cho Yin YongShare this article
Work with us
We build the systems these posts describe, and we'll tell you in the first call whether yours is worth building.
Start a project