AI ExplainersNov 9, 202610 min read

System One Model Benchmark: How DecideBench v2 Works

A System One model benchmark should price every right answer. DecideBench v2 scores 26 models on accuracy, cost and speed, and maps trade-offs, not one ranking.

On this page
  1. What does DecideBench measure, and why can a frontier LLM solve it?
  2. What do the 450 decisions look like?
  3. Why do contrastive pairs matter?
  4. How are cost and latency measured?
  5. What is a Pareto frontier, and which models sit on it?
  6. What changed from v1 to v2?
  7. What are DecideBench's limitations?
  8. How does DecideBench compare with other decision model benchmarks?
  9. How do you reproduce DecideBench or add a model?
  10. FAQ
  11. Sources

DecideBench v2 is a System One model benchmark published on 9 November 2026, with open data (CC BY 4.0) and open code (MIT). It measures accuracy, cost per decision and latency for 26 System One models on 450 decisions written in production input format. Instead of one ranking, it reports Pareto frontiers: the models no other model beats on both price and accuracy, or on both speed and accuracy.

A System One model, or decision model, reads an input, a question and a short list of options, and returns one option key. TypeSafe coined the term when it launched Jev, its own decision model, on 15 September 2026, and what is a System One model covers the category. For the picks the results lead to, read the best System One models of 2026.

  • What: 450 multiple-choice decisions in 225 contrastive pairs, 9 task families, 3 to 10 options each.
  • Who made it: Cho Yin Yong at XY Space. XY Space also makes the Yev models and runs AI Space.
  • Reference: Claude Opus 4.8 scores 99.3% at $26,542.70 per million decisions and 1,336 ms.
  • Most accurate decision model: Drex Max (Nace.AI), 94.4% at $91.00 per million and 258 ms.
  • Where to check: the GitHub repo, the leaderboard and the dataset.

What does DecideBench measure, and why can a frontier LLM solve it?

DecideBench measures how close a cheap decision model gets to a frontier LLM on decisions the LLM already gets right. Claude Opus 4.8 answers 447 of the 450 items correctly, so the open question is price.

Building the test around a model that already passes it puts price at the centre. Opus 4.8 costs about $26,500 per million decisions and takes 1.3 seconds each. The benchmark asks how close far cheaper models get to it.

Drex Max, the closest, costs about 1/290 as much per decision. It answers in about a fifth of the time. It gets 94 of every 100 right, where Opus 4.8 gets 99.

Every entry gets four numbers:

MetricWhat it is
AccuracyShare of the 450 items answered correctly
Pair accuracyShare of the 225 pairs with both halves right
Cost per 1M tasksWhat a million decisions cost at the measured rate
Latency p50Median time per request, 4 requests in flight

Claude Opus 4.8 is the reference the items were built against, not a decision model, and it does not compete in the decision-model results.

What do the 450 decisions look like?

Each item is a decision a piece of software would face in production, written in the format it would arrive in. There are 50 items in each of nine families:

FamilyTypical input
Approving an AI agent's proposed actionThe agent's proposed tool call as JSON
Routing an analytics agent to a toolThe conversation so far and the data on hand
Checking a drafted claim against its sourcesA retrieval result with its sources
Moderating a postThe post and the written policy
Ruling on a returnOrder dates, amounts and the returns policy
Gating a staged software releaseA release controller's metrics and rules
Reading a customer's intentA customer message
Triaging an inbound emailA raw email with headers
Triaging a bug reportA bug report

The inputs are what the model would see at runtime: a JSON payload, a raw email with its headers, a release controller's metrics, a retrieval result with its sources. Where the decision depends on a policy, the full written policy sits in the question. The median input runs to 893 characters, and items offer between 3 and 10 options.

The two halves of each pair differ by one small, realistic edit that changes the right answer. In one routing pair, the data the agent needs is a file attached to the chat in one half and a database table in the other, so the right tool changes. A model that answers from topic words picks the same tool for both.

Every entry that accepts worked examples sees one solved example per option before each item, drawn from a separate pool of 382 examples. Three small encoder models (Laya, CLM and Julia-1) have no place to take examples, so they run zero-shot, meaning with no examples at all. TEV, Together AI's decision model, and OpenAI's Decisions API were also run zero-shot, reported in their own column.

Why do contrastive pairs matter?

Pair accuracy is the stricter score: a pair counts only when the model gets both halves right. It separates models that read the detail from models that recognise the topic.

On plain accuracy, Drex Max leads Jev by 8.2 points. On pair accuracy the gap doubles to 16.4 points.

Accuracy vs pair accuracy on DecideBench v2
Accuracy vs pair accuracy on DecideBench v2
AccuracyPair accuracy
Drex Max94.4%89.3%
REAX 1 27B92.4%87.1%
pplx-decider v1.2 27B (self-hosted)92.2%86.2%
Jev86.2%72.9%
GPT-6 Luna84%72%

Source: DecideBench v2, measured 9 November 2026

Drex Max loses 5.1 points going from accuracy to pair accuracy. Jev loses 13.3. That gap means Jev more often gets one half of a pair right and the other wrong, which is what reading the topic and missing the edit looks like. The method behind this is the same one we describe in how to test a classifier: build the hard twin of each case, and score the pair.

How are cost and latency measured?

Hosted entries pay for the tokens each API billed, at the provider's list price on a recorded date. Self-hosted entries pay for GPU time.

Hosted (API)Self-hosted (open weights)
Cost basisBilled tokens at dated list priceWall-clock time for all 450 items at 4 in flight, on one NVIDIA RTX PRO 6000 Blackwell (AWS g7e.2xlarge, $3.36/h on demand)
LatencyMedian per request, timed from one client machine, network includedMedian per request, timed on the GPU box, no network
Where it is recordedresults/v2/meta/<entry>.json: model served, endpoint, price, price source, dateSame file, plus the GPU and hourly rate

Two consequences follow. Self-hosted latency has no network in it, so those entries stay off the latency frontier rather than beating hosted models on an unfair clock. And a self-hosted model on a busier server would cost less per task than on a dedicated GPU at 4 requests in flight.

The meta files also catch quiet changes. Drex Max's file records that Nace.AI served drex-max-v1.1 when drex-max was requested, at $0.06 per million input tokens as of 19 October 2026. Jev's records typesafe-jev-1.13.0, called through AI Space, XY Space's platform, with a gateway hop on the clock.

What is a Pareto frontier, and which models sit on it?

A model is on a frontier when no other model is at least as good on both axes and better on one. On the cost frontier, nothing is both cheaper and more accurate. On the latency frontier, nothing is both faster and more accurate.

A frontier answers "what is the best I can get for this price" for every price at once. A single ranking hides that: the third most accurate model can be the right buy if it costs a third as much.

FrontierEntries on it (accuracy, cost per 1M, median latency)
Cost (all entries)Drex Max (94.4%, $91.00), Jev (86.2%, $75.04), s1-fast Blink v0.3 (82.0%, $29.82), Laya typed-decisions 421M (39.1%, $13.96), Julia-1 144M (26.9%, $9.13)
Latency (hosted)Drex Max (94.4%, 258 ms), GPT-6 Luna (84.0%, 196 ms)
Self-hosted costpplx-decider v1.2 27B (92.2%, $386.34), Clef 27B (85.3%, $216.42), yev1-4b (79.3%, $71.15), OpenJev Flash 9B (75.3%, $41.33), Laya (39.1%, $13.96), Julia-1 (26.9%, $9.13)

The Opus 4.8 reference and the random baseline sit at the two ends of the cost and latency frontiers and are left out above. yev1-4b is ours, made by XY Space. Laya and Julia-1 are on the cost frontiers only because they are the cheapest entries. Their accuracy sits close to random guessing, which scores 25.1%.

What changed from v1 to v2?

v2 is a new, harder test, and its scores are not comparable with v1's. v1 had 400 items in 8 families with short summarised inputs, and most entries scored above 90% on it. That ceiling is why v2 exists.

v1 (29 September 2026)v2 (9 November 2026)
Items400 (200 pairs)450 (225 pairs)
Families89: adds triaging an inbound email and gating a staged release; drops review sentiment
InputsShort summaries, median 279 charactersProduction format, median 893 characters
Worked examples297382
Self-hosted GPUNVIDIA L4NVIDIA RTX PRO 6000

To show the size of the change: Drex Max scored 98.8% on v1.1 and 94.4% on v2. That is a different test, not a regression in the model. Other posts on this site quote v1 numbers; read them as results on the easier test.

Gating a staged release is the family that pulls the field apart. The model has to compute a crash-free rate or an error ratio and check it against an ordered list of rules. Opus 4.8 gets all 50 right; no decision model gets more than 37 (Drex Max, 74%). Ruling on a return, with dates and amounts under a written policy, is the next widest gap.

What are DecideBench's limitations?

The main ones are who wrote the items, how many there are, and the fact that they are public.

  • A language model wrote the items. Claude wrote every item, example and policy; none comes from a public dataset. Labels were written from each item's policy and checked by Claude Opus, Sonnet and Haiku. A careful reader could dispute a few. Of Opus 4.8's three misses, the changelog argues two are label-correct and one is arguable. Using models to check labels has the same strengths and blind spots as LLM-as-a-judge.
  • 450 items is not many. Each family has 50, so a one-item swing moves a family score by 2 points. Family numbers have wide intervals.
  • The data is public. A model trained on it later may score higher. Every data line carries a canary string so training pipelines can exclude it.
  • Some models are not in it. v2 did not test Liquid AI d1-ultra, Microsoft-Decision-1, Strands Decider 2B, Upstage Solar Decide or meraGPT Decider 1.
  • Some models could not be run. Drex 1.5 is missing because Nace.AI now serves Drex Max under that name (all 450 answers matched). Quyet-1.0-Large was retired on 3 November 2026. Jev without examples is missing because AI Space had no gateway credits for Jev during the runs.
  • Two servers are wrappers written for the benchmark. Perplexity ships pplx-decider as a Python library, which DecideBench batches behind a Jev-style API.

How does DecideBench compare with other decision model benchmarks?

The other public decision model leaderboards measure different things, so their numbers do not line up with DecideBench's.

BenchmarkPublisherWhat it scoresCost and latency
DecideBench v2XY Space (Cho Yin Yong), 9 Nov 2026450 production-format decisions across 9 business families, against a frontier LLM referenceMeasured per entry, shown as frontiers
Artificial Analysis Decision IndexArtificial Analysis, v1.0 on 13 Oct 2026Weighted average of four evaluations: knowledge with an "I don't know" option, multiple-choice reasoning from Humanity's Last Exam, NVIDIA's When2Call and Ai2's WildGuardTestCost per 1M tasks and median time reported beside the index
AIM System One IndexAIMultiple, v1.2 updated 3 Nov 2026z-scored average of browser tasks, a 1,655-question classification set and a bias test, across 64 modelsShown for reference, not counted

Artificial Analysis and AIMultiple cover more models, including several DecideBench has not tested. DecideBench covers fewer models on decisions shaped like a production workflow. If your decisions look like its nine families, its numbers are the closer guide; for knowledge or reasoning questions, the other two are.

How do you reproduce DecideBench or add a model?

Clone the repo and run the published numbers back from the stored predictions:

uv sync
uv run python -m decidebench.dataset       # validate items, example pool and canary
uv run python -m decidebench.report        # rebuild the tables from results/v1 and results/v2
uv run python tools/v2/breakdown.py        # rebuild the per-option breakdown
uv run pytest                              # regression tests pin every published number

To re-run a hosted entry, put its API key in .env and run uv run python tools/v2/run_v2.py --entry <name>. Self-hosted entries need an NVIDIA GPU machine reachable over SSH, through tools/selfhosted. MAINTAINING.md in the repo covers adding a model: write an adapter, give it worked examples, register it, run it, then regenerate the tables, re-pin the tests and add a dated changelog line.

DecideBench is useful to XY Space because we build agentic workflows that call decision models, and AI Space hosts Jev and open models on Western infrastructure. The benchmark is how we choose which model sits behind which decision. For what those choices cost at volume, see System One model pricing; for the leader against the original, Drex Max vs Jev.

FAQ

Is DecideBench independent?

No. DecideBench is built by Cho Yin Yong at XY Space, which also makes the Yev models (yev0-4b and yev1-4b) and runs AI Space, the platform Jev was called through in v2. What it offers instead of independence is a public method: the items, every model's raw predictions, each run's price source and date, and regression tests that pin every published number. Anyone can rerun it and check.

Why is Claude Opus 4.8 the reference and not Claude Opus 5.5?

Claude Opus 5.5 was run. Anthropic's cyber safeguard declined 7 of the 50 items on routing an analytics agent to a tool before the model could answer. Opus 5.5 got 442 of the other 443 right, but a reference needs an answer on every item, so Opus 4.8, at 447 of 450, is the reference. The Opus 5.5 raw run is in the repo.

How often is DecideBench updated?

When models launch, change price or retire. v1.0 shipped on 29 September 2026, with dated updates through October (for example Drex Max on 19 October and Decision Models' four models on 25 October). v2.0 and v1.1 both landed on 9 November 2026. Every change is a dated line in the changelog.

Can I submit a model to DecideBench?

Yes, by adding it yourself. MAINTAINING.md explains how: write an adapter for your model's API or server, register it, run it, and publish the regenerated results with a changelog line. Hosted entries are priced at a dated list price, so the run records where the price came from. Self-hosted models run on the same RTX PRO 6000 setup, so their cost is measured on the GPU it is priced by.

What is the best System One model on DecideBench?

The best System One model on DecideBench depends on what you trade. Drex Max is the most accurate decision model at 94.4%. s1-fast Blink v0.3 gets 82.0% at $29.82 per million decisions. Self-hosted, pplx-decider v1.2 27B gets 92.2%. The pillar post gives a pick per reader, and open source System One models covers the self-hosted options.

Sources

Written by

Cho Yin Yong

Principal AI Solutions Engineer, XY Space

Principal AI Solutions Engineer at XY Space. University of Toronto lecturer for five years, co-author of two patents, winner of two competitive AI awards, and nine years of regulated engineering leadership.

More from Cho Yin Yong

Share this article

Work with us

We build the systems these posts describe, and we'll tell you in the first call whether yours is worth building.

Start a project
Work with us

Book a call.We'll come back with specifics.

Start with the map of your organization, or with the one job that hurts. Measured in hours and money, and everything we build stays yours.

Loading form…