How to Build a Decision Model: What Training Yev Taught Us

A decision model makes one call, such as approving a refund. To build one, make a small open model reply with one letter, train it on cases that differ by one detail, then calibrate it so a 90% answer is right about 90% of the time. Yev, our open model, trained in 4.71 hours on one graphics card. Developer Nish Tahir showed the one-letter step on 10 October 2026.

• Base model: Qwen3.5-4B-Base, which anyone can download.
• Output: one option letter, read from the model's scores for the next word. Nothing is generated or parsed.
• Data: 44,459 examples. Over half were written for this model, many on the edge of a rule.
• Training: 4.71 hours on one cloud graphics card for the run we shipped.
• Result: 94.2% against 98.0% for Jev, TypeSafe's hosted decision model. Long cases are the weak spot.

What did Nish Tahir's decision model post show?

Tahir's post, "Build your own decision model", on his blog Another Dev's Two Cents, turned a general model into a multiple-choice decider with no new architecture. It reached 476 points on Hacker News.

He took Qwen3-1.7B, a model with 1.7 billion parameters, and kept only its scores for the letters A to E as the next token. Everything else was masked. A softmax over those five scores gives a probability for each option. He then ran a quick fine-tune on CommonsenseQA, a public set of five-option questions. On 1,221 questions held out from training, accuracy rose from 59.38% to 62.41%. Last, he fitted a temperature of about 3.8 to calibrate the scores.

Nine days earlier, on 1 October, Strands Agents released Strands Decider 2B, which takes the other route: it replaces the model's output layer. The two posts cover the first big choice you make. The rest of this guide covers the choices after it, using the run behind Yev.

What is a decision model?

A decision model reads some text, a question and a short list of options, and returns one option with a probability. It writes no prose. Approving a refund, routing a ticket and flagging a post are typical jobs.

TypeSafe's Jev made the category popular in September 2026. Our guide to what is Jev explains the request shape. Our model handles three question types: pick one option (Choice), yes or no (Noul), and a point on an ordered scale (Score).

Which base model and output head should you use?

Start with a small open base model and read the answer from its letter scores. Tahir did this, and so did we, because it keeps the model's existing vocabulary. Strands replaced the output layer instead.

The letter route needs no new code at serving time. Any server that returns next-token scores can run it, and Yev's server also answers through an OpenAI-style chat endpoint with one letter. The pointer route needs its own head loaded at serving time, but it does not depend on option letters.

Where does the training data come from?

Yev's 44,459 rows came from four places. Public datasets give breadth. Rule-generated and model-written rows give the edges that a real policy turns on.

Contrastive pairs, two cases that differ by one detail that flips the answer, matter because a model trained only on public data learns topic words. A refund request that mentions "refund" looks like every other refund request. The answer often turns on one detail, such as a request one day past a 30-day window. So the rule-generated rows put cases on the last day of the window and one day either side, with labels computed from the rule.

The Opus rows come in small families. One base case gets variants that each change about 15 tokens to flip the answer. A fresh Opus session then had to name the edit that flipped it. If it could not, the pair was dropped. That blind check is the only label review those rows had. No person read them, which the model card lists as a limit.

Three in four rows were Choice questions. About 35% of rows carry worked examples, one solved example for each answer option sent with the question, so the model learns to use them when a caller sends some. The repository builds each part with its own command:

yev build-public                      # public data
yev gen-rules --family <name>         # rule-based contrastive clusters
yev synth ...                         # Opus-written clusters, with blind checks
yev filter && yev shortcut            # licence, contamination, dedupe and shortcut filters
yev mix --recipe recipes/stage0.json  # train / dev / calibration split

How do you fine-tune it?

Train a LoRA adapter, a small set of extra weights on top of the frozen base model, for one pass over the data. One NVIDIA L40S graphics card, with 48 gigabytes of memory, was enough for a model with 4 billion parameters.

The loss is where a decision model differs from a chat fine-tune. Yev's loss is cross-entropy over the valid letters only, plus a tenth of ordinary full-vocabulary cross-entropy so the model keeps its language. Score rows add a ranked probability score at half weight, which penalises answering 1 more than answering 3 when the right answer is 4. Label smoothing of 0.05 (0.10 on Score rows) stops the model from betting everything on one letter.

The shipped config is configs/stage0/lc100.json in the repository, and the run starts with:

uv sync --extra train --extra serve
yev train --config configs/stage0/lc100.json

Most of the 25.3 hours went on the five runs that did not ship. The repository keeps their configs too.

How do you calibrate the confidence?

Hold back rows the model never trains on, then fit one temperature per question type so that a 90% answer is right about 90% of the time. We held back 2,000 rows. Most were Choice questions; only 122 were Score questions.

A temperature above 1 softens an overconfident model; below 1 sharpens an underconfident one. Tahir's value of about 3.8 means his model was far too sure of itself. Ours all sit within 7% of 1, so the model came out of training close to calibrated. Fit one per type anyway. A yes-or-no question and a six-option question are not equally hard, and one temperature would split the difference.

Calibration is what makes the model usable. On our development set, answers at 0.9 confidence or higher were right 99.3% of the time and covered 80.8% of items. You can act on those and send the rest to a person or a larger model. LLM cascade shows that setup.

How do you know it works?

Keep the test set out of training, read it once, and choose the shipped run by a rule written before you start. We tested on DecideBench, our public test of 400 decisions, such as whether a refund falls inside a 30-day window, set as 200 contrastive pairs, so we had to be stricter than usual about leaks.

Every training row was checked against all 697 DecideBench items, the 400 test items and 297 worked examples. Rows failed if they shared an eight-word run with an item, came too close in meaning (cosine similarity above 0.85), or carried the benchmark's canary string. A final check found none of the 400 test items in training. We also checked TypeSafe's WorkflowEvals: 68 rows, or 0.15%, overlapped it, mostly stock phrases.

The selection rule looked first at DecideBench's practice set, scoring whole groups of related items. Ties went to plain practice-set accuracy, then to accuracy on held-out Opus rows. The shipped run scored 73.02 on the first measure against 68.25 for the best ablation. One ablation did better on our own held-out data, and it lost under the rule.

Yev lands 3.8 points behind Jev and three items behind imajev-4b, a gap too small to count. Jev was called through AI Space, XY Space's model gateway, and its latency includes that hop. How to test a classifier explains why we score pairs as well as single items.

The weak spot is long cases. On WorkflowEvals, which asks many questions about one long document such as an invoice packet, Yev scored 51.5% against Jev's 67.8%. Stage 0 trained on short cases, capped at 2,048 tokens (4,096 with worked examples). The benchmark measured what we did not train for.

Should you build one or use an existing one?

Use an existing decision model unless you need something no hosted or open model gives you. On DecideBench, imajev-4b, another open model, scores 95.0% at $28.57 per million decisions on one rented L4 card. Ours scores 94.2% at $58.10 on the same card.

Build your own when:

• Your decisions turn on rules nobody else has. A public model never saw your returns policy. Rule-generated pairs from your own policy are the part of Yev's recipe that transfers best.
• You need the training record. Some reviewers want to know what data went in. Yev's filters and the configs for all six runs are public for that reason.
• Your text cannot leave your network and no open model is accurate enough on your cases.

Otherwise, start with the Jev alternatives we tested, and weigh a general model against a decision model with decision model vs LLM. Serving Yev, including the request format, is covered in the Yev post.

FAQ

How much does it cost to train a decision model?

The shipped run took 4.71 hours on one L40S graphics card, and all six runs took 25.3 hours. The larger costs were elsewhere: writing and checking 9,562 model-written rows, and running the benchmark. Nish Tahir's 10 October post used a model under half the size of ours.

Do I need a GPU to build one?

For training, yes. Ours trained on one card with 48 gigabytes of memory, and a smaller model needs less. For serving, one card is enough. We benchmarked it on an L4, a smaller data-centre card with 24 gigabytes. It also runs on an ordinary processor, slowly.

How much training data does a decision model need?

Yev used 44,459 rows. Nish Tahir moved his model 3 points with a quick fine-tune on a public question set. The count matters less than the pairs: rows where one small edit flips the answer teach the model to read the detail instead of the topic.

What is temperature scaling?

Temperature scaling divides the model's raw option scores by one number, the temperature, before turning them into probabilities. Above 1 lowers confidence, below 1 raises it. You fit the number on held-out data so stated confidence matches real accuracy. It never changes which option ranks first.

Can a decision model replace an LLM?

For short, structured choices, often. Yev comes within four points of Jev on DecideBench in a single pass. General models still score higher on DecideBench (up to 99.8%) at six to fourteen times Jev's cost. On WorkflowEvals, the model card's two large general models score 74.1% and 73.1%.

Sources

• Nish Tahir, Build your own decision model, 10 October 2026
• Strands Agents, Introducing Strands Decider, 1 October 2026
• XY Space, yev0-4b model card, 4 October 2026
• XY Space, yev code repository, accessed 14 October 2026
• XY Space, DecideBench dataset and results, v1.1, accessed 14 October 2026
• XY Space, DecideBench leaderboard, accessed 14 October 2026
• TypeSafe, WorkflowEvals, accessed 14 October 2026
• Qwen, Qwen3.5-4B-Base, accessed 14 October 2026