How to Evaluate an LLM Classifier: Test Pairs That Differ by One Word

To evaluate an LLM classifier, write new test items in pairs that differ by one small edit that flips the right answer. Then count a pair as right only when both halves are right. Together released Tev on 23 September 2026 with no accuracy numbers. We tested it this way on 25 September. It got 90.0% of items right, but only 80.0% of pairs.

Accuracy tells you how often a classifier is right. Pair accuracy tells you whether it read the input.

• The test: 400 new items in 200 pairs, across 8 kinds of task, written for this run. The data, code and results are public in our benchmark.
• What we measured: accuracy, pair accuracy, a range of likely accuracy, a head-to-head check, calibration, speed, cost per task and unusable replies.
• What we tested: Tev from Together and Jev from TypeSafe, two small models that pick one option from a list. Two larger general-purpose models ran alongside for reference.
• What the whole run cost: $1.10 for 4,946 billed calls, including five prompt versions for each classifier.
• What moved accuracy most: removing the one-sentence option descriptions. Tev lost 6.8 points and Jev 4.5.

A classifier here means a model that reads some text and picks one label from a fixed list: a support intent, a refund decision, approve or block. The method works for any of them, including one you prompt yourself on a general model.

Why is accuracy alone misleading?

Accuracy alone rewards a model that matches on topic words. Such a model gets one half of a contrastive pair right and the other half wrong, and still scores well.

Tev is the clearest case. It got 360 of 400 items right, which is 90.0%. It got both halves right in 160 of 200 pairs, which is 80.0%. That leaves 40 pairs with exactly one half right: every one of Tev's 40 misses sat in a different pair. Tev never failed both sides of a pair; it failed on the edit.

The second trap is an uneven mix of right answers. In three of our task families, always picking the most common answer scores about 50%. That shortcut is called the majority baseline:

So 82.0% on the action family is 34 points above guessing, not 82 points. Always print the baseline beside each family's score. For the full comparison of the two models, see Jev vs Tev.

How do you build contrastive pairs?

Write two items that share the same question and the same options, and change one realistic detail so the right answer changes. The two correct answers in a pair must differ.

Here is one real pair from our agent actions family. The question is how to handle an AI agent's proposed action: approve it, send it to a person, or block it. The policy approves read-only actions and sends production writes to a person:

Both touch the same orders table, so a model that matches on topic words may treat them alike. Only a model that reads the target and the verb gets the pair.

The rules we followed, from our dataset spec:

• Same question, same options, same order for both halves.
• One small, realistic edit: a negation, a date one day past a window, --dry-run removed from a command, an internal email recipient swapped for an external one.
• Two to eight options, each with a one-sentence description, plus a none or other option where the real task has one.
• Move the right answer around the option list, so position gives nothing away.
• Exactly one defensible answer. If a careful person would argue, rewrite the item.
• About 40% hard pairs: heavy distractors, negation, sarcasm or rule edge cases.
• Invented names, numbers and companies only. No real personal data.

Why write new classifier test items instead of using public datasets?

Write new items because the model may have trained on the public ones. A score on training data measures memory, not reading.

Together's launch post says Tev was fine-tuned on 37,840 examples. They include five well-known public datasets, among them Banking77 for support intents and MultiNLI for evidence checks. Those are the obvious public sets for intent, topic, sentiment and evidence tasks, so none of our items come from them. Claude wrote all 400 items as original text.

The same rule holds for your own classifier. If you tuned a prompt on a set of examples, those examples are no longer a fair test. Keep a set aside that nobody looked at while building. We did the same when we rebuilt an inbox around Jev.

What should you measure to evaluate a classifier besides accuracy?

Measure pair accuracy, precision and recall for each answer, a likely range for accuracy, whether the gap between two models is chance, calibration, speed, cost per task and unusable replies. Each one answers a question accuracy cannot.

Precision and recall for each answer. Accuracy counts items, so it rewards getting the common answers right. Score each answer on its own instead. Precision is how often the model is right when it gives that answer. Recall is how often it gives that answer when it is the right one. Tev's precision on "ask a clarifying question" was 100%, but its recall was 25%: four items called for that answer and Tev gave it once. Its accuracy of 90.0% hides that. Macro F1 combines the two for each answer and averages them with equal weight, so a model that ignores rare answers scores lower. Tev scored 88.5 and Jev 95.5. The two reference models did better still: GLM 98.0 and Claude 99.4. Per-answer precision also tells you which answers to double-check; LLM cascade uses it to decide which tasks go to a larger model.

The interval. Resample whole pairs many times and read off the middle 95% of scores. Resample pairs, not items, because the two halves of a pair are linked.

Is the gap chance? Look only at the items where exactly one model is right. A split of 33 to 4 is too large to be chance (p = 1.08e-06, an exact McNemar test).

Calibration. When a model says it is 90% sure, is it right about 90% of the time? We report the average gap, called expected calibration error. Good calibration lets you send low-confidence answers to a person.

Speed and cost. We timed calls from our own machine, so the numbers include the network. For cost, multiply the tokens each API actually billed by its list price. A token is about three quarters of a word. Tev and Jev share a list price, yet Jev bills about twice the input tokens for the same text.

Unusable output. Count replies that map to no option, and errors after 5 retries, as wrong. All four models scored 0.0% here.

Checking free-text outputs is a different job; see what is LLM-as-a-judge for that.

Should you test the prompt as well as the model?

Yes. Run the same items through a few prompt versions and see what moves. In our run, only one change mattered: taking away the option descriptions.

We ran five versions for both classifiers:

• Default: the vendor's recommended prompt, with a one-sentence description per option.
• Careful: one added line asking the model to watch negations, dates, exceptions and who is speaking.
• Keys only: option names with no descriptions.
• Reversed: the same options in reverse order.
• Generic question: the task question replaced with "Which option best fits the input?"

Without descriptions, Tev dropped 6.8 points and Jev 4.5. Pair accuracy fell further: Tev from 80.0% to 68.0%, Jev from 95.0% to 86.0%. Rewording, reordering and extra guidance each moved accuracy by under a point.

The lesson for your own classifier: spend your effort on clear option descriptions, not on clever instructions. Also count how many answers change between versions. A prompt can fix some items and break others while the score stays the same. For related failure modes in free-text models, see how to reduce LLM hallucination.

What does it cost to run this kind of test?

Our whole experiment cost $1.10 at list prices. That covers 4,946 billed calls across four models, five prompt versions, warm-ups and retries.

The two classifiers ran about five times as many calls and still cost about 6 cents together. The reference models ran once each and made up most of the bill. Writing the items is the real cost, and that is time, not money.

List prices per million tokens, reading and writing: Tev $0.042 and $0 on Together. Jev $0.042 and $0 from TypeSafe. GLM 5.3 $1.40 and $4.40 on Cloudflare Workers AI. Claude Opus 5.5 $4 and $20 from Anthropic. For picking models by what each correct answer costs, see the cheapest model per solved task.

What are the limits of this method?

The main limit is that a language model wrote the items and the same kind of model checked the labels. We checked the labels by structure and by spot checks, not by a full human review.

Tev and Jev missed the same 7 items, and we reviewed each. Four are clearly labelled and both models got them wrong: a percentage calculation, two date-window rules, and a moderator report that quotes a threat. Three can be argued: review_sentiment-021a, ticket_triage-022b and claim_support-006b. We kept all labels. Shared misses do not change the gap between the two models.

The set is small. 400 items gives a tight overall range but wide ranges per family, so trust the headline range and the head-to-head test over any single family.

Speed depends on the route. We reached Jev through the AI Space gateway, so its times include that extra hop. We called Tev on Together directly. The two reference models also went through AI Space, with reasoning at their defaults.

We tested one question type, single choice. Jev also answers scores and yes-or-no questions, which Tev has no match for here.

Scores drift, so rerun the test when a provider updates a model. Version drift on Terminal-Bench shows how far a score can move.

When a classifier is only right on one half of a pair, it is matching words. If you are choosing between a classifier and a general model for a task, decision model vs LLM covers the trade.

FAQ

How many test items do I need to evaluate a classifier?

We used 400 items in 200 pairs, and the overall accuracy range came out 5.5 points wide for Tev. That is enough to separate two models 7 points apart, but too few to rank them on a single family of 50 items. If your decision hinges on one task type, write more pairs for that type.

What is pair accuracy?

Pair accuracy is the share of contrastive pairs where the model got both halves right. The halves differ by one edit that changes the answer. A model that reads closely scores near its accuracy. A model that matches on topic words scores well below it. Tev scored 90.0% accuracy and 80.0% pair accuracy.

Can I use an LLM to write my test items?

Yes, with checks. Claude wrote all 400 of ours from a written spec. Check every label's structure, spot-check by hand, and read every item that all models miss. Of our 7 shared misses, 3 had arguable labels. A person who knows the task should sign off on anything you ship on.

How much did this benchmark cost to run?

The run on 25 September 2026 cost $1.10 at list prices, for 4,946 billed calls. The two classifiers, Tev and Jev, cost $0.0201 and $0.0400 across all five prompt versions. The two general models used as reference points cost $0.3345 and $0.7043.

Does prompt wording change a classifier's accuracy?

Only a little, in our test, except for one change. Rewording the question, reordering options or adding a line of guidance moved accuracy under a point. Removing the option descriptions cost Tev 6.8 points and Jev 4.5. Write a clear one-sentence description for every option.

Sources

• XY Space, Jev vs Tev benchmark: results, methodology and dataset spec, 25 September 2026
• Together AI, How to train your own Jev for $17, 23 September 2026
• Together AI, Tev1-4B-experimental weights on Hugging Face, accessed 25 September 2026
• Together AI, Tev serverless pricing on X, accessed 25 September 2026
• TypeSafe, Introducing System One Models and Jev, 15 September 2026
• Cloudflare, GLM-5.3 on Workers AI, accessed 25 September 2026