Should a Model Make This Decision?

A lot of model calls in an application are not asking for prose. They pick a route, approve or reject a reply, or decide whether a request needs a person. These are bounded decisions: the answer comes from a known set. This guide walks through how I decide what should make one, using the Jev workshop as the worked example.

Start with the shape of the decision

Write down the allowed outcomes first. If you cannot list them, the task is open-ended and a generator is a reasonable tool. If you can list them and they rarely change, you have a classification-shaped problem, and it should be measured like one.

Lambert Leong makes this argument in When All You Have Are Decoders, Every Decision Looks Like Generation (Towards Data Science, September 29, 2026). He proposes a routing contract: explicit candidates, a score for each, a versioned policy that routes or abstains, and logs of all of it. He also says that scoring a candidate set does not solve how the candidates are built or what to do when the right one is missing.

Try the cheapest option that can be checked

I go down this list and stop at the first option whose evidence I can produce:

1. Code. If a rule decides it from structured fields, write the rule and test it. Permissions, spending limits, retries, exact arithmetic, and hard safety rules stay in code in every design below.
2. A classifier. If the label set is stable and you have labeled history, an encoder with a classification head is the baseline to beat. Count its training and retraining cost.
3. A structured-output LLM. If the policy is written in prose and changes often, or the decision needs generated arguments, a general model with a strict schema is often simplest. A confidence it writes in its output is generated text, not a measured probability.
4. A decision model like Jev. If code supplies the candidates and you want a probability to gate on, try one. TypeSafe's docs say its confidence summarizes the returned distribution and that thresholds need testing on your own data. The workshop gates on the top probability, which the same page lists as a simpler alternative.
5. A cascade. A cheap first stage accepts confident cases and a stronger model or a person handles the rest. Cases both stages get wrong cap the benefit.
6. A person. If your evidence cannot bound the error below your limit, keep a person deciding and collect labels.

The workshop's routing exercise runs keyword rules and a scored policy on the same requests. The rules handle a duplicate charge and can miss the same problem in other words. Hard rules in code run before any score, and an instruction written inside a customer's message is customer text, not policy.

Set an error limit before you look at thresholds

Pick the error you can accept on decisions made without a person, for example 5%. Then ask how much evidence that needs. With zero errors, an exact one-sided 95% bound drops below 5% only after 59 labeled cases. At 10% it takes 29. A pilot with four correct cases supports neither, whatever its accuracy says.

In the workshop, set an error limit and, once there are labeled judgments, it reports one of three outcomes on the cases it selects from (the development split when the pack has a holdout): a selected gate with its coverage and remaining errors, no gate that meets the limit, or insufficient evidence. With no labeled judgments, selection is unavailable.

Freeze, then test once on fresh cases

Choosing a threshold on cases you have already seen makes those cases part of the choice. Freeze the policy and its identity, then evaluate it once on cases nobody has tuned against. Report the result whatever it shows. A frozen policy does not make public or already-inspected cases blind.

The workshop's support-review experiment follows this order. Forty-two fresh holdout cases are frozen by hash and kept out of the site's public files. Two reviewers who did not write the cases will label every study case, holdout included, before seeing any model output; that review is pending. Model collection waits for it and for a budget approval.

What the workshop has shown so far

The recorded pilot is four authored cases. Both judges got all four right, and every Jev judgment put probability 1 on its answer, so the threshold could not trade coverage for error. That checks the integration, not the model. The larger experiment has not been run. Until it is, the workshop's answer for its own support task is "insufficient evidence."