How do you model a classification problem with Jev?
AI Architect
Key takeaways
- Jev exposes exactly three question types, and each carries a different payload back to the caller.
- A noul answer omits the confidence field by design, since its probability already describes the yes or no uncertainty.
- Choice and score both return full probability distributions alongside a confidence derived from how concentrated that spread was.
- A choice question accepts at most 255 options and a score question between two and ten ordered levels.
- The type system forces you to fix the answer set before the call, which is the real modelling constraint.
Jev answers three kinds of question, and the
API reference is unusually short about it. A
noul returns a probability. A choice picks one option from a set. A score
places the input on an ordered scale.
The modelling work sits in deciding which of the three a real decision is, and in noticing what each one hands back.
What does each question type return?
Different payloads, and the difference is deliberate.
A noul answer carries type and noul, a number between 0 and 1, and the
Pydantic integration maps a plain boolean field onto it. It has
no confidence field. A choice answer carries the selected option, a
probabilities map across every option, and a confidence between 0 and 1. A
score answer carries a probability-weighted score, a legend mapping
level indices to descriptions, the same probabilities map, and a
confidence.
The three question types and what each returns
Show as text
| # | Layer | Note |
|---|---|---|
| 1 | noul: probability a proposition is true | Returns the number. No confidence field. |
| 2 | choice: one option from a defined set | Option, probabilities, confidence. |
| 3 | score: a position on an ordered scale | Score, legend, probabilities, confidence. |
| · | fixed by the API above (breakpoint) | |
| 4 | Which threshold turns a noul into a decision | Yours to choose and to record. |
| 5 | Whether the options cover your real cases | The modelling work nobody can do for you. |
The limits are worth memorising because they shape the design. A choice takes at most 255 options, and TypeSafe’s launch post frames the model around exactly that kind of fixed decision. A score takes at least two levels and at most ten. Those ceilings are low enough that a taxonomy with hundreds of leaf categories needs splitting into a routing question followed by a narrower one.
Why does noul omit the confidence field?
Because a second number would say nothing the first one does not.
A noul is the probability that a proposition is true. All the uncertainty is already in that value: 0.5 is maximal doubt and 0.99 is near certainty. There is no spread to summarise.
Choice and score are different, because their probability mass is distributed across several options. Knowing that “billing” won tells you the argmax, and it does not tell you whether billing took 0.9 of the mass or 0.3 of it with “account” close behind. Confidence summarises that spread, which is why the docs define it as derived from the distribution.
The practical consequence is that confidence and probability are different quantities. A choice with confidence 0.8 and a noul of 0.8 are making different statements, and code that treats them interchangeably will route badly.
What does confidence actually promise?
Concentration, and only that.
A high confidence means the model put its mass on one answer. It is entirely possible for a question to return confident answers that are frequently wrong, and nothing in the response would reveal it. Confidence describes the shape of the distribution and correctness is a property of the world.
What comes back from a choice question
Show as text
| # | From | To | Message |
|---|---|---|---|
| 1 | Caller | Jev | question with up to 255 named options. The answer set is fixed before the call. |
| 2 | Jev | Caller | chosen option, plus probabilities for all of them |
| 3 | Jev | Caller | confidence, derived from that distribution. Describes spread rather than correctness. |
| 4 | Caller | Caller | Route on the distribution, log the rest. The full spread is the useful audit record. |
Because the full probabilities map comes back, routing can use more than the
winning label. A gap of 0.9 to 0.05 and a gap of 0.4 to 0.35 both produce the
same chosen option, and the second is a case a human should probably see. That
distinction is available for free in every response and is thrown away by any
handler that reads only the selected value.
This is the same trap as the type guarantee covered separately: a structural property gets read as a quality property. The check that closes the gap is calibration, measured on your own labelled data, per question. It is also cheap, given that the cost model here charges on input and leaves output free.
How should you shape the option set?
As if the model will be forced to choose, because it will.
A choice question returns one of your options every time. Leave out the ambiguous case and it gets absorbed into whichever neighbour looks closest, with a confident number attached. The failure is silent and it looks exactly like a correct answer in the logs.
The fix is to model the awkward cases explicitly. An “ambiguous” option, an “out of scope” option, and where the integration supports it, the optional literal that adds “None of these”. Each one converts a silent misclassification into a label you can count, and counting it is what tells you whether the taxonomy needs another branch.
Which type should you reach for?
My default is a noul wherever the decision is genuinely binary, because the single number is easier to threshold, easier to calibrate and easier to explain to whoever asks why the system did something.
Choice earns its place when the outcomes are genuinely disjoint and you want the distribution for routing. Score is the one I would use most carefully: ordered levels invite a team to argue about whether something is a 3 or a 4, and unless the legend describes observable criteria, that argument never resolves.
What I would avoid is reaching for score because the decision felt fuzzy. Fuzziness in the decision usually means the question has not been written precisely enough yet, and a ten-level scale will hide that rather than fix it.
Do this
Turn a real classification problem into Jev questions
The work is choosing the shape before the call. Each type answers a different question, and the constraint that matters is that the answer set is fixed at request time.
Write the decision as a sentence before choosing a type
If the sentence is a proposition that can be true or false, it is a noul. If it names a fixed set of outcomes, it is a choice. If it places something on an ordered scale, it is a score. Decisions that resist all three usually contain two questions wearing one name.
Enumerate the option set and include the awkward cases
A choice takes up to 255 options. Include the ambiguous and the out-of-scope ones explicitly, because an option you leave out becomes a wrong answer with a confident number attached rather than an obvious gap.
Give a score question a legend, and keep the levels few
Score accepts between two and ten ordered levels and returns a legend mapping index to description. Write those descriptions as observable criteria, since they are the entire definition of what a 3 means.
Choose the noul threshold deliberately and write it down
A noul is a probability, so the cut point is your decision rather than the model's. The 0.5 default in the Pydantic integration is a convention. Pick from the relative cost of a false positive against a false negative in your system.
Log the full probability distribution rather than only the winner
Choice and score both return probabilities across every option. Store them. They cost nothing to keep and they are the raw material for every calibration check and threshold change you will want later.
Check calibration per question rather than per model
Calibration is a property of a question on a distribution of inputs. One question can be well calibrated while another on the same state is not, so bucket by predicted probability and measure hit rate for each question separately.
Frequently asked questions
- Why does a noul answer have no confidence score?
- Because it would be redundant. A noul returns the probability that a yes or no proposition is true, and that single number already describes the uncertainty completely. Choice and score need a separate confidence because their probability mass is spread across several options, so how concentrated that spread is carries information the selected option alone does not.
- What does confidence mean for a choice question?
- How concentrated the distribution was, derived from the probabilities across the option set. A high confidence means the mass sat on one option. It is a statement about the shape of the answer rather than a claim that the answer is correct, and treating it as an accuracy estimate is the common mistake.
- How many options can a choice question have?
- Up to 255, and in the Pydantic integration that ceiling counts route choices and tools together. Score questions take at least two levels and at most ten, with a legend mapping each level index to its description.
- What if none of my options apply?
- Model it explicitly. The Pydantic integration adds a "None of these" option when the field type is an optional literal, which is cleaner than inferring absence from a low confidence value. An escape hatch in the option set beats a threshold you tuned by hand.
- Can I ask it to count or compute something?
- No. Arithmetic, counting and date reasoning are documented as unreliable. Compute those in your own code and pass the result into the state, where they cost nothing and are exact.