Set a failure budget before you read the eval score
A single pass rate hides which failures your product is accepting. Turn an eval into a failure budget with explicit limits, owners and release decisions.
On this page 5 sections
- Set separate limits for distinct failure classes instead of trusting one blended score.
- Weight failures by consequence and exposure, not by how easy they are to generate.
- Make every exceeded limit trigger a named release decision.
A model can improve its overall eval score while getting worse at the one mistake users cannot safely recover from. The aggregate number will still look reassuring if enough easy cases pass. I would treat an eval as a failure budget: a small set of explicit limits on the failures the product can tolerate, each tied to a decision.
A pass rate is not a risk statement#
Suppose an assistant answers questions from a support knowledge base. An eval has 100 examples: 80 ordinary questions, 15 questions with incomplete information, and 5 where the correct response is to refuse or ask for clarification. A model that passes all ordinary questions and fails all five refusal cases posts a 95% pass rate. That number says little about whether the assistant is safe to expose to users.
The denominator is doing too much work. Each example counts equally, but examples differ in consequence and frequency. Worse, the eval designer may have included many variations of the easy cases because they were simple to write. The resulting score measures the shape of the test set as much as product quality.
A failure budget makes the hidden trade-off visible. Instead of one threshold, define limits by failure class. For example:
| Failure class | Release limit | Product response |
|---|---|---|
| Incorrect answer when source is clear | At most 3 of 80 | Fix retrieval or answer grounding |
| Confident answer when source is missing | 0 of 15 | Block release; require abstention or clarification |
| Refusal on an answerable question | At most 2 of 30 | Review refusal boundary and user recovery |
These are illustrative limits, not universal benchmarks. The important move is to make the threshold a product decision before looking at the result.
Define failures in terms users experience#
A taxonomy should describe what went wrong for the user, not only which model component may have caused it. “Retrieval failure” is useful to engineers, but it does not tell a PM what to count. “The assistant gave a specific answer despite having no supporting source” is observable and actionable.
For each class, write three things:
- Observable condition: What would a reviewer mark as a failure? Avoid labels such as “bad answer.”
- User consequence: What happens next if this failure reaches production? Can the user notice, correct, or reverse it?
- Exposure: How often is the triggering situation likely to occur in the real workflow?
Consider a generated summary that omits a qualification. A generic omission might be minor when a user can inspect the original document. The same omission could be critical if the summary is used as the only input to a later decision. The output text alone may be identical; the workflow context changes the consequence.
A practical annotation rule could say: “Mark unsupported certainty when the answer asserts a fact absent from the provided sources, without signalling uncertainty or asking for the missing information.” Two reviewers can apply that rule more consistently than “mark hallucinations.” Include counterexamples: a concise answer that leaves out irrelevant detail should not fail merely because it is not exhaustive.
Set limits using consequence and exposure#
A failure budget is not just a severity ranking. A rare catastrophic failure and a frequent recoverable error require different controls. Start with a simple risk sketch for each failure class:
- Severity: What is the worst plausible user consequence?
- Exposure: How often does the relevant situation appear?
- Recoverability: Can the user detect and undo the result before harm?
This need not pretend to be precise math. A low/medium/high rating is often more honest than multiplying made-up numbers into a risk score. Use the ratings to decide whether a class needs a zero-tolerance gate, a maximum count, or monitoring after launch.
For instance, unsupported certainty in a high-consequence workflow might have a zero-failure release limit on a targeted challenge set. A stylistic mismatch in a low-stakes draft assistant might be tracked as a trend rather than a blocker. The test size also matters: “zero failures” across five examples is weak evidence. State the number and type of examples next to the limit, then expand the set as you observe real cases.
Do not silently change the limit when the model misses it. If a release is still justified, record who accepts the exception, the affected workflow, the temporary mitigation, and when the decision expires. Otherwise a budget is just a dashboard decoration.
Make every limit lead to an action#
A threshold without an owner creates debate at the worst time: after a release candidate has failed. For each failure class, decide in advance what happens when the limit is exceeded.
A compact release table can include:
| Field | Question to answer |
|---|---|
| Failure class | What user-visible outcome are we counting? |
| Test slice | Which examples exercise it, and how many? |
| Limit | What is the maximum acceptable count? |
| Owner | Who decides whether the result is understood? |
| Action | Block, fix, add a guardrail, or accept a time-limited exception? |
The action can depend on the cause. If a failure appears only when source material is absent, a product-level clarification step may be more effective than prompt tuning. If the model ignores a clear source, retrieval or generation may need investigation. The taxonomy should support diagnosis without assuming the cause in advance.
This also separates two questions teams often collapse: “Did the system get better?” and “Is it safe enough for this workflow?” A model may improve on average while still exceeding a critical failure limit. Conversely, a small regression in a low-consequence class may be acceptable if a major user risk has been reduced. The budget preserves both facts instead of squeezing them into one score.
Keep the budget alive after launch#
Offline examples are a sample, not a complete map of production. When users correct an answer, abandon a flow, or ask for a human, those events can reveal missing failure classes. Review a small, privacy-appropriate sample of these signals regularly and turn confirmed patterns into eval cases.
Do not add every odd output as a new test. First ask whether it represents a distinct user-visible failure, whether the existing taxonomy already covers it, and what decision the test would change. A test that has no plausible effect on a release or product decision adds maintenance without adding control.
On Monday, take your current eval and write down its three most consequential failure classes. For each one, name the observable failure, the test slice and denominator, the release limit, and the person who owns the next action. If you cannot fill those in, the headline score is not yet a release criterion.