Write the error budget before you add an AI feature
A product brief for an AI feature should name which mistakes are tolerable, who absorbs them, and what happens when the system runs out of room. Here is a practical way to set that boundary.
On this page 6 sections
- Define unacceptable outcomes before debating model quality.
- Assign a limited error budget by impact, not by one blended accuracy score.
- Specify the fallback and owner for every exhausted budget.
An AI feature can look reliable in a demo and still be a poor product bet. The missing question is often not “How accurate is it?” but “Which mistakes are we willing to let reach a user, and what do we do when they happen?” If the team cannot answer that before launch, it has not defined the feature’s reliability contract.
Start with the user-level harm#
“Wrong answer” is too broad to guide a product decision. A spelling mistake in a generated draft and a fabricated eligibility reason may both count as incorrect outputs, but the cost and recovery path are different.
Write down a small set of outcome classes before selecting a metric. For a hypothetical assistant that drafts replies to customer messages, the classes might be:
| Outcome | Example | Product response |
|---|---|---|
| Cosmetic defect | Awkward phrasing, meaning preserved | User edits before sending |
| Material omission | Misses a detail needed for a complete reply | Show the source details and require review |
| Unsupported claim | Adds a fact absent from the message or policy | Block sending; let the user rewrite |
| Sensitive disclosure | Includes private information belonging to someone else | Stop the workflow and route for investigation |
This is not a universal taxonomy. It is a prompt to describe failures in terms of what a user experiences and what the product must do next. Keep the categories few enough that a reviewer can classify a test case without a debate.
Give each class its own boundary#
A single “acceptable error rate” hides the distribution of risk. If harmless style defects dominate a test set, the overall score can look strong while a rarer but consequential failure remains invisible.
For each outcome class, set a boundary in plain language. For instance: “A draft may need stylistic edits, but it must not invent a policy or send without a human confirmation.” If you have enough representative examples to quantify the boundary, record the denominator and the threshold. If you do not, say that the evidence is insufficient; do not manufacture a percentage to make the brief look precise.
A useful planning table has four fields:
- Failure class: the observable user-level mistake.
- Exposure: where in the workflow it can reach a person or trigger an action.
- Limit: what frequency or severity is tolerable for this release, supported by evidence where possible.
- Containment: what prevents further impact once the limit is crossed.
“Improve accuracy” belongs nowhere in that table. It is an intention, not an operating boundary.
Make the denominator match the decision#
A model score is meaningful only when the test examples represent the product decision. For a drafting feature, counting every generated sentence as an observation can obscure the more useful unit: a draft that a user might send. For a triage feature, the consequential unit may be a case routed to the wrong queue.
Write the denominator beside every proposed threshold. “No more than two unsupported claims per hundred evaluated drafts” is interpretable only if the drafts cover the relevant customer intents, policy exceptions, and incomplete inputs. A hundred near-identical happy-path prompts do not provide that evidence.
Separate severity from frequency as well. A low-frequency event can still be unacceptable if its impact is severe. In that case, the release boundary may be categorical: the feature cannot take the consequential action autonomously, regardless of its average score. That can be a more defensible product decision than trying to justify a tiny numerical allowance from a small test set.
Treat the budget as something that can be spent#
An error budget is useful only if a team knows what consumes it. Decide whether a failure counts when it appears in pre-release evaluation, in monitored production traffic, or both. These answer different questions: evaluation estimates readiness; production monitoring tracks what users actually encounter.
For example, a team might require zero observed unsupported policy claims in a defined set of high-risk test scenarios before enabling automatic sending. In production, it could track user-reported or reviewer-confirmed incidents separately, with a pause condition if any such incident occurs. The exact gate depends on the product’s consequences and evidence; the important point is to define the event and response before launch.
Do not let a budget become a target to spend. A threshold is a ceiling for decision-making, not permission to expose users to predictable failures. If the budget is exceeded, pause the relevant capability, preserve the context needed to investigate, and assign an owner to decide whether to repair, narrow, or remove it.
Design the exhausted-budget state#
Many briefs describe only the successful path. The reliability contract must also say what happens when the system is uncertain, unavailable, or beyond its error boundary.
For the draft assistant, the exhausted-budget state could be: stop generating send-ready text, keep the original customer message visible, and offer a blank reply editor. The user still completes the task, but the product no longer presents an unsafe draft as if it were usable. That fallback should preserve the user’s progress and explain the limitation without claiming certainty about why the model failed.
Assign an operational owner too. Who reviews a flagged incident? Who can disable the capability? What signal tells them to act? A feature flag without a named responder is not a complete containment plan.
Put the contract in the brief#
Before implementation, add a compact reliability section to the product brief:
- Allowed: errors the user can safely notice and correct.
- Blocked: errors or actions that must not reach the user uncontained.
- Evidence: evaluation examples, denominator, and known gaps.
- Fallback: the usable non-AI path when confidence or service is insufficient.
- Tripwire: the event that pauses or narrows the feature.
- Owner: the person responsible for reviewing the tripwire and deciding next steps.
Then ask one question in review: If this exact failure happens tomorrow, what does the user see, and who acts? If the answer is vague, the feature’s error budget is not ready to ship.