Your eval set is the real PRD
For AI features, the document that actually decides what ships isn't the spec. It's the list of inputs and the answers you'll accept. Write that first.
On this page 6 sections
- A PRD for an AI feature that has no example inputs and accepted outputs is a wish, not a spec.
- Start with 30-50 hand-written cases before any prompt work; grow it from real failures.
- Grade with rules first, rubrics second, LLM judges last, and audit the judge.
- Make the eval set the launch gate, so "is it good enough?" stops being a meeting.
Every AI feature I’ve seen go sideways had a beautiful PRD. Problem statement, personas, success metrics, a crisp scope section. What it didn’t have was a single page that said: here are fifty things a user will actually type, and here is what a good answer looks like for each.
That missing page is the spec. Everything else is context.
Why the classic PRD breaks for AI features#
A traditional PRD describes deterministic behaviour. “When the user clicks Export, generate a CSV with these columns.” Engineering can read it, build it, and QA can check it. The behaviour is the spec.
An LLM-backed feature doesn’t work that way. “Summarise the customer’s support history” has a thousand valid outputs and a million bad ones. Two engineers reading the same sentence will build two different prompts, and both will pass a demo. The disagreement only shows up in production, on the inputs nobody imagined.
So the question a PM has to answer is no longer what should it do? It is how will we know, case by case, whether it did it? That answer lives in an eval set.
What a useful eval set looks like#
Not a benchmark. Not a leaderboard. A spreadsheet the whole team can read, with one row per case:
| Column | What goes in it |
|---|---|
| Input | The exact text, file or context the model receives |
| Slice | Which user, segment or edge case this represents |
| Must | Hard requirements: facts that must appear, formats, refusals |
| Must not | Things that make the answer wrong regardless of quality |
| Good looks like | A reference answer, or a short note on what “great” means |
| Severity | What a failure here costs: annoying, wrong, or dangerous |
The Must not column does more work than people expect. Most AI failures that hurt users aren’t vague answers. They’re confident, specific, wrong answers: an invented number, a policy the company doesn’t have, a recommendation outside what you’re allowed to give. Writing those down up front is how you turn “be accurate” into something testable.
The Severity column is what lets you make launch decisions later. A 92% pass rate means nothing on its own. A 92% pass rate where every failure is a formatting nit is shippable. A 99% pass rate where the 1% is a wrong compliance answer might not be.
Write the first 40 cases by hand#
The temptation is to generate the eval set with the same model you’re evaluating. Resist it for the first batch. Synthetic cases inherit the model’s blind spots; you end up testing whether it agrees with itself.
Instead, before anyone touches a prompt:
- Pull 15-20 real inputs from support tickets, sales calls, search logs, whatever exists. Messy, typo-filled, half-formed. That’s the point.
- Write 10 adversarial ones yourself: ambiguous requests, out-of-scope asks, inputs that tempt the model to guess.
- Add 10 boring ones. The happy path still needs to be fast and right, and it’s most of your traffic.
- Fill in Must / Must not for each with whoever owns the domain: ops, legal, the support lead. This is the conversation that used to happen after launch, moved to week one.
Forty cases sounds small. It’s enough to catch most prompt regressions and, more importantly, it forces the team to agree on what “good” means before there’s a working demo to fall in love with.
If you can’t write the expected answer for a case, the model can’t either. That’s a scoping problem, not a prompting problem.
Grade cheaply, then carefully#
Order your graders by how much you can trust them:
- Rules first. Exact matches, regex, JSON schema validation, “does the cited ID exist in the source”. Fast, free, never wrong about what they check.
- Rubrics second. For judgement calls, a short rubric (3-5 criteria, each pass/fail) that a human can apply in under a minute.
- LLM-as-judge last. Useful for scale, but it’s another model with its own biases: it tends to favour longer answers and answers that sound like itself. Before trusting it, grade 30 cases by hand, run the judge on the same 30, and look at every disagreement.
Make it the launch gate#
The real payoff comes when the eval set stops being a QA artefact and becomes the contract:
- Every prompt or model change runs the full set before merge. Results are posted where the team can see them.
- Launch criteria are written against it: zero failures on severity-3 cases, at least 90% on severity-2, latency under X at p95.
- Every production failure a user reports becomes a new row. The set grows from reality, not imagination.
This changes the texture of the work. “Is it good enough to ship?” stops being a vibes debate in a meeting room and becomes a number with a known denominator. Model upgrades become a 20-minute experiment instead of a two-week anxiety spiral. And new teammates can read the eval set and understand the product’s edges faster than any PRD would teach them.
The Monday version#
If you own an AI feature and don’t have this yet, here’s the smallest useful step: open a spreadsheet, add the six columns above, and write ten real inputs with their Must and Must not. Show it to your engineer and your domain expert in the same meeting.
You’ll find at least one case where the three of you disagree on the right answer. That disagreement was always going to surface. Better in a spreadsheet this week than in a support ticket next quarter.