Why your LLM tests miss edge cases

Your eval set has a leak. It’s not a bug in the harness or a flaky assertion. It’s a systematic gap: you’ve written tests the model can pass, which is not the same as tests the model will pass.

When you test software, you map inputs to outputs. You cover branches: the if-statement fires, the loop runs, the edge case at zero doesn’t crash. You achieve coverage. The model, though, doesn’t execute branches. It computes probability distributions over tokens. A test that passes isn’t evidence the model is robust; it’s evidence the test was written in a way the model learned to handle.

The deterministic illusion#

You write a test like:

Input: "What is 2 + 2?"
Expected: "4"
Result: PASS ✓

You write fifty more like it. The model passes all fifty. You think you’re done.

What you’ve actually done: you’ve found fifty points in semantic space where the model learned to output the right token. You haven’t tested robustness. You’ve tested fifty inputs that look like the ones you thought of.

The problem: your eval set covers a small patch of a very large input space. The moment a user writes something you didn’t anticipate, a different phrasing, a harder number, a logical twist, you have no evidence either way about what the model will do.

Where edge cases hide#

In software, edge cases cluster at boundaries: zero, negative numbers, empty lists, null pointers. You find them by reasoning about the code. In LLMs, edge cases are distributed across semantic space. They’re not at boundaries; they’re everywhere you didn’t write a test.

Consider a feature that summarizes legal documents. Your eval set has:

• 40 summaries of mortgage agreements ✓
• 30 summaries of employment contracts ✓
• 20 summaries of NDAs ✓
• 10 edge cases: very short documents, documents with unusual formatting

You test all ninety. The model passes. You ship.

A user gives it a contract written in fragments, with embedded definitions that reference other clauses. The model hasn’t seen this structure in your eval set. It fails.

The test you missed wasn’t at a boundary. It was in a region of input space your examples didn’t cover. You can’t reason your way to it. You can’t enumerate all possibilities. You have to find it.

Two practical moves#

Adversarial sampling. Have a person—or another LLM—write inputs specifically designed to break your model. Not random inputs. Targeted attacks: ambiguous phrasings, contextual traps, logical puzzles where the model usually fails. Evaluate your model on these adversarial examples. Every failure is a new eval row; every pass is evidence for that region, not the whole space.

Statistical coverage instead of case coverage. Instead of “I’ve tested fifty inputs,” think: “I’ve sampled from a distribution of inputs. How much variance does my eval set cover?” Take your 50 test cases and measure the semantic diversity: use embeddings, cluster them, plot them. Are you sampling from across the space, or just hitting the same region repeatedly? If you see three tight clusters in embedding space, you’re testing three easy cases over and over.

The pattern to watch for#

If your eval set is written by the team building the feature, it will lean toward cases the team thought of and cases the prompt was already tuned against. Ask someone outside the team, a user, a support lead, another PM, to write test cases without seeing your prompt. Compare their eval set to yours. The tests they write that your model fails are the gaps you can’t see from the inside.

The real rule#

A passing eval set is not evidence of robustness. It’s a report of what your tests could measure. Robustness is what happens when the model sees inputs outside your test’s semantic cluster.

Build an eval set that moves. Rotate in fresh adversarial examples. Sample from outside your comfort zone. The edge cases you find by searching are edge cases you can fix.