Audit your RAG chunks before you tune the model
If retrieval hands the model the wrong passage, no prompt or model swap can recover the answer. Grade your chunks before you tune anything downstream.
On this page 5 sections
- Every retrieval failure starts as a chunking failure.
- Two signals to measure: retrieval relevance and chunk coverage.
- The right chunk size depends on the query shape you expect.
- Run the audit before you start tuning the prompt or model.
A common RAG habit is to tune the prompt first: pick a chunk size that sounds plausible, 256 or 512 tokens, then spend weeks swapping models and rewriting the prompt to get better answers.
This is backwards. If your retrieval returns the wrong context, no model will give you the right answer. The chunk step is the foundation; everything else builds on it.
What a chunk audit looks like#
Take 100 random queries from your logs, or write them by hand if you’re still in design. For each query, look at the top retrieved chunk and score two things:
- Retrieval relevance — Is the returned chunk about the thing the user asked?
- Chunk coverage — Does the chunk contain enough information to answer the query?
The first tells you whether retrieval is pointing at the right neighbourhood. The second tells you whether you’re showing the whole answer or a sentence fragment.
Write both scores in a spreadsheet. Look for patterns.
The patterns#
| Pattern | What it means | Fix |
|---|---|---|
| Low relevance, low coverage | Split points cut through passages, so no single chunk carries the answer and the embedding is diluted. | Split on structure (headings, paragraphs) instead of a fixed token count. |
| High relevance, low coverage | The chunk is about the right topic but doesn’t contain the full answer. You need more context. | Make chunks larger, or implement sliding-window retrieval. |
| Low relevance, high coverage | Your embeddings are off—or your query is vague. | Improve the query (query expansion, rewrite) or the embedding model. |
| High relevance, high coverage | You’re done. Move on. | Proceed to prompt tuning. |
Choosing a chunk size#
There is no universal best size. It depends on:
- How your users ask — Short, precise queries need smaller chunks; conversational queries need larger ones.
- Your document structure — Academic papers, Slack threads, and Notion databases split differently.
- Your tolerance for missing information — Medical and legal applications cannot afford gaps.
Test three sizes per document type: (average_paragraph_size), (average_paragraph_size × 2), (average_paragraph_size ÷ 2).
Before you tune the prompt#
Set a bar before you look at answers. I’d start at 80 of 100 queries scoring high on both. Below that, stop tuning the prompt: the model is answering from the wrong material.
Go back to chunking. A model swap changes how well the answer is written; it cannot add a fact that retrieval never returned.
The rule#
Write your eval set for retrieval first. One row per query: expected chunk IDs, coverage completeness score. Grade your chunking before you grade your answers.
Your chunk decisions lock in your answer ceiling. Get them right at the start.