A field guide to evaluating retrieval systems
Recall@k tells you if you found the chunk. It tells you nothing about whether the chunk was useful.
Vedant · 1 note in the margin · reviewed 3mo ago
Viewing revision 3 — changes shown against revision 2. Written 3 months ago.
← Back to the current versionconflating them is why retrieval evals feel like astrology
The astrology comparison lands because it is about a diagnostic that cannot disagree with you. A recall number goes up when you improve chunking, when you get lucky, and when you make the eval set easier. Nothing in the number tells you which.
Reply
Report
Almost every team that builds a retrieval system evaluates it the same way: hand-label a few hundred query–document pairs, measure recall at k, declare victory. The number goes up. Users do not get better answers. I have watched this happen four times, and in three of them the team had a dashboard that looked like it was working right up until the week it shipped.
Recall@k answers "was the gold document in the top k". That is a property of the index. Whether the reader — in this case the model — could use what it retrieved is a completely different question, and it is the one that determines whether anyone notices the feature.
| Layer | Question | Metric | What it diagnoses |
|---|---|---|---|
| Index | Did we find the right chunk? | recall@k, nDCG | Chunking, embeddings, index config |
| Context | Was it the right amount? | context precision, fill rate | Reranking, dedup, budget |
| Answer | Did it use the chunk? | groundedness, citation accuracy | The generator, not the retriever |
Teams that skip the middle row get a very confusing position: perfect recall, useless answers. The chunk was retrieved and then drowned by eleven near-duplicates of itself, and the model — correctly — ignored the pile.
Fix the index
One trap specific to the middle row: context precision measured against what the model cited under-reports, because a model can use a chunk without citing it. Pair it with an entailment check against the answer and the two together are considerably more honest than either alone.
Context precision is the most under-measured number in the stack and the one that moves outcomes the most. It is also trivial to compute: of the chunks you actually put in the prompt, what fraction earned their place.
def context_precision(retrieved, used, k=10):
"""Fraction of supplied chunks the generator actually relied on."""
supplied = retrieved[:k]
if not supplied:
return 0.0
return sum(1 for c in supplied if c.id in used) / len(supplied)
def diagnose(recall_at_k, context_precision, groundedness):
if recall_at_k < 0.7:
return "retrieval is broken — fix the index first"
if context_precision < 0.4:
return "retrieval is drowning the answer — add reranking, not a bigger k"
if groundedness < 0.6:
return "retrieval is fine, generation is not"
return "healthy"
def context_precision(retrieved, used, k=10):
"""Fraction of supplied chunks the generator actually relied on."""
supplied = retrieved[:k]
if not supplied:
return 0.0
return sum(1 for c in supplied if c.id in used) / len(supplied)
def diagnose(recall_at_k, context_precision, groundedness):
if recall_at_k < 0.7:
return "retrieval is broken — fix the index first"
if context_precision < 0.4:
return "retrieval is drowning the answer — add reranking, not a bigger k"
if groundedness < 0.6:
return "retrieval is fine, generation is not"
return "healthy"
These thresholds are calibrated against a mixed set of roughly 40 systems and should be treated as a starting point, not law. The relative ordering of the checks is the durable part; the numbers will need recalibrating as the models underneath you change. We have had to lower the groundedness bar twice as models got better at hedging.
The ordering matters more than any individual threshold. Improving groundedness when recall is 0.4 is a very expensive way to produce confidently wrong answers with citations.
How deep do you want this?
First time here? Set your depth once and every post honors it — skim the argument, read it properly, or take the footnotes too.
Full argument, prerequisites explained inline.
Revision history
current →2 added
Marginalia
1 note from 1 reader, attached to specific sentences. Select any sentence in the text to add yours.
Ask this article
Answers are quotes from this post, not generated text — so they can be wrong by omission but never by invention. What it does not cover, it says so. If you have brought your own model key, a model summarizes those same quotes, and the quotes stay the source of truth.
Answer with a specific model
Your key is used for your provider if you added one, otherwise the site's. Automatic stays on free-tier models only.
Try to break this argument
The claims this post makes most concretely — the ones with numbers, absolutes or prescriptions in them — put to a model that has read only the sentence in question. Every objection quotes the post, and a claim nothing can be said against is reported as standing.
Where this sits
It leans on nothing else here. 2 posts have picked it up since. Oldest at the top, so a claim resting on something written after it would be visible as an arc running upward.
When this changes
This post has been revised 2 times. Leave an address and you will hear the next time it moves — once, and only for a substantial change, not a typo.