A field guide to evaluating retrieval systems
Recall@k tells you if you found the chunk. It tells you nothing about whether the chunk was useful.
Vedant
@vedant
Works on retrieval infrastructure. Writes about the parts of ML that turn out to be distributed systems problems.
- 5
- 12
- 4
- 1
The revision count is the point. A follower count measures how many people looked; this measures how many times the author was wrong in public and fixed it, which is the only credibility signal on this site that cannot be faked. The ochre number is a standing reminder, not a punishment — a post nobody has touched in six months is probably wrong somewhere.
Evergreen
Cache invalidation is a distributed systems problem
The TTL is not a cache policy. It is a probability distribution, and somebody has to own it.
Budding
Why your embeddings are worse than you think
Dimension buys separation, not meaning. The curve flattens hard, and three pipeline bugs were costing more than the model.
The p99 is a lie you tell yourself
You cannot add percentiles. Every latency budget that does is a budget nobody is actually keeping.
Seedlings
We deleted our staging environment
Six months in. The best thing we did was delete a database nobody had ever tested against.
Joined 3 weeks ago. Writes here →