Why your embeddings are worse than you think
Dimension buys separation, not meaning. The curve flattens hard, and three pipeline bugs were costing more than the model.
Vedant · 1 note in the margin · reviewed 28d ago
There is a particular kind of infrastructure decision that gets made by default rather than by measurement: how many dimensions. 1536 feels safe. 3072 feels careful. 768 feels like a compromise, and compromises are what people choose when they have not measured. I spent most of a quarter measuring it, because we had a very expensive index and no evidence about whether the width was the reason it was expensive.
1536 feels safe
The "decisions made by default rather than by measurement" framing is the whole post. We spent six months and a very large index re-embedding our entire corpus to move from 1536 to 768, having never run the measurement that would have told us to do it in week one.
Reply
Report
Dimension buys capacity to separate things that should be separated. It does not buy fidelity of meaning. Past a modest threshold, the returns are dominated by the quality of the training data and the normalisation, not the width.
71.0% recall at 128 dimensions — +0.0 points over the 128-dimension baseline.
The curve flattens hard around 768. Everything past that is index size and query latency for a fraction of a point of recall, and that fraction is inside the noise of a 300-query eval set.
I originally attributed the chunking gain to overlap. Overlap is almost irrelevant. The gain was entirely from not embedding navigation text — with a 512-token window and a 200-token overlap, a third of every chunk was table of contents links whose embeddings are near-identical across the entire corpus, and near-duplicates in an ANN index are exactly what destroys context precision.
- Normalising vectors before the ANN index. Free, and worth about two points.
- Stripping boilerplate — nav, footers, cookie banners — from chunks before embedding. Worth about four.
- Chunking on structure rather than a fixed token window. Worth about six, and it is the only one that helped context precision too.
Notice that none of those are about the model. The first three are about your pipeline being honest about what it is embedding.
The measurement was run twice: once on support tickets, once on internal queries. The curve shape held on both, but the absolute values were about four points higher on internal queries, because internal query phrasing is more regular. If you are picking a corpus for your own eval set, that gap is worth knowing about.
How deep do you want this?
First time here? Set your depth once and every post honors it — skim the argument, read it properly, or take the footnotes too.
Full argument, prerequisites explained inline.
Revision history
Claim a handle to fork this post into your own seedling.
Marginalia
1 note from 1 reader, attached to specific sentences. Select any sentence in the text to add yours.
Ask this article
Answers are quotes from this post, not generated text — so they can be wrong by omission but never by invention. What it does not cover, it says so. If you have brought your own model key, a model summarizes those same quotes, and the quotes stay the source of truth.
Answer with a specific model
Your key is used for your provider if you added one, otherwise the site's. Automatic stays on free-tier models only.
Try to break this argument
The claims this post makes most concretely — the ones with numbers, absolutes or prescriptions in them — put to a model that has read only the sentence in question. Every objection quotes the post, and a claim nothing can be said against is reported as standing.
Where this sits
It leans on 1 other post. Nothing has picked it up yet. Oldest at the top, so a claim resting on something written after it would be visible as an arc running upward.
When this changes
This post has been revised 1 time. Leave an address and you will hear the next time it moves — once, and only for a substantial change, not a typo.