The p99 is a lie you tell yourself

There is a specific kind of incident where every dashboard is green and the customers are still complaining, and the cause is almost always this: someone averaged a percentile.

Why percentiles do not add

A p99 is the value that 99% of requests came in under. It is a property of a distribution, not a quantity you can decompose. If service A has a p99 of 40ms and service B has a p99 of 60ms, you have learned essentially nothing about the p99 of the pair, because the 1% that is slow in A and the 1% that is slow in B are very unlikely to be the same 1% of requests.

The inequality is the interesting part. p99(A + B) ≤ p99(A) + p99(B) holds always, so summing can only ever overstate — the direction is not in doubt. What is in doubt is the size of the error, and that depends entirely on how correlated the tails are. If your hops fail together, the sum is exactly right. If they fail independently, a real fraction of the budget you just committed to is imaginary. Nothing in either dashboard tells you which world you are in.

Worth being precise about why 4% is a floor rather than an estimate. Real hops are correlated by traffic shape — a cache stampede hits the edge, the service and the database simultaneously — which pushes the true figure higher. You will not see a spike where all four are slow at once very often, but when you do, it is catastrophic rather than merely bad.

What to do instead

Measure the percentile on the whole request, at the edge, with the trace id attached. That is the only p99 that means anything. Everything else is a component-level diagnostic wearing a system-level label.

• —Instrument at the edge. One p99 per incoming request, not four per hop.
• —Keep per-service percentiles, but label them as diagnostics. Never sum them into a budget.
• —If you need an early warning, alert on the edge p99 and use per-hop data to explain it.
• —Budget in milliseconds of expected latency and verify the tail separately — the two are different questions.

The uncomfortable part: this requires trace propagation to be correct, which is a larger project than the SLO was. That is the real cost of a meaningful latency budget, and it is the cost people avoid by averaging. I have now watched three teams take the averaging route and every one of them ended up with an SLO that improved exactly as much as the service got worse.

A caveat on the 4% figure, which readers have correctly pushed back on twice. It assumes the slow tails of the four hops are independent. Under a shared cause — a noisy neighbour, a deploy that touched every service, a GC pause — the tails are highly correlated and naive addition is closer to right. That is precisely why the mistake survives: it is right often enough to feel validated, and the incidents where it is badly wrong are the ones you do not get a page for.