The mean lies, and here's proof
You've rolled out a new caching tier. The dashboard shows mean latency went from 112 ms to 122 ms. SEV time, right? Not so fast. When you look at the median, it dropped from 120 ms to 60 ms. The p99 more than doubled. Same data, three different stories. This isn't a contrived example—it's a real problem the author hit while validating lld performance improvements at their day job.
One number, four stories
Percentiles are the standard fix. But as the author shows with a synthetic dataset (fixed seed, available in a gist), the p50, mean, and p99 can point in opposite directions. The mean says regression, the median says huge win, the p99 says SEV1. Everyone is right, and that's the problem.
Look at the shape
Plotting the density of the before and after distributions reveals the culprit: the after distribution is bimodal. One hump for fast requests, another for slow ones. The mean gets dragged up by the slow hump, while the median reflects the fast one. But density plots are fiddly—the smoothing parameter changes the shape, and it's hard to read percentiles off them.
The CDF: your new best friend
The cumulative distribution function (CDF) answers: what fraction of requests finished in ≤ x milliseconds? Plotting before and after CDFs on the same chart shows the entire distribution in one view. In the example, the after curve is higher (better) for latencies below 140 ms, and lower (worse) above it. The curves cross at ~140 ms—the tipping point. That's the unmistakable signature of a change that no single percentile can summarize.
The shift function: how much, at each percentile
The CDF tells you the sign changes, but not the magnitude. The shift function plots after-latency minus before-latency for every percentile. It shows exactly how much faster or slower each percentile got. In the example, the p50 improved by ~60 ms, but the p99 got worse by ~200 ms. That's actionable.
Rollouts are not instant
Looking at snapshots misses the trend. The author simulated a week-long rollout, 0% to 100% traffic. A ridgeline plot (one density per day) shows the fast peak sliding left as the rollout progresses, while a slow peak emerges on the right. A heatmap with one column per day shows the same. Any aggregate over the whole week would have hidden this entirely.
Why bimodal? Because of cache hits vs misses
Splitting the after requests by cache hit or miss shows each population is unimodal. Cache hits are faster than the old baseline; misses are slower due to the extra hop. But why do some requests miss? The author adds response size: a jointplot of latency vs size, colored by hit/miss, shows two clean clusters—small-and-fast (hits) and large-and-slow (misses). The bimodality in latency is caused by bimodality in response size. Actionable fix: raise the max object size or split large responses.
Do this now
Stop trusting the mean. For any latency-sensitive system, add a CDF to your dashboard. When comparing before/after, plot both CDFs on the same axes. If they cross, you have a bimodal distribution—dig deeper. Use a shift function to quantify the change per percentile. And always condition on likely causal variables (like cache hit/miss, response size) before concluding anything.
The takeaway
The author's post, with its reproducible nix-shell script, is a masterclass in data visualization for debugging. As they quote Churchill: "Do not trust any statistics you did not fake yourself." In this case, the data was faked—but the lesson is real. Your dashboards are lying to you. Learn to read the shape.
