Stop Averaging Search Relevance

Looking for TL;DR? Check key takeaways

Any serious team tracks latency as p50, p90, p99. Reporting just "our average latency is 40ms" is naive, because it hides the p99 150ms tail that's hurting users. Yet we report search relevance as a single average NDCG all the time, and somehow that's normal. It's the same mistake, and for relevance the average is still the default.

I ran a plain word-BM25 baseline over 12 NanoBEIR datasets (599 queries). Instead of averaging each metric, I pooled the per-query scores and read the percentiles. The mean NDCG@10 is 0.526, which feels respectable, as if every query comes back roughly half-right. In reality, they range from perfect to nothing: one in five queries retrieves nothing relevant in the top 10, and the average buries every one.

The mean hides the floor

Pool the per-query scores and the distribution shows what the mean can't:

p10 is 0.000 for both NDCG@10 and recall@10: over 10% of queries score a hard zero while the mean sits near 0.53. Two subtler things the average also hides:

1. recall@100: the typical query is perfect, the mean says 0.765. The median query recalls every relevant doc (p50 = 1.000), but a tail flooring at p10 = 0.126 drags the mean down to a number no real query got.
2. The mean's error flips sign between metrics. It overstates recall@10's median (0.563 vs 0.500) and understates NDCG@10's (0.526 vs 0.554), so you can't correct for it in your head.

The distribution is the whole story. Here's per-query recall@10, bucketed:

Two spikes, a hollow middle: 43% of queries score exactly 1.0 and 20% exactly 0 (63% at the extremes), while the mean, 0.563, sits in the (.5,.6] valley where about 8% of queries live. It describes almost no real query, which is why p50 and p10 tell you more than the average can.

It hides your improvements, too

The mean also hides your gains. Take a real improvement: plain word BM25, then apply a Porter stemmer and a stopword list (word_en). It moves mean recall@10 by +0.022, a bump you'd squint at. But look where the gain lands:

Same gain, six ways: the mean smears it across every query (+0.022); the median reads it where the queries can actually move (+0.100, 4.6×). The p10, p75, and p90 bars sit at +0.000 because those percentiles are saturated (p10 at 0, p75 and p90 at 1.0), and a saturated percentile can't show a gain. The pattern holds across metrics:

The gain shows up wherever the live mass sits: the median for recall@10, the lower quartiles for recall@100 whose median is already maxed. A saturated percentile moves nothing; the informative one moves 2–5× the mean.

Why p10, not p90

Latency and relevance are the same problem pointed at opposite tails. For latency, higher is worse, so you watch the high end: p90, p99. For relevance, higher is better, so the tail that hurts is the low end: p10, the worst-query floor. It's the same instinct, just mirrored.

And it is instinct for latency. I timed this BM25 baseline at 3.9ms p50, 8.6ms p90. Reporting only the 5ms mean would be a mistake, because the p90 is the request a real user is waiting on. p10 relevance is that same user, on the tail that actually hurts. We just forget to report it.

The ask

Report the median, the failure rate, and the p10, not the mean. Keep the mean only as a comparison column if a baseline reports it. Better still, show the whole distribution, whether a histogram or the five-number summary. It costs nothing: you already computed every per-query score to average them, then threw the shape away.

None of this is new machinery. IR has had geometric-mean AP and per-topic zero-rates since the TREC Robust track; they never caught on because leaderboards need one number to rank on and every eval tool prints the mean by default.

Concretely, for any retrieval eval:

• p50: the typical query. Your headline, instead of the mean.
• p25: the lower quartile, where the weaker queries sit and where a real improvement usually shows first.
• zero-rate: the share of queries that return nothing. This is your failure number when the tail is fat.
• p10: the worst-query floor, useful when failures are rarer than 10%; past that it pins at 0 and the zero-rate takes over.
• mean: fine for comparing runs, just not on its own.

Reporting the distribution also keeps you honest. You stay aware of the queries that fail instead of averaging them out of sight, and a genuine improvement stays visible where it lands instead of shrinking into a +0.022 you can't feel. Percentiles keep you closer to the reality your users actually experience.

Beyond search: any eval with graded per-item scores, an LLM judge's rubric, a similarity score, a per-query metric, hides its tail in the mean the same way. (Binary pass/fail evals are the exception: there the mean already tells you everything.)

One number can't capture search relevance. We worked this out for latency a decade ago. Relevance deserves the same.

(This fell out of benchmarking token search, where I was macro-averaging NDCG across datasets and realised the average was quietly deciding my conclusions for me.)

Key Takeaways

• One in five queries retrieves nothing relevant. On 12 NanoBEIR datasets the mean looks like ~0.53, but 20% of queries return nothing in the top 10 (so p10 = 0.000), and the average hides every failure.
• The mean hides gains, too. Adding a stemmer + stopwords moves mean recall@10 by a forgettable +0.022, while its median jumps 0.50 → 0.60 (+0.100, 4.6×), visible only in the percentile where the mass can move.
• Report p50 + p10, or the full distribution. For relevance, higher is better, so the tail that hurts is the low end: p10 is the mirror of p90/p99 latency.