Stop Averaging Search Evals

Looking for TL;DR? Check key takeaways

Any serious team tracks latency as p50, p90, p99. Reporting just "our average latency is 40ms" is naive, because it hides the p99 150ms tail that's hurting real users. Yet we report search relevance as a single average NDCG all the time, and somehow that's normal. Relevance has a tail too, but the average hides it.

Distribution vs average

I ran a plain word-BM25 baseline over 12 NanoBEIR datasets (599 queries). Instead of averaging each metric, I calculated percentiles over the per-query scores. The mean NDCG@10 is 52.5, which feels respectable, as if every query comes back roughly half-right. In reality, they range from perfect to nothing: one in five queries retrieves nothing relevant in the top 10, and the average buries every one.

(Every score here is ×100, the way IR papers report NDCG. A mean of 52.5 reads better than 0.525, and a +2.2 gain reads as a real change where +0.022 reads as rounding.)

Let's look at the distribution.

Observations:

• 119 queries score exactly 0 and 129 score exactly 100. That's 41% of all queries sitting at the two ends, and the mean is neither.
• Only 43 of 599 queries land within 5 of the mean (52.5), so it describes a query that barely exists.
• Switch to recall@10 and it gets worse: 63% of queries at the two ends, 13 near the mean.

The mean hides the reality

Read the same three runs as percentiles instead of averages:

Observations:

• p10 is 0.0 on both NDCG@10 and recall@10, so over 10% of queries score a hard zero while the mean sits near 53.
• On recall@100 the typical query is perfect (p50 = 100), but a tail that bottoms out at p10 = 12.6 pulls the mean to 76.5, a number no real query got.
• The mean's error flips sign between metrics. It overstates recall@10's median (56.3 vs 50.0) and understates NDCG@10's (52.5 vs 55.4), so you can't correct for it in your head.

It hides your improvements, too

Take a real improvement: plain word BM25, then apply a Porter stemmer and a stopword list (word_en). It moves mean NDCG@10 by +1.9, a bump you'd squint at.

A +1.9 mean could be every query gaining 1.9. Here it's three different groups:

• 294 of the 599 queries don't move at all
• 172 improve, by 19.5 on average
• 133 get worse, by 16.7 on average

(172 × 19.5 − 133 × 16.7) / 599 = +1.9

So report percentiles of Δ, not the Δ of percentiles. Subtracting two summaries says NDCG@10's median gained +3.7. Asking each query how far it moved says it gained 0.

Observations:

• The median Δ is 0 on all three metrics. The typical query is untouched.
• NDCG@10 moves the most queries and is the least one-sided: 172 wins against 133 losses, p10 Δ −11.7.
• recall@100 is the most forgivingit only asks whether a relevant doc landed anywhere in the top 100, so a doc can slip 90 places and still count metric and yet we see 84 wins against 32 losses. So even a stemmer that's considered helpful can hurtECS stems to ec, so the only distinctive term in one ClimateFEVER query stops being distinctive and it drops 33 → 0 at times.
• Some queries swing the whole way, 100 → 0 and 0 → 100 (hover a bar to see which).

Reported properly, that run is:

The mean is the smallest number in that Δ column, and p25 moves 2.4× as far.

Then there's win/tie/loss: 172/294/133. It needs a baseline scored on the same queries, which is why it rarely gets reported. 56% of the queries that moved got better, and +1.9 doesn't say that.

Splitting the mean into what was gained and what was given back:

Every mean here is a small difference between two larger numbers. NDCG@10 gives back two thirds of what it wins, recall@100 a quarter. Same-looking gains, different trades.

A saturated percentile can't show a gain. NDCG@10's p90 is already at 100, so a better method moves it +0.0. The 22% of queries already scoring 100 have no room left to improve, and they dilute the mean.

The relevance tail

Latency and relevance are the same problem pointed at opposite tails. For latency, higher is worse, so you watch the high end: p90, p99. For relevance, higher is better, so the tail that hurts is the low end. That's the relevance tail: the weaker half of your queries, which you read at p25.

Report p25, not p10. p10 pins at 0 once more than 10% of queries fail, and 20% do here, so it can't show the tail moving. p25 is 21.9 on NDCG@10 and still moves when the method changes.

I timed this BM25 baseline at 3.2ms p50, 7.2ms p90. Nobody would report only the 4.4ms mean, because the p90 is the request a real user is waiting on. A p25 relevance score is that same user's search. We just forget to report it.

So what should we report instead?

Improving a system and reporting one need different things.

To improve it, start with real queries and why they ranked what they did. Then the shape of the distribution. Summary numbers come last.

To report it, print p50 and p25 next to the mean. Three numbers is still few enough to remember and compare, and you get the typical query and the weak tail.

Zero-rate and win/tie/loss come out of the same per-query scores if you want them. Win/tie/loss earns its keep when you're comparing two of your own runs, since it needs a baseline scored on the same queries.

Beyond search: any eval with graded per-item scores, an LLM judge's rubric, a similarity score, a per-query metric, hides its tail in the mean the same way.*Binary pass/fail evals are the exception: there the mean already tells you everything.

Mean alone can't capture a distribution. We realised this for latency decades ago. Search relevance deserves the same.

(This fell out of benchmarking token search, where I was macro-averaging NDCG across datasets and realised the average was quietly deciding my conclusions for me.)

Key Takeaways

• A good average can hide queries that are hurting real users. A mean of 50 could be every query scoring 50, or half the queries scoring 100 and half scoring 0.
• It hides your gains too. Adding a stemmer and stopwords moves mean NDCG@10 by a forgettable +1.9, while 133 of 599 queries actually get worse.
• Report p50 and p25 next to the mean. Higher is better for relevance, so the tail that hurts is the low end. p25 is the relevance equivalent of p90/p99 latency.