- Published on
Stop Averaging Search Evals
A better average can still hurt queries
Looking for TL;DR? Check key takeaways
Adding a stemmerit cuts words down to a shared root, so returns becomes return and a stopword listit drops common words like the, of and in is standard advice for keyword search. I got a gain of +3.9 in mean NDCG@10a 0 to 100 score for one query, checked against docs that humans labelled relevant. 100 means the most relevant docs sit at the top of the results, 0 means none made the top 10. Scores here are ×100 of the usual 0 to 1 scale, the way IR papers report them on DBPedia, which feels like a clear win. But 29% of queries got worse, and imdb batman returns started returning Return of the Joker instead of Batman Returns.
Any serious team tracks latency as p50, p90, p99. Reporting just "our average latency is 40ms" is naive, because it hides the p99 150ms tail that's hurting real users. Yet we report search relevance as a single average NDCG all the time, and somehow that's normal. Relevance has a tail too, and the average hides it.
I'll use search as the example, but this applies to any eval with graded scores, like LLM-judge ratings.
Average can't represent the distribution
I ran a plain BM25the standard ranking function for keyword search baseline (word: lowercase and split into words, nothing else) on the full DBPedia-entitya BEIR retrieval benchmark: 4.6M Wikipedia entity pages, title plus abstract, and 400 queries with graded relevance labels set: 4.6M Wikipedia pages and 400 queries, with about 16 relevant pages per query. Instead of averaging each metric, I calculated percentiles over the per-query scores. The mean NDCG@10 is 27.1, which reads like every query finds a couple of the right pages. In reality, 25.5% of queries retrieve nothing relevantnothing the dataset's judges marked relevant. On these zero queries, 68% of the top-10 pages were never judged at all, so some zeros are missing labels, not failures. The stemmed run also lands on judged pages more often, 38% unjudged against 43%, which flatters it a little in the top 10, and the average hides this truth.
Let's look at the distribution.
Observations:
- 25.5% of queries score exactly 0. That's the tallest bar by far.
- The rest spread out from 3 to 100, so almost no query scores near the mean of 27.1. It describes a query that barely exists.
- A perfect score is rare (1.25% of queries), because the top 10 must hold the highest-graded pages, in grade order.
The relevance tail
For latency, higher is worse, so you watch the high end: p90, p99. For relevance, higher is better, so the tail that hurts is the low end. That's the relevance tail, the weaker half of your queries. p25 is the score a quarter of your queries get or worse, so it's the number I watch.
Report p25, not p10. p10 pins at 0 once more than 10% of queries fail, and 25.5% do here, so it can't show the tail moving. For plain BM25 even p25 sits at 0 here. It moves once a change pushes the zero-rate (the share of queries with nothing relevant in the top 10) under 25%, and the stemmer below does exactly that.
Understanding impact of your changes
Take the stemmer from the start: plain word BM25, then apply a Porter stemmer and a stopword list (word_en). It moves mean NDCG@10 by +3.9, and that gain is real (a paired t-test gives t = 5.4). The mean is right, but it misses that a quarter of the queries don't move and 29% get worse.
(Setup: BM25 with k1=1.5, b=0.75 and Lucene IDF, over each page's title and abstract. word lowercases and splits on word characters. word_en also drops NLTK's English stopword list and applies NLTK's Porter stemmer.)
What the percentiles show
The first thing I've started reading is p25 and p50 along with mean. It comes out of the per-query scores I already have with a 1 line change (np.percentile(scores, [25, 50])). Toggle the statistic (the chart also shows recall@10 and recall@100the share of a query's relevant pages that show up in the top 10, or the top 100):
Observations:
- In this run, on every metric, p25 or p50 moves further than the mean. On NDCG@10, p50 moves +6.4 and p25 +6.9 while the mean moves +3.9.
- p25 on NDCG@10 goes from 0.00 to 6.94 because the stemmer pushes the zero-rate from 25.5% to 19.0%, so read the two together.
Which queries moved
p50/p25/mean give you a quick read. But if you wanna drill down and properly compare two methods, you should look into exact queries that are improved/degraded.
That's why I like to bucket the exact deltas per query for my metrics and use that to find queries to focus on:
Observations:
- 47% of queries got better, 29% got worse and the rest didn't move (186 wins, 115 losses, 99 ties). That's why the mean goes up, but only by 3.9.
- Most of the ties score 0 both before and after (59 of the 99).
- The stemmer and stopword list rescued 43 queries from 0, like
Which airports are located in California, USA?(0 to 89.1), and sent 17 to 0, likemason ohio. That moves the zero-rate from 25.5% to 19.0%.
I like to ask an LLM to build a UI for me to explore the dataset. It just needs basic sorting by metrics, filtering, and search (yeah. use search to improve search :D). Below is a cut-down version with three queries from this run, click one to see what each #[analyzer](the step that turns text into index terms, here word or word_en) sent to the index and what came back:
Some of the queries it broke:
imdb batman returnswent from 64.3 to 22.9 on NDCG@10. The stemmer turnsreturnsintoreturn, so three Return of the Joker pages now take the top 3. Batman Returns falls from rank 1 to 4, and Batman: The Dark Knight Returns (film), the page the judges rated most relevant, falls from rank 2 to 8.mason ohiowent from 38.5 to 0. The query doesn't change at all, since neither word gets stemmed. The index does: Porter turns "Masonic" intomason, so Masonic temples across Ohio crowd the top 10 and push out the town.organic food advantages disadvantageswent from 39.4 to 6.9.organicbecomes organPorter gives organic, organism, organization and organ the same stem, and the top results turn into pages like Revealed comparative advantage.
So what should we actually track?
The goal is to see the distribution. A few numbers give you most of it, and they come from the per-query scores you already have. (No labels yet? Hand-label 50 to 100 real queries, or use an LLM judge.)
- p25 and the zero-rate show how the worst queries do, and that's often where you might wanna spend effort. A query going from 0 to 40 means someone now finds an answer. A query going from 70 to 80 means a good result got a little better.
- p50 shows how the typical query does, and it often catches a difference the mean hides. In a recent experiment, two approaches landed within 0.5 on the mean but 3 points apart on the median. That helped me decide which one to go ahead with.
- The mean stays, so you can compare with papers and other runs. Split its change into gained and lost too. The stemmer's +3.9 is +6.3 won by the queries that got better and −2.5 lost by the ones that got worse.
- A significance test tells you whether a gain is real or noise. If the stemmer did nothing, each query that moved would go up or down like a coin flip. 186 up and 115 down is very unlikely by chance (p = 0.00005). StatQuest explains p-values well.
Percentiles work best when queries have many relevant docs, as on DBPedia. On sets with one relevant doc per query, like NQNatural Questions: real Google search questions, each with one Wikipedia answer page, about half the queries score 0 and the median barely moves, so lean on the zero-rate and better / same / worse.
This is the table I print after every run while iterating. For this post's change it looks like this:
| analyzer | NDCG@10 p25 | p50 | mean | zero-rate | better / same / worse | gained / lost |
| -------- | ----------- | ----- | ----- | --------- | --------------------- | ------------- |
| word | 0.00 | 21.13 | 27.06 | 25.5% | | |
| word_en | 6.94 | 27.50 | 30.94 | 19.0% | 46.5% / 24.8% / 28.8% | +6.33 / -2.46 |
Every now and then I ask agents to surface the worst queries and build a UI for me to explore them interactively. That's how this run turned up organic turning into organ and the Joker taking over Batman Returns.
Beyond search: any eval with graded per-item scores (an LLM judge's rubric, a similarity score, a per-query metric) hides its tail in the mean the same way.*Binary pass/fail evals hide less in the mean, but you still want the flip counts: how many items went from pass to fail and from fail to pass.
Mean alone can't capture a distribution. We realised this for latency decades ago, and search relevance deserves the same. So please look into your distribution and your queries.
Key Takeaways
- A good average can hide queries that are hurting real users. Plain BM25 averages 27.1 NDCG@10 on DBPedia, yet 25.5% of queries find nothing relevant in the top 10.
- Even standard advice isn't a free lunch. A stemmer and stopword list move mean NDCG@10 by +3.9, yet 29% of queries get worse.
imdb batman returnsdrops from 64.3 to 22.9 becausereturnsnow matches Return of the Joker. - Report p50, p25 and the zero-rate next to the mean, then look into the queries that moved. p25 and the zero-rate show how the worst queries do, which is often where the effort should go.