What metrics and test sets should I use to measure retrieval quality in production (NDCG@10, recall@k, MRR) and catch regressions?

Quick Answer: Use NDCG@10, recall@k, and MRR together, grounded in a stable, labeled test set that mirrors your real queries. Then wire those metrics into CI and offline evals so every retriever, embedding, or reranker change is checked for regressions before it hits production.

Frequently Asked Questions

How should I think about NDCG@10, recall@k, and MRR for production retrieval?

Short Answer: Use NDCG@10 as your primary “ranking quality” metric, recall@k to make sure your candidate set actually contains the right documents, and MRR to track “time to first relevant result” for chatbots and assistants.

Expanded Explanation:
These three metrics answer different questions about your retrieval stack:

• NDCG@10: “Are the top 10 results ordered in a way that matches human relevance judgments?” This is critical for search interfaces where users rarely go beyond the first page. It’s also the metric most tightly coupled to downstream LLM answer quality in RAG and agents, because your LLM usually only sees the top-k chunks.
• Recall@k: “Does my candidate set contain enough relevant docs at all?” This is retriever health, especially for domains where you care about completeness (legal, medical, audit/compliance). If recall is low, no reranker can fix you.
• MRR (Mean Reciprocal Rank): “How quickly do I surface the first useful result?” This is ideal for chatbots and agentic systems where the user (or LLM) primarily acts on the best single answer.

In practice, you’ll pick one primary metric (NDCG@10 for search, MRR for chatbots, recall@k for compliance-style discovery) and track the others as guardrails so you don’t optimize one dimension while quietly regressing another.

Key Takeaways:

• NDCG@10 measures ranking quality where it matters: the first screen of results.
• Recall@k and MRR protect you from “looks fine at the top 3 docs, but misses critical evidence” failure modes.

---

How do I actually evaluate these metrics and wire them into my workflow?

Short Answer: Build a labeled test set from real queries, compute NDCG@10, recall@k, and MRR offline for every change, and add a regression gate in CI so no retriever, embedding, or reranker update ships without passing your thresholds.

Expanded Explanation:
Evaluation has to be repeatable and boring if you want to catch regressions. You start by collecting real production queries (or representative synthetic queries), label which documents are relevant and how relevant, then compute your metrics on that set every time you:

• Swap embeddings or change vector DB configs
• Adjust hybrid retrieval weights (dense/sparse)
• Change the reranker model or k candidates
• Modify chunking, filtering, or metadata rules

With ZeroEntropy, teams typically start by measuring baseline NDCG@10 and recall@k with their current stack, then do an API swap to zerank-2 for reranking and zembed-1 for embeddings. We provide evaluation scripts so you can reproduce benchmark-style comparisons on your own corpus, not just ours.

Steps:

1. Collect and clean queries: Sample real user queries and system prompts (for RAG/agents). Normalize and de-duplicate.
2. Label relevance: For each query, label relevant documents and assign graded relevance (e.g., 0–3). Use experts for complex domains (legal, medical).
3. Compute metrics per candidate k: Run your current system to produce top-k results per query; compute NDCG@10, recall@k, and MRR. Then swap in new retrieval/rerank configs and recompute.
4. Add regression thresholds to CI: Set “no-merge” rules like “NDCG@10 must not drop >1% and recall@50 must not drop >2%,” and run evaluations on each change.
5. Monitor drift: Periodically refresh your test set with new queries so your metrics follow real behavior as your product and users change.

---

When should I optimize for NDCG@10 vs recall@k vs MRR?

Short Answer: Use NDCG@10 for most search UX and RAG systems, recall@k when completeness is non-negotiable (compliance, legal, clinical), and MRR when the user or LLM mostly acts on the first answer (chatbots, assistants).

Expanded Explanation:
Different applications have different failure costs:

• A support search that misses an edge-case article is annoying, but not catastrophic; here you care about NDCG@10 so the most common, high-impact answers are always at the top.
• A regulatory audit search that misses one critical document is a disaster; here recall@k is non-negotiable, and you accept slightly worse precision as long as all relevant docs are somewhere in the candidate set.
• A customer-facing chatbot that returns one bad answer is worse than returning a slightly less ideal but “safe” one; here MRR (how fast you surface the first good answer) is the most aligned metric.

Comparison Snapshot:

• Option A: NDCG@10
  • Optimizes: ordering of top 10 results
  • Ideal for: search interfaces, RAG pipelines, internal knowledge search
• Option B: recall@k
  • Optimizes: coverage of relevant docs in the candidate set
  • Ideal for: legal/clinical research, compliance, audit, analytics
• Best for:
  • Chatbots and assistants: track MRR primarily, with NDCG@10 as a secondary metric so the first answer is both fast and well-ordered relative to other candidates.

---

How many results should I rerank to optimize NDCG@10 and cost?

Short Answer: Rerank 50–75 candidates per query for most applications; beyond 100, NDCG@10 gains usually flatten while latency and cost increase linearly.

Expanded Explanation:
Reranking is where you convert “roughly right” candidate sets into “human-level” orders, but it’s also where you pay most of your compute. The question isn’t “should I rerank 500 docs?” but “what’s the smallest k that still maximizes NDCG@10 and doesn’t blow up p99 latency?”

From our internal evaluations and customer deployments:

• 50–75 candidates: Sweet spot for most production systems, especially where first-stage retrieval is decent (NDCG@50 > 0.7).
• >100 candidates: Diminishing returns; quality improvements plateau while costs and latency increase linearly.
• Noisy first-stage retrievers (poor candidate quality): You may need larger k, but you should simultaneously invest in better embeddings, hybrid retrieval, or index hygiene rather than just increasing rerank k.

With zerank-2, we see durable NDCG@10 lifts vs. Cohere rerank-3.5 and Jina rerank-m0 at k=50–75 on financial and technical corpora, while keeping p99 latency within tight production budgets.

What You Need:

• A way to generate candidate sets (dense, sparse, or hybrid) with configurable k.
• Metric tracking that lets you plot NDCG@10 and latency vs. k and stop increasing k when gains fall below ~2% per extra 25 candidates.

---

How do I build test sets that actually catch regressions and not just overfit benchmarks?

Short Answer: Build domain-specific, labeled test sets from your own traffic, stratified by intent and difficulty, and keep a “frozen” core set for regression detection while periodically adding new queries as your product evolves.

Expanded Explanation:
Public benchmarks are useful sanity checks, but they don’t match your domain jargon, document structure, or user behavior. If you want to catch meaningful regressions, your test sets must look like your real failure modes: ambiguous queries, long documents, multi-hop questions, and rare-but-critical edge cases.

A robust production test set usually has:

• Real queries: Taken from logs or closely-designed analogs based on product requirements (e.g., “find the exact clause about termination fees in vendor contracts”).
• Graded relevance: 0–3 or similar, so NDCG@10 can distinguish “perfect match” vs “loosely related.”
• Intent diversity: Navigational (“open policy XYZ”), informational (“what’s our PTO policy?”), and investigative (“find all contracts with MFN clauses”).
• Difficulty buckets: Easy, medium, hard (multi-hop, long-tail, heavy jargon).

We recommend a two-tier approach:

1. Frozen regression set (core): A small, highly-curated set (e.g., 200–500 queries) that never changes. You run this in CI to detect regressions with high sensitivity.
2. Rolling evaluation set (extended): A larger, evolving set that you refresh quarterly from new traffic and edge cases discovered via support, logs, and incident reviews.

Why It Matters:

• You avoid shipping a new retriever/reranker stack that looks good on generic benchmarks but silently breaks your highest-value workflows.
• You get repeatable, apples-to-apples comparisons across embeddings, hybrid configs, and rerankers like zerank-2 vs Cohere rerank-3.5 vs Jina rerank-m0.

---

Quick Recap

Measuring retrieval quality in production isn’t about chasing a single magic metric; it’s about matching metrics to behavior and wiring them into your development loop. Use NDCG@10 to evaluate how well your top results match human relevance, recall@k to ensure your candidate sets actually contain the right evidence, and MRR to track how quickly users or LLMs see the first good result. Build domain-specific test sets from real queries, keep a frozen core for regression detection, and evaluate changes to embeddings, hybrid retrieval settings, and rerankers offline before they hit production. In most stacks, reranking 50–75 candidates with a calibrated reranker like zerank-2 gives the best NDCG@10 vs. cost and latency tradeoff.

Next Step

Get Started