Part IV · Applied concepts · Chapter 26

Evaluating RAG

A RAG system can fail in two places: retrieval can miss the passage, or the model can misuse the passage it was given. Evaluating it means measuring the two halves separately, with labelled questions, a handful of ranking metrics you can compute by hand, and honest error bars.

What you need: labelled questions

For each question, write down which passages answer it. That is the whole labelling job, and it is the expensive part. Good sets include:

• questions phrased the way real users phrase them, not copied from the documents' headings;
• questions whose answer spans two passages, and questions with no answer in the corpus;
• near misses: a question about one fund where five other funds have similar sections.

Five ranking metrics by hand

Take one question with 3 relevant passages in the corpus. Retrieval returns 5 passages, and the 2nd and 4th are relevant; the third relevant passage was not retrieved at all. Write relevance as 0 1 0 1 0.

Averages over many questions give the headline numbers. The hit rate is the average hit@k. The mean reciprocal rank (MRR) is the average reciprocal rank, counting 0 when nothing relevant was retrieved: for four questions whose first relevant passage sits at ranks 1, 3, nowhere and 2, MRR = (1 + 1/3 + 0 + 1/2) / 4 = 0.458.

nDCG step by step

Discounted cumulative gain adds up the relevance of each result, divided by log₂(rank + 1), so a relevant passage at rank 1 counts in full and lower ranks count less:

• DCG@5 for 0 1 0 1 0 = 1 / log₂3 + 1 / log₂5 = 0.631 + 0.431 = 1.062.
• The ideal ranking puts all 3 relevant passages first: 1 / log₂2 + 1 / log₂3 + 1 / log₂4 = 1 + 0.631 + 0.5 = 2.131.
• nDCG@5 = 1.062 / 2.131 = 0.498.

nDCG is the only one of the five that uses grades of relevance ("answers it" versus "related") and the only one that rewards putting the best passage first among several relevant ones. Some definitions use 2^grade − 1 as the gain instead of the grade; the widget uses the grade.

Which metric when

• Hit@k when the model only needs one good passage and you pass it the top k. It matches how most RAG answers are built, and it is the easiest to explain.
• Recall@k when the answer needs several passages, such as a question about two policies.
• MRR when the position of the first good passage matters, because you pass few passages or the model trusts the first one most.
• nDCG when you have graded labels and care about the whole ordering, for example when tuning a reranker.
• Precision@k when irrelevant passages are costly: many tokens each, or likely to mislead.

How sure can you be?

Ten questions is a small sample. With 5 hits out of 10, the 95% Wilson interval for the true hit rate runs from 0.237 to 0.763; with 10 out of 10 it runs from 0.722 to 1. The intervals overlap, so by themselves they do not prove the second system is better (chapter 36 works these intervals by hand).

Comparing two systems on the same questions is stronger than comparing two separate scores. Count only the questions where they disagree: if the new system wins 5 and loses none, then, were the two systems equally good, each disagreement would be a coin toss, and 5 wins out of 5 happens with probability 2 × 0.5⁵ = 0.0625 in either direction. Suggestive, not conclusive. More questions, especially hard ones, are the only real cure.

A retrieval eval script

One small script can do the whole retrieval measurement. Keep the labelled set as a plain file with an id, the question and the expected (document, section) pairs. For each question the script runs your retrieval function on its own, with no answer-writing model, takes the top five and marks a HIT at 3 and at 5 when an expected pair appears among them (comparing sections by their last heading is enough). It prints each question's top results with HIT or MISS, the totals, the p50 and p95 latency (nearest-rank percentiles) and how many embedding and rerank calls it made. Because no answer model is involved, a change in the score can only come from retrieval.

Two habits keep the run honest. Index the evaluation corpus into a store of its own, so a run can never change real data. And rebuild the index with the same corpus folder and settings the questions were written for. The hit@3 scores in chapter 24 (5 out of 10, then 10 out of 10) came from a run of this kind.

Evaluating the generation half

Good retrieval does not guarantee a good answer. For the answer, measure:

• Faithfulness (groundedness): is every claim supported by the passages given? An answer can be true and still unfaithful, if it came from the model's memory.
• Citation correctness: does each cited source exist, and does it say what the answer claims? The first half can be checked by code; the second needs a person or a judge model.
• Answer relevance: does it answer the question asked?
• Honest gaps: on questions with no answer in the corpus, does it say so instead of inventing one?

These are usually scored by a judge model against a rubric, with its agreement with people checked on a sample (chapter 44). Citation checks are code: refuse an answer whose cited source was never retrieved for it, or whose cited section does not exist.