Part V · Build it

Lab 4: measure your retrieval

Score the retriever you built in Lab 3 on 25 labelled questions, with the ranking metrics of chapter 26. Add a keyword retriever, BM25, and fuse the two; compare chunk sizes; and put an error bar on every score, so you can see how much 25 questions can tell you. No model is called, so the lab costs nothing and gives the same numbers on every run.

What you will build

One file, labs/04-retrieval-eval/retrieval_eval.py, and a question set, questions.jsonl. It imports Lab 3's rag module for the chunks, the tokens and the TF-IDF index.

Before you start

• Do Lab 3 first: this lab scores its chunks and its TF-IDF index, and imports its module by adding labs/03-rag to sys.path.
• Python and NumPy as in Lab 0. No key, no backend setting, no network: nothing here calls a model. (The optional --ollama flag calls a local embedding model, which is also free.)
• Run the tests from the labs/ folder:

cd labs && python -m pytest 04-retrieval-eval
.............                                                            [100%]
13 passed in 0.08s

Step 1: labelled questions

For each question, write down which passages answer it. That is the whole labelling job, and it is the expensive part (chapter 26). Each line of questions.jsonl is one question:

{"id": "q13", "kind": "direct", "question": "Do children pay late charges?", "relevant": ["fines#late-returns", "membership#childrens-cards"]}
{"id": "q14", "kind": "paraphrase", "question": "How long can I keep a book?", "relevant": ["loans#loan-periods"]}

Three choices matter:

• Labels name sections, not chunks. A section keeps its id whatever the chunk size, so the same labels score a 20-word index and a 120-word one. A chunk counts as relevant when its section is.
• Two kinds of question. 13 are direct: they use the pages' own words, as people do after reading the site. 12 are paraphrases, written the way people actually ask: "My nan is hard of hearing. Is there anything to help her at the counter?" shares one word with the section that answers it. Scoring the kinds separately shows what a word-based retriever cannot do.
• Some questions have two right answers. q13 and q22 are each answered by two sections, so recall@3 can be ½.

A label that names a section which does not exist would quietly score as a miss forever. test_every_label_names_a_real_section checks every label against Lab 3's chunks, and fails the moment a heading is renamed.

Step 2: the metrics

Each metric takes a ranked list of section ids, best first, and the set of relevant ones (chapter 26):

def hit_at_k(ranked, relevant, k):
    return 1.0 if any(r in relevant for r in ranked[:k]) else 0.0

def recall_at_k(ranked, relevant, k):
    return sum(r in relevant for r in ranked[:k]) / len(relevant)

def reciprocal_rank(ranked, relevant):
    for rank, r in enumerate(ranked, start=1):
        if r in relevant:
            return 1.0 / rank
    return 0.0

def ndcg_at_k(ranked, relevant, k):
    dcg = sum(1 / math.log2(rank + 1) for rank, r in enumerate(ranked[:k], start=1) if r in relevant)
    ideal = sum(1 / math.log2(rank + 1) for rank in range(1, min(len(relevant), k) + 1))
    return dcg / ideal

The tests use chapter 26's worked examples as fixtures. For relevance 0 1 0 1 0 with three relevant passages, they expect recall@5 = 0.667, a reciprocal rank of 0.5 and nDCG@5 = 0.498; for first relevant ranks of 1, 3, none and 2, an MRR of 0.458. If you change a metric and a test fails, compare your code with the hand calculation in the chapter. Every relevant section counts with a gain of 1, because the labels have no grades. precision_at_k is in the file too, but not in the table: with one relevant section per question, precision@3 can never exceed ⅓, so it says little here.

Step 3: a keyword retriever, BM25

TF-IDF and BM25 both count shared words, but BM25 gives repeated words diminishing returns and corrects for passage length (chapter 25):

def scores(self, query):
    words = self.tokenize(query)
    out = []
    for counts, length in zip(self.counts, self.lengths):
        norm = self.k1 * (1 - self.b + self.b * length / self.avglen)
        out.append(sum(self.idf[w] * counts[w] * (self.k1 + 1) / (counts[w] + norm)
                       for w in words if counts[w]))
    return out

with idf = ln(1 + (N − n + 0.5) / (n + 0.5)), k₁ = 1.2 and b = 0.75. It uses Lab 3's tokens, so both retrievers see the same words. test_bm25_on_the_chapter_25_example feeds it the chapter's three documents with plain whitespace splitting and expects the chapter's scores for "fixed deposit": 0.894, 0.482 and 0.482.

Step 4: fuse the two

TF-IDF scores are cosines between 0 and 1; BM25 scores have no upper limit. Rather than rescale them, reciprocal rank fusion ignores the scores and adds 1 / (60 + rank) from each list (chapter 25):

def rrf(rankings, k=60):
    total = defaultdict(float)
    for ranking in rankings:
        for rank, item in enumerate(ranking, start=1):
            total[item] += 1 / (k + rank)
    return sorted(total.items(), key=lambda kv: -kv[1])

The tests check the chapter's worked numbers: a passage 1st by vectors and 3rd by keywords scores 1/61 + 1/63 = 0.03227, and loses to one 2nd and 1st, at 0.03252.

Step 5: from chunks to sections

Each retriever returns its top 10 chunks that share at least one word with the question. sections replaces each chunk by its section and keeps each section once, at its best rank. Without that, two pieces of the same long section would fill two of the top three places and could count twice. A question whose relevant section is not in the top 10 gets a reciprocal rank of 0, so the MRR here is MRR@10.

Step 6: how sure can you be?

Twenty-five questions are a small sample of the questions people will ask. If you had written 25 different ones, every score would come out a little different. The bootstrap estimates by how much, from the 25 you have. Draw 25 questions from your set at random, with replacement, so some appear twice and some not at all, and average their reciprocal ranks. Do that 2,000 times. Sort the 2,000 averages and drop the lowest 50 and the highest 50: what remains spans the middle 95%, a confidence interval for the MRR.

def bootstrap_ci(values, n=2000, seed=0, level=0.95):
    rng = random.Random(seed)
    m = len(values)
    means = sorted(mean([values[rng.randrange(m)] for _ in range(m)]) for _ in range(n))
    tail = (1 - level) / 2
    return means[int(n * tail)], means[int(n * (1 - tail)) - 1]

Comparing two retrievers on the same questions is sharper than comparing their two intervals, because questions differ in difficulty far more than retrievers do (chapter 26). paired_difference takes the difference in reciprocal rank question by question and bootstraps that list. If its interval contains 0, these questions cannot tell the two apart. The seed is fixed, so the intervals below come out the same on every run; change --seed and they move in the second decimal place.

Checkpoints

• Run the tests: 13 pass.
• Read questions.jsonl and find a direct and a paraphrase question about the same section.
• Compute hit@3, recall@3, the reciprocal rank and nDCG@5 for one question by hand.
• Match BM25's 0.894 for chapter 25's example by hand.
• Run the report and say which retriever and chunk size you would choose, and how sure you are.
• Find the questions every retriever misses with --per-question.
• Try at least two of the experiments.

Run it

From the repository root. This is the whole output of a real run:

$ python labs/04-retrieval-eval/retrieval_eval.py
25 questions (13 direct, 12 paraphrase), heading paths on, overlap 0. No model is called.

retriever  max words  chunks  hit@1  hit@3  recall@3    MRR  MRR 95% CI     nDCG@5
tfidf             20     137   0.64   0.80      0.78   0.73  [0.59, 0.87]     0.74
bm25              20     137   0.64   0.80      0.78   0.74  [0.59, 0.88]     0.77
rrf               20     137   0.64   0.76      0.74   0.72  [0.57, 0.87]     0.72
tfidf             40      79   0.68   0.80      0.78   0.77  [0.63, 0.90]     0.78
bm25              40      79   0.72   0.84      0.82   0.80  [0.66, 0.92]     0.81
rrf               40      79   0.72   0.80      0.78   0.80  [0.66, 0.92]     0.81
tfidf            120      59   0.68   0.80      0.76   0.77  [0.63, 0.90]     0.78
bm25             120      59   0.72   0.88      0.86   0.79  [0.66, 0.92]     0.80
rrf              120      59   0.68   0.80      0.76   0.77  [0.63, 0.90]     0.79

hit@3 by kind of question, max words 120
retriever    direct  paraphrase
tfidf          1.00        0.58
bm25           1.00        0.75
rrf            1.00        0.58

MRR difference from tfidf, same 25 questions, max words 120, 2000 resamples
  bm25 - tfidf: +0.027, 95% CI [-0.007, +0.077]
  rrf - tfidf: +0.004, 95% CI [-0.008, +0.020]

Reading the table

• The hit rate at 3 is 0.80 for Lab 3's own setup (TF-IDF, 120 words): 20 of 25 questions have a right section in the top 3, the passages rag.py sends by default. The five misses are what a model would see as the wrong passages.
• Direct questions are solved; paraphrases are not. Every retriever finds all 13 direct questions in its top 3, and only 7 to 9 of the 12 paraphrases. All three retrievers count shared words, so a question in other words defeats them. This is the gap a trained embedding model is for.
• BM25 looks best, and the evidence is thin. At 120 words its hit@3 is 0.88 against 0.80, a difference of two questions (q14 and q16, both third for BM25 and fourth for TF-IDF). The paired MRR difference is +0.027 with an interval from −0.007 to +0.077: it contains 0, so these 25 questions cannot show that BM25 is better.
• Fusion did not help. RRF scores at or below its better input. Both lists fail on the same paraphrases, and fusion cannot add a passage that neither list ranks highly. Chapter 25 found the same when both lists were weak. Fusion pays off when the two retrievers fail on different questions, such as keywords and a trained embedding model.
• Smaller chunks did not help either. At 20 words, TF-IDF's hit@1 falls from 0.68 to 0.64 and its MRR from 0.77 to 0.73. A 20-word window loses words that the rest of its section supplied.
• Every MRR interval is 0.26 to 0.30 wide. TF-IDF's MRR of 0.77 could plausibly be anything from 0.63 to 0.90 on another 25 questions of the same kind. Differences of 0.03 between rows are well inside that.

--per-question adds the rank of each question's first relevant section. The paraphrases, from the same run:

$ python labs/04-retrieval-eval/retrieval_eval.py --per-question
...
rank of the first relevant section (- = not in the top 10), max words 120
id   kind              tfidf       bm25        rrf  question
...
q14  paraphrase            4          3          4  How long can I keep a book?
q15  paraphrase            1          1          1  Is there a penalty if I bring things back late?
q16  paraphrase            4          3          4  I was in hospital and my books are overdue. Do I still have to pay?
q17  paraphrase            3          2          2  Can someone drop books round to my house? I hardly go out these days.
q18  paraphrase            6          -         10  My granddaughter is seven. Can she get her own card?
q19  paraphrase            9          9          9  I'm only in town for the summer. Can I still borrow?
q20  paraphrase            1          1          1  Do you keep a record of everything I've read?
q21  paraphrase            1          1          1  What time do you shut on Christmas Eve?
q22  paraphrase            1          1          1  I want to get rid of some old paperbacks. Will you take them?
q23  paraphrase            1          1          1  My nan is hard of hearing. Is there anything to help her at the counter?
q24  paraphrase            1          1          1  I left my umbrella behind on Tuesday. Have you got it?
q25  paraphrase            4          4          4  Where can four of us sit and work together quietly?

q14 is Lab 3's miss, fourth for TF-IDF exactly as -k 5 showed. q18 asks about a seven-year-old; the answer says "Children under 16", and no retriever links "granddaughter" or "seven" to "children". q19 says "only in town for the summer", while the answer says "staying … for less than three months", and "summer" pulls in the Summer Reading Challenge instead, first for TF-IDF and second for BM25. These are the questions to read first when you try a new retriever.

What it costs

Nothing. The report makes no model call: it counts words, multiplies small matrices and resamples lists, and finishes in well under a second on a laptop. test_the_report_runs_without_calling_a_model replaces the adapter's chat with a function that fails, and the report still runs. With --ollama, the embeddings come from a model on your own computer, which is also free.

Experiments

1. Headings out. Run with --no-headings. In the run here, at 120 words, TF-IDF's hit@1 fell from 0.68 to 0.60 and its MRR from 0.77 to 0.73, while its hit@3 rose from 0.80 to 0.84 and its paraphrase hit@3 from 0.58 to 0.75. Headings helped questions that use the pages' words (q10, "Local Studies room", went from 8th to 3rd with them) and hurt paraphrases that share a word with another page's heading: q14 went from 2nd without them to 4th with them, lured by "How long we keep your account". Which metric decides depends on how many passages you send.
2. Overlap. Run --sizes 20 40 --overlap 10. In the run here, at 20 words TF-IDF's hit@3 rose from 0.80 to 0.84 and its MRR from 0.73 to 0.77, the same MRR as the 120-word index: overlap gave the small windows back some of the words their neighbours held. It cost 187 chunks instead of 137.
3. Pair the heading runs. The report pairs retrievers within one run. Write a short script that runs evaluate twice, with heading paths on and off, and passes the two lists of reciprocal ranks to paired_difference. Is the MRR change in experiment 1 bigger than chance?
4. More questions. Write 25 more questions, half of them paraphrases, or better, ask someone who has not read the pages to write them and label them yourself. Watch the intervals narrow. Four times as many questions should roughly halve their width.
5. Tune BM25. Set b=0 to turn off the length correction, or k1=0 to make one occurrence count as much as ten. Which questions change rank?
6. Change the fusion constant. Try rrf(..., k=1). With a small constant, first place in either list dominates. Does it help or hurt here?
7. Real embeddings. With Ollama running and nomic-embed-text pulled, add --ollama. Watch the paraphrase column: if a trained model closes the gap, then fusing it with BM25, the rrf-ollama row, is where fusion can finally help. These runs were not made for this page; your numbers are your own.

Going further

• Graded labels. Mark each relevant section 2 (answers it) or 1 (related), and use the grade as the gain in nDCG (chapter 26). It is the only metric here that can use them.
• Questions with no answer. Retrieval metrics are undefined when nothing is relevant, so score those questions on the answer instead: does the model say the passages do not answer it?
• The generation half. Good retrieval does not guarantee a good answer. Lab 8 builds a judge for faithfulness and citation support, and checks it against people.
• Guard against regressions. Run this report in your test suite and fail the build if hit@3 drops below a floor you choose. With 25 questions, set the floor a few questions below today's score, or ordinary noise will fail it.