Part V · Build it

Lab 8: an LLM judge you can trust

Write a pass-or-fail judge with a rubric, then test it the way you would test any classifier: against labels a person wrote, on answers with defects you planted yourself, and with each pair of answers shown in both orders. You will compute agreement, Cohen's kappa (by hand first), a confusion matrix, the judge's recall on each kind of defect, and how often its choice flips when the order does.

What you will build

One file, labs/08-judge/judge.py, of about 330 lines, a rule card and 40 labelled answers in labs/08-judge/data/, and the tests in labs/08-judge/test_judge.py. It:

• asks a judge to grade each answer against a four-part rubric and reply with JSON {"reason": …, "verdict": "PASS" | "FAIL"};
• parses every reply strictly and counts the ones that do not parse;
• compares the verdicts with a person's labels: agreement, Cohen's kappa, a confusion matrix and recall per defect type;
• runs a pairwise judge on each (good, bad) pair twice, once in each order, and counts the flips;
• prints what the run would cost on claude-haiku-5-5.

Before you start

Finish Lab 0, and read chapter 44 for the ideas behind each measurement. Any backend works. On the fake backend a keyword heuristic stands in for the model, so the offline run prints real numbers:

export LLM_BACKEND=fake

Step 1: a task, and labels you trust

The task is invented: a city library's help desk answers borrowers' questions from a rule card of ten short rules (labs/08-judge/data/library_rules.txt), such as "Books are lent for 21 days and DVDs for 7 days" and "Late fees for adults are 25p per day per book, capped at £6 per item."

The test set, labs/08-judge/data/items.json, holds 20 questions. Each has a good answer, labelled PASS, and a copy with exactly one planted defect, labelled FAIL with its defect type. Planting one defect per copy means you know what the judge should fail and why (chapter 44). There are four types, five of each:

The labels are the "truth" every number below is measured against, so they are worth checking. In a real project two people label the answers blind, without seeing which ones were planted, and you measure their agreement with each other before you measure the judge (chapter 44).

Step 2: the judge prompt

The rubric names four tests, each one a thing a person could check in seconds. A verdict is binary, because two careful people agree far more often on pass or fail with a written rule than on a score out of ten (chapter 44). The reason comes before the verdict, so the verdict follows from the reasoning rather than the other way round:

You grade answers written by a city library's help desk.
Read the library's rules, the borrower's question and the answer. The answer passes only
if all four tests hold:
1. Correct: every number, time and price in it matches the rules.
2. Supported: it claims nothing that the rules do not say.
3. On the question: it answers what was asked.
4. Courteous: no sarcasm, no blame, no talking down.
If any test fails, the answer fails. Write the reason first, then the verdict. Reply with
one JSON object and nothing else:
{"reason": "one short sentence naming the test that failed, if any", "verdict": "PASS" or "FAIL"}

The user message carries the rules, the question and the answer in tags: <rules>, <question>, <answer>. Each call sets effort="low" and leaves max_tokens at 2,048, because Claude's thinking counts toward max_tokens and a reply cut off mid-JSON does not parse.

Step 3: parse strictly, count what does not parse

A judge that breaks its output contract is a fact to measure, not a mess to tidy. If the parser dug a verdict out of prose with a regular expression, you would never learn how often the judge failed to follow instructions, and a guess would sometimes be wrong. So the parser accepts exactly one JSON object with exactly two keys:

def parse_verdict(text):
    try:
        data = json.loads(text)
    except ValueError:
        return None
    if (not isinstance(data, dict) or set(data) != {"verdict", "reason"}
            or data["verdict"] not in ("PASS", "FAIL") or not isinstance(data["reason"], str)):
        return None
    return data

It refuses prose before the JSON, a code fence around it, "pass" in lower case, a missing reason, an extra key and unescaped quotes; the tests try each one. In the run below one reply fails to parse:

q08-fail: {"reason": "this claim is not in the rules: Reserved books wait for a month on the "Collect here" shelf.", "verdict": "FAIL"}

The fake judge pastes its reason into a JSON template, much as a model writes JSON one token at a time, and its reason quoted an answer that holds double quotes. The verdict inside is even right. It still counts as unparseable, and the report says so.

Step 4: agreement and Cohen's kappa

Agreement is the share of answers where the judge's verdict matches the person's. It flatters, because two raters agree some of the time by luck. Cohen's kappa subtracts the agreement you would expect by chance (chapter 44). Work it by hand once on ten answers:

person: P P P P P P F F F F
judge:  P P P P P F F F P P

1. Observed agreement. The diagonal, 5 + 2 = 7 of 10, so p_o = 0.70.
2. Chance agreement. The person says PASS to 6/10 of answers and the judge to 7/10. Raters who labelled at random with those shares would both say PASS on 0.6 × 0.7 = 0.42 of answers, and both say FAIL on 0.4 × 0.3 = 0.12. So p_e = 0.42 + 0.12 = 0.54.
3. Kappa. Of the agreement possible beyond chance, 1 − 0.54 = 0.46, they reached 0.70 − 0.54 = 0.16. So κ = 0.16 / 0.46 ≈ 0.348.

Seventy per cent agreement sounds respectable; a kappa of 0.35 says most of it is what two coin-flippers with the same habits would manage. In code:

def cohen_kappa(a, b):
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum((ca[label] / n) * (cb[label] / n) for label in set(ca) | set(cb))
    return None if pe == 1 else (po - pe) / (1 - pe)

$ python labs/08-judge/judge.py kappa-example
person: P P P P P P F F F F
judge:  P P P P P F F F P P
kappa = 0.348

When both raters give one label to everything, p_e = 1 and kappa is not defined, so the function returns None rather than divide by zero. The tests also check that perfect agreement gives 1 and perfect disagreement gives −1.

Step 5: which way does the judge err?

Kappa is one number. The confusion matrix shows the direction of the mistakes: rows are the person's labels, columns the judge's verdicts, and unparseable replies get a column of their own (chapter 44). Treat FAIL as the thing being detected:

• Recall on defects: of the answers the person failed, the share the judge failed too. A miss here ships a bad answer.
• Precision of a FAIL: of the answers the judge failed, the share the person failed too. A miss here blocks or flags a good answer.

Planted defects add one more view: recall for each defect type. A judge with a fine overall kappa can still be blind to one kind of mistake, and that kind is the one that will reach your users.

Step 6: position bias in a pairwise judge

A pairwise judge sees two answers to the same question and picks the better one (chapter 44). Comparing is easier than grading on an absolute scale, but a judge may lean towards whichever answer it sees first. The test is cheap: show every (good, bad) pair twice, once in each order, map each reply back to the answer it chose, and count the pairs where the choice changed:

def position_flips(first_order, swapped):
    out = Counter()
    for w1, w2 in zip(first_order, swapped):
        if w1 is None or w2 is None:
            out["unparseable"] += 1
            continue
        chose_1 = "good" if w1 == "1" else "bad"   # shown (good, bad)
        chose_2 = "good" if w2 == "2" else "bad"   # shown (bad, good)
        out[f"{chose_1} both times" if chose_1 == chose_2 else "flipped"] += 1
    return dict(out)

A flip means the order decided, not the answers. The usual remedy is to keep the double judging and count a win only when both orders agree, calling the rest a tie (chapter 44). That doubles the cost of pairwise judging, which is why you measure the flip rate first.

The stand-in judge

On the fake backend, heuristic_judge answers both kinds of prompt with four keyword rules, one per test: a number, time or price in the answer that is not on the rule card fails correct; a sentence of four or more content words, fewer than 60% of which appear in the rules or the question, fails supported; no content word shared with the question fails on the question; and phrases such as "obviously" or "as I already said" fail courteous. For a pair, the answer with fewer problems wins, and on a tie it picks the first answer shown. That last rule is a position bias, built in where a weak judge would have one: when it cannot tell two answers apart.

Checklist

• I read the rule card and a few labelled pairs, and can name the four defect types.
• I can say why the verdict is binary and why the reason comes first.
• I saw a reply counted as unparseable and can say why the parser does not guess.
• I worked the ten-answer kappa by hand and matched the code's 0.348.
• I read the confusion matrix and recall per defect type from a full run.
• I ran the position-bias test and can explain every flip.
• The lab's tests pass.

Run it

The whole run on the fake backend, from the repository root:

$ LLM_BACKEND=fake python labs/08-judge/judge.py run
Judge run: 40 items on the fake backend (keyword heuristic: these numbers measure the plumbing, not a model)
unparseable replies: 1 of 40 ['q08-fail']
  q08-fail: {"reason": "this claim is not in the rules: Reserved books wait for a month on the "Collect here" shelf.", "verdict": "FAIL"}
agreement on the 39 parsed: 30/39 = 0.769; counting unparseable as wrong: 30/40 = 0.750
Cohen's kappa on the parsed: 0.535
confusion matrix (rows: person, columns: judge)
                PASS  FAIL  unparseable
  person PASS     18     2            0
  person FAIL      7    12            1
recall per defect type (planted defects the judge called FAIL)
  wrong_number  3 of 5
  unsupported   2 of 5
  off_question  3 of 5
  rude          4 of 5
disagreements: q04-fail (judge PASS), q05-fail (judge PASS), q06-fail (judge PASS), q07-fail (judge PASS), q08-pass (judge FAIL), q12-fail (judge PASS), q15-pass (judge FAIL), q15-fail (judge PASS), q19-fail (judge PASS)
position bias: 20 pairs, each judged in both orders: bad both times 1, flipped 7, good both times 12
calls: 80, 29131 input and 1224 output tokens (fake); on claude-haiku-5-5 about $0.0035, at most $0.0848 if every reply used max_tokens=2048

Reading the numbers

• Kappa. p_o = 30/39 = 0.769. The person says PASS to 20 of the 39 parsed answers and the judge to 25, so p_e = (20 × 25 + 19 × 14) / 39² = 766 / 1,521 = 0.504, and κ = (0.769 − 0.504) / (1 − 0.504) = 0.535. If you had decided in advance that a judge needs κ of at least 0.6 and a catch rate of at least 0.8 on every defect type, as chapter 44 suggests, this judge fails both bars. That is the harness doing its job.
• Which way it errs. Precision of a FAIL is 12 / 14 = 0.86: when it fails an answer it is usually right. Recall is 12 / 19 = 0.63 on the parsed answers, or 12 / 20 = 0.60 counting the unparseable one as a miss. It lets defects through far more often than it blocks good answers.
• The misses are realistic. q04-fail says the library opens at 9:00 on Saturdays; 9:00 is on the rule card, for weekdays, so a check that only asks "is this number in the rules?" passes it. q05-fail writes "five times" in words, which no digit pattern sees. q12-fail answers about black-and-white printing when asked about colour, and shares enough words with the question to look on topic.
• The false alarms are realistic too. q08-pass ("Reservations cost 50p each") and q15-pass (about renewing) answer the question in different words from it, and a word-overlap rule cannot see a paraphrase.
• Position. Twelve pairs were judged the same way in both orders. Seven flipped, and in every one the judge picked whichever answer came first: these are the pairs where it found no difference, so its tie rule decided. One pair, q15, went to the bad answer both times, because the judge wrongly failed the good one.

Read the reasons, not only the verdicts

Recall on rude is 4 of 5. Here are the replies for those five items, from the same heuristic:

q16-fail {"reason": "this claim is not in the rules: The library is closed on Sundays, as the sign on the door says.", "verdict": "FAIL"}
q17-fail {"reason": "the tone is not courteous ('as i already said')", "verdict": "FAIL"}
q18-fail {"reason": "the answer does not address the question", "verdict": "FAIL"}
q19-fail {"reason": "passes all four tests", "verdict": "PASS"}
q20-fail {"reason": "this claim is not in the rules: Please don't waste our time asking again.", "verdict": "FAIL"}

Only one of the four catches was for rudeness. The others were right for the wrong reason, and a small change elsewhere could turn them into misses. Recall per defect type counts verdicts; the reasons tell you whether the judge understood. Read a sample of them every time you change the judge.

The tests check the parser, kappa against the worked example, the confusion matrix, recall, the flip counting, the heuristic's broken JSON and the exact numbers of the run above. From labs/:

$ python -m pytest 08-judge
....................                                                     [100%]
20 passed in 0.03s

What it costs. A run makes 80 calls: 40 single judgements and 20 pairs in two orders. On the fake backend the token counts are estimates at about four characters per token, the system prompt included: 29,131 input and 1,224 output tokens. On claude-haiku-5-5 that is 29,131 × $0.10 / 1,000,000 + 1,224 × $0.50 / 1,000,000 = $0.0035. A real model writes longer reasons and its thinking counts as output, so the script also prints the ceiling, every reply using all 2,048 of its max_tokens: 29,131 × 0.10 / 1,000,000 + 80 × 2,048 × 0.50 / 1,000,000 = $0.0848, under nine US cents. Ollama and the fake backend cost nothing.

Experiments

1. Measure a real judge. Run LLM_BACKEND=anthropic python labs/08-judge/judge.py run, or use Ollama. Compare kappa, recall per defect type and the flip count with the heuristic's. Write down the model id with the numbers: a new model version means a new measurement.
2. Make the rubric vague. Replace the four tests with "Is this a good answer?" and run it on a real model. What happens to kappa, and to recall on off_question?
3. Label the set yourself. Shuffle the 40 answers, hide the label and defect columns, and label them against the rubric. Compute kappa between your labels and the file's with cohen_kappa. How good is the "truth"?
4. Fix the heuristic, carefully. Make the number rule check numbers in context, and read number words. Recall on wrong_number should rise. Then ask: you tuned the rules on the test set, so are the new numbers still honest? Split the questions in half, tune on one half and measure on the other.
5. Add a defect type. Plant five copies that leave out a condition, such as "unless someone else has reserved it". Which rubric test should catch it? Does the judge?
6. Use both orders. Count a pairwise win only when both orders agree. How many of the 20 pairs become ties?
7. Verdict first. On a real model, swap the order of the keys in the prompt so the verdict comes before the reason. Does agreement change?

Going further

• Put error bars on recall. Five items per defect type is a small sample: 3 of 5 could easily be 1 of 5 or 5 of 5 on another draw. A Wilson interval shows how wide the uncertainty is (chapter 44).
• A verbosity probe. Add one true but useless sentence to every good answer and rerun. A judge that starts failing them, or prefers them in pairs, is reacting to length, not content.
• Per-criterion verdicts. Ask for a verdict on each of the four tests rather than one overall verdict, and measure recall per test. It makes "right for the wrong reason" countable.
• A regression gate. Once the judge clears your bar, run it in your test suite on every prompt change, and fail the build when the pass rate drops (Lab 2 builds the bench).
• Judge in production. Lab 9 records every call as a span; a sampled judge score belongs on the same trace.