Part IV · Applied concepts · Chapter 44

Evaluating LLM output: judges and agreement

How do you know the answers are good, and how do you know the thing that grades them is any good? This chapter covers building a test set, checking with code what code can check, using a model as a judge for the rest, and testing that judge against people: agreement, Cohen's kappa, its precision and recall on planted defects, and its biases. It ends with thresholds on small samples, offline suites against live sampling, and what evaluations cost.

Why evaluate

An eval is a repeatable test of an AI system: a fixed set of cases, a way to run the system on each, and a scoring rule. You need one because three kinds of change alter a model application's behaviour without any error message:

• Prompt changes. You fix one bad answer by adding a sentence to the prompt. That sentence also changes how the model handles every other question. Some answers get better and some get worse; without a test set you see only the one you fixed.
• Model swaps. A new model version, a cheaper model for an easy step, or a provider updating the model behind a name. Each answers differently, and the question is whether your cases still pass.
• Changes around the model. New documents in the index, a different chunk size, a new tool, a changed output schema. The model is the same; what it sees is not.

A regression is a case that passed before a change and fails after it. An eval suite turns regressions into a number per change instead of complaints a week later. Run it before every change ships, compare with the last good run, and read the cases that flipped, not only the total.

Test sets

A test case is an input plus a description of what a good output must do. For a city library's help desk assistant, which answers questions from the library's policy documents, one case might be:

{
  "id": "renew-07",
  "question": "Can I renew a book that someone else has reserved?",
  "expect": "answer",
  "must_cite": ["loans-policy#renewals"],
  "notes": "Reserved items cannot be renewed; the answer must say so."
}

The expected behaviour is one of "answer", "refuse" (out of scope or unsafe) or "not in the documents". The ids it must cite let code check the citations. The notes are for whoever grades the free text, a person or a judge.

Where cases come from

Synthetic or real? Real questions tell you what the system will face. Synthetic ones fill the gaps. A good set starts with real questions, adds hand-written edge cases, and uses synthetic ones to cover topics the others miss, with a person reading every synthetic case before it goes in. Fifty careful cases beat five hundred unread ones.

Seeded defects: one defect per case

To test a grader, a code check or a judge, you need cases where you already know the right verdict. The cheapest way is to take answers you know are good and break each one on purpose: a seeded defect. Record which criterion each defect should fail.

One defect per case. If a copy has a wrong number and an unsupported claim and the judge fails it, you do not know which defect it saw; if it passes, you do not know which it missed. With one defect per case every result is attributable: a miss is a miss of that defect class, and you can report a catch rate per class. Copies like these can be built by code with no model call (change a number, swap a citation, insert a sentence), so they are cheap and the same every time.

Tune on one part, report on another

Split the cases in two, exactly as for any model you tune (chapter 11):

• The development set is the one you look at. Read its failures, change the prompt, the rubric or the retrieval, and rerun it as often as you like.
• The held-out set is run only to report a number, once the tuning is done. You do not read its individual failures while you are still tuning.

Every change you make because of a held-out failure fits the system a little to those exact cases. Do it a few times and the held-out score measures how well you have learnt the held-out set, not how the system does on new questions. That is leakage, and it always makes the number look better than the truth. When you have used a held-out set up, move it into the development set and write fresh held-out cases. Two thirds for development and one third held out is a common start. The same rule applies to the human labels you use to calibrate a judge (below): tune the rubric on one part of them and report agreement on the other.

Deterministic checks first

Before you ask a model to grade, check everything code can check. Code checks are free, fast and exact, and they give the same verdict every time. Run them on every case, first; a case that fails one needs no judge.

Many of these same checks run live as output guardrails (chapter 34). What is left for a judge is what needs reading: is every claim supported by the passages, does it answer the question, is it clear? Those become the rubric. Keeping the judge's job small makes it cheaper and makes its mistakes easier to see.

LLM-as-judge

An LLM judge is a model asked to grade another model's output against written criteria. It reads like a person, works at machine speed and costs a fraction of a person's time. It also makes mistakes, has biases, and can change when its own model is updated. Treat it as a measuring instrument that has to be calibrated.

The rubric

A rubric lists the criteria and says, for each one, what earns each verdict. Vague criteria ("is it good?") give vague, unstable scores; specific ones give scores that people and judges agree on. A rubric for the library assistant:

Write the rubric before you look at judge results, try it on a handful of answers with two people, and tighten any criterion they disagree on. Version it like a prompt (chapter 21): a score means nothing without the rubric version that produced it.

Pointwise or pairwise

• Pointwise: show one answer and the rubric, get a verdict per criterion. This suits a regression suite, because every case gets a verdict you can compare across runs.
• Pairwise: show two answers to the same question, say from prompt v6 and prompt v7, and ask which is better on a criterion, or whether they tie. Comparing is easier than scoring on an absolute scale, for judges and people alike, so pairwise suits choosing between two versions. It cannot tell you whether either answer is good enough, and it is open to position bias (below).

Binary verdicts with a reason, as JSON

Prefer a binary verdict, pass or fail, per criterion over a score from 1 to 10. On a ten-point scale two careful people rarely give the same number, and nobody can say what separates a 6 from a 7; on pass or fail with a written rule they usually agree. If you need more than two levels, use three (0, 1, 2), with a sentence describing each.

Ask for a short reason with each verdict, written before the verdict. The reason shows you why the judge failed an answer, which is how you find problems in the rubric, and asking for it first makes the verdict follow from the reasoning instead of the other way round. Ask for the result as structured output, JSON that matches a schema, so code can read it:

{
  "faithful":         {"reason": "The answer says 12 books; the loans policy says 8.", "verdict": "fail"},
  "answers_question": {"reason": "It deals with renewing a reserved book directly.",   "verdict": "pass"},
  "clear":            {"reason": "Short and plain; no jargon.",                         "verdict": "pass"}
}

Validate the reply. If it does not parse or a criterion is missing, ask once more; if it fails again, record the case as a judge error, never as a pass. A grader that counts its own failure as a pass is worse than no grader: it should fail closed.

Give the judge what it needs and nothing more: the question, the passages the answer was given, the answer and the rubric. A judge asked about faithfulness without the passages can only guess.

Calibrating a judge against people

A judge earns trust the way a new colleague does: by agreeing with careful people on cases where the answer is known. The procedure:

1. Take 50 to 200 answers, a mix of real answers and seeded defects, and shuffle them.
2. Have one or two people label them against the rubric, blind: they do not see which answers are seeded, which system wrote them, or what the judge said. Give each answer a sheet id and keep the key that maps ids back to their source in a separate file.
3. Run the judge on the same answers.
4. Measure agreement per criterion: the people against each other, and the judge against the people.

The agreement between two people is the ceiling. A judge cannot be expected to agree with a person more often than two people agree with each other, and if the people disagree a lot, the rubric is the problem, not the judge.

Agreement and Cohen's kappa

Two raters scoring the same answers will agree some of the time by luck alone, so raw agreement flatters. Cohen's kappa (from "A Coefficient of Agreement for Nominal Scales", 1960) subtracts the agreement you would expect by chance.

Worked: two people score the same 20 library answers on answers_question, using the three-level scale 0, 1, 2. In the table, row = person A's score, column = person B's.

1. Observed agreement. The diagonal counts the answers they scored the same: 3 + 4 + 8 = 15, so 15 / 20 = 0.75.
2. Chance agreement. A gives 0 to 4/20 of answers and B to 4/20; if they scored at random with those shares, they would both give 0 to (4/20)(4/20) = 0.04 of answers. Likewise (7/20)(6/20) = 0.105 for 1 and (9/20)(10/20) = 0.225 for 2. Total: 0.37.
3. Kappa. Of the agreement that was possible beyond chance (1 − 0.37 = 0.63), they achieved 0.75 − 0.37 = 0.38. So κ = 0.38 / 0.63 ≈ 0.603.

In symbols, with n answers, and a_s and b_s the number of answers each rater gave score s:

p_o = (answers on the diagonal) / n

p_e = Σ_s (a_s/n)(b_s/n)

κ = (p_o − p_e) / (1 − p_e)

κ = 1 is perfect agreement, 0 is no better than chance, and below 0 is worse than chance. A rough reading that is widely quoted (Landis and Koch, 1977) calls 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and above 0.80 almost perfect. Those bands are a convention, not a test: decide the bar your judge must clear before you measure it. When both raters gave every answer the same score, p_e = 1 and kappa is not defined.

The judge as a defect detector: precision and recall

Kappa says how well the judge and a person agree overall. For a pass/fail criterion you also want to know which way the judge errs. Treat "fail" as the thing being detected, take the person's label as the truth, and count four cells, as for any check that flags things (chapter 34).

Worked: a person and the judge both grade the same 100 answers on faithful. The person fails 30 of them.

• Recall on defects: of the 30 answers the person failed, the judge failed 24, so 24 / 30 = 0.80. One defect in five gets through.
• Precision: of the 31 answers the judge failed, 24 really were bad, so 24 / 31 ≈ 0.774. The other 7 (about 23%) are false alarms: good answers marked down.
• Agreement and kappa: p_o = (24 + 63) / 100 = 0.87; p_e = (30/100)(31/100) + (70/100)(69/100) = 0.093 + 0.483 = 0.576; κ = (0.87 − 0.576) / (1 − 0.576) = 0.294 / 0.424 ≈ 0.693.

Which error matters more depends on the job. A judge that gates releases needs high recall, because a missed defect ships. A judge that reports a quality trend can live with lower recall if it is stable, because a trend compares the judge with itself. See precision and recall.

Seeded defects give you recall without labelling: every seeded copy is a known fail, so the share the judge fails is its recall on that defect class, its catch rate. A judge that catches 26 of 30 wrong numbers but 2 of 10 unsupported claims needs work on faithfulness, whatever its overall kappa. Seeded copies cannot measure precision, because they contain no good answers; that is why the calibration set mixes seeded copies with real answers.

Judge biases and how to test for them

A judge should grade what an answer says, not how it is presented. Three biases are well documented (Zheng and others, 2023, in further reading):

Worked: a judge compares 50 pairs, each in both orders. On 41 pairs it picks the same answer both times. On 9 it flips, and in all 9 it picks whichever answer came first. A flip rate of 9 / 50 = 18%, all in one direction, is position bias. The usual remedy is to keep the double judging: count a win only when both orders agree, and call the rest a tie. That doubles the cost of pairwise judging (below).

A bias probe is a test like any other: it passes when every verdict is unchanged. A probe that reorders the sources in an answer, or pads it, should leave every score where it was. Keep the probes in the suite and rerun them whenever the judge's model or prompt changes.

Thresholds and small samples

An eval ends in a decision: ship or not, keep the judge or not. Write the rule down before you run the eval, so the number cannot argue you into a new rule. For example: "the held-out pass rate on faithful may not drop by more than 2 points"; "every case tagged must-pass must pass"; "the judge needs κ of at least 0.6 against people, and a catch rate of at least 0.8 on every defect class".

Then remember how few cases you have. A catch rate of 26 out of 30 is 0.867, but the 95% Wilson interval runs from about 0.703 to 0.947. With three to five copies per defect class, a single class's rate is rougher still. Likewise, prompt v7 passing 46 of 50 cases (92%, interval 81.2% to 96.8%) and v6 passing 43 of 50 (86%, interval 73.8% to 93.0%) does not prove v7 is better: the intervals overlap widely. Chapter 36 works these intervals by hand and has a widget for them.

Two things help. Compare the versions on the same cases and count only the cases where they disagree (chapter 26 shows how). And read the flipped cases: three cases that went from pass to fail tell you more than the total, because each is a specific thing the change broke.

Offline suites and online sampling

Offline, the suite is a test like any other: the same cases and the same rubric version, compared with the last good run. Online, a background job judges a sample of answers after they have been sent, so the user never waits for it. Mask personal data before an answer reaches the judge, as for any other model call (chapter 34), and store the judge's verdicts with the trace of the answer they grade, so you can open the answer behind a bad score (chapter 45).

The two feed each other. An online failure, once masked and labelled, becomes a new offline case, so the same mistake cannot return unnoticed.

What evals cost

An eval suite pays for model calls every time it runs, and a judge call is often bigger than the call it grades, because it reads the question, the passages, the answer and the rubric. Worked, with the prices per million input and output tokens of Claude Haiku 5.5 ($0.10 and $0.50) and Claude Sonnet 5.5 ($2 and $10): a suite of 200 cases, each answered by Haiku with 1,500 input and 300 output tokens, and graded by a Sonnet judge reading 2,000 tokens and writing 200.

Here the judge costs twenty times as much as the answers. Pairwise judging in both orders, reading two answers each time (about 3,500 input tokens), would cost 2 × 200 × (3,500 × 2 + 200 × 10) / 10⁶ = $3.60. Current Claude models think before they answer, and that thinking is billed as output, so take real token counts from a few calls before you budget (chapter 45).

Ways to spend less:

• Run the code checks first and skip the judge for cases they have already failed.
• Record each model call of an eval run, keyed by a hash of its request, and replay the recording when the request has not changed. A rubric change then reruns only the judge, not the answers.
• Try a smaller judge model. A Haiku judge on the same 200 cases would cost $0.06, but calibrate it against people again before you trust it; never assume it agrees as well as the bigger one.
• Run a small smoke set on every change and the full suite nightly or before a release.

People's time is a cost too. Labelling takes a careful person a minute or more per answer per criterion. It is the one part you cannot automate, and the one that makes every other number mean something.