Part IV · Applied concepts · Chapter 36

Hallucination

A hallucination is a fluent, confident statement that is not supported by the facts the model was given or by the world: a number that was never in the data, a citation to a paper that does not exist, a policy clause that says something else. This chapter explains why models do it, how systems are built so it cannot reach a customer unnoticed, and how to measure how often it happens with honest error bars.

Kinds of hallucination

The first kind, unfaithfulness to sources you supplied, is the one you can check by machine, because you hold the sources. That is why grounded designs try to turn every claim into one that points at a source.

Why models make things up

A language model is trained to produce likely text. When it does not know, a likely-sounding answer is still likely text. The paper "Why Language Models Hallucinate" (2025) adds a sharper argument: most training signals and benchmarks score a right answer 1 and both a wrong answer and "I don't know" 0, so guessing is always the better strategy.

Worked: a model is 30% sure of an answer. Under right = 1, wrong = 0, abstain = 0, guessing scores 0.3 on average and abstaining scores 0, so it guesses. Now score a wrong answer −1. Guessing scores 0.3 × 1 + 0.7 × (−1) = −0.4, worse than abstaining.

In symbols: with a penalty k for a wrong answer, guessing beats abstaining when p − k(1 − p) > 0, that is when p > k / (1 + k). A penalty of 1 means "answer only when more than 50% sure"; a penalty of 3 means "more than 75%". How you score your own evaluations shapes what you select for.

Designing it out

• Ground the answer. Give the model the facts (retrieval, tool results) and tell it to use only those.
• Let code produce numbers. Calculations happen in tools; the model refers to results instead of typing figures (chapter 28). A workable shape: the answer is a list of prose segments and typed references, each naming a tool, a field path and a display format, and code fills in the value. Deterministic checks then block any digit typed in prose, a reference to a tool that was not called, and a path that does not resolve. One exception is worth allowing: a news headline quoted word for word from this answer's own tool results may stand in the prose, digits and all, while a headline changed by even one character is checked like any other prose.
• Make claims checkable. Require a citation to a section, or a verbatim quote, and check it exists in the source before the answer is shown. Refuse a quote that is not on the page it names (ignore spacing, but require every letter, digit and mark to match in order), read figures only from quotes that passed, block a cited section that does not exist, and accept a web citation only if its URL is one a lookup tool returned in the same run.
• Allow "I don't know". Give the model a legitimate way out, and do not punish it in your evaluations.
• Check faithfulness separately. A second pass, by code or by a model with a rubric, asks whether each claim is supported. A useful first criterion reads: "Is every claim faithful to the tool data and the retrieved text shown with it?", scored 0 to 2 and gated. In a multi-step system, judge the steps where the fault starts: whether each worker's answer follows from its own tool results, and whether the merged answer kept every figure and claim and added none of its own (chapter 44).

Lowering the sampling temperature does not cure hallucination: it makes the model pick its most likely continuation more often, and the most likely continuation can be wrong.

Measuring the rate, with error bars

Check a sample of answers and count the ones with a hallucination: k out of n. The observed rate is k/n, but with small samples the true rate could be far from it. A confidence interval says how far.

Worked: 3 hallucinations in 50 answers is 6%. The textbook (Wald) interval is 0.06 ± 1.96√(0.06 × 0.94 / 50) = 0.06 ± 0.066, which runs from −0.6% to 12.6%. A negative rate is impossible: the formula breaks down when k is small. The Wilson interval fixes this; for 3 in 50 it is about 2.1% to 16.2%. So "6%" really means "somewhere between 2 and 16%".

In symbols, with p = k/n and z = 1.96 for 95%:

Two shortcuts. Rule of three: if you saw no hallucinations in n answers, the true rate is below about 3/n with 95% confidence; zero in 50 still allows up to 6%. Sample size: to pin a rate near 5% to within ±2 points, you need about n = 1.96² × 0.05 × 0.95 / 0.02² ≈ 457 answers.

Seeded errors. To test a checker or judge, write answers with defects planted on purpose, so you know the right verdict in advance. Worked: a judge catches 27 of 30 planted defects, 90%. The Wilson interval for 27 in 30 is about 74% to 97%, so a sample this small cannot tell a 75% judge from a 95% one. Read pass rates from samples of this size with the intervals above in mind.