Part I · Maths · Chapter 7

Probability and Bayes

Probability puts a number on uncertainty. A language model's output is a probability for every possible next token, and every safety check in an AI system is judged by conditional probabilities. Bayes' rule ties them together.

Probability as a fraction

A fair die has six equally likely faces. The probability of rolling a 4 is 1 out of 6, written P(4) = 1/6 ≈ 0.167. The probability of an even number is 3/6 = 0.5. In general:

• Every probability is between 0 (never) and 1 (certain).
• The probabilities of all possible outcomes add up to exactly 1.
• The complement: P(not A) = 1 − P(A). P(not a 4) = 5/6.

"And" and "or"

If two events do not affect each other (they are independent), the chance of both is the product. Two dice both showing 6: 1/6 × 1/6 = 1/36 ≈ 0.028.

The chance of one or the other adds, minus the overlap you counted twice. Drawing a card that is a heart or a king: 13/52 + 4/52 − 1/52 = 16/52 ≈ 0.31. (The king of hearts is in both groups.)

Conditional probability

Often we know something already and want to update. "The chance of A, given that B happened" is written P(A | B). Worked with a table of 1,000 chat messages:

• P(tax) = 150 / 1,000 = 0.15.
• P(tax | mentions Spain): look only at the 200 messages that mention Spain. 60 of them ask about tax, so 60 / 200 = 0.30.
• P(mentions Spain | tax): look only at the 150 tax messages. 60 / 150 = 0.40. A different question, a different answer.

Bayes' rule

Confusing P(A | B) with P(B | A) is the most common mistake in reasoning about tests, detectors and classifiers. Bayes' rule converts one into the other:

Worked: a detector for bad messages

Suppose 1% of messages are prompt-injection attempts. A detector catches 95% of them (its recall), and wrongly flags 5% of normal messages (its false positive rate). A message is flagged. How likely is it to be a real attack?

Count it out with 10,000 messages:

1. Attacks: 1% of 10,000 = 100. The detector flags 95 of them.
2. Normal messages: 9,900. The detector wrongly flags 5% of them, which is 495.
3. Flagged in total: 95 + 495 = 590.
4. Real attacks among the flagged: 95 / 590 = 16.1%.

So "95% accurate" turns out to mean "wrong five times out of six when it fires". Nothing is broken; the attacks are simply rare, so even a small false-alarm rate on the huge pile of normal messages outnumbers them. Bayes gives the same number in one line: 0.95 × 0.01 / (0.95 × 0.01 + 0.05 × 0.99) = 0.0095 / 0.059 = 0.161.

Distributions

A probability distribution lists every possible outcome with its probability. A die's distribution gives 1/6 to each face. A language model, given "The capital of France is", produces a distribution over its whole vocabulary of tokens: perhaps " Paris" 0.92, " a" 0.02, " the" 0.01, and tiny amounts for tens of thousands of others. All of them add up to 1.

Sampling means drawing one outcome at random according to those probabilities. Over many draws, " Paris" comes up about 92% of the time. This is why a model can answer the same question differently twice; chapter 16 shows how temperature reshapes the distribution before the draw.

Expected value

The expected value is the probability-weighted average outcome. For a die: (1 + 2 + 3 + 4 + 5 + 6) / 6 = 3.5. For a bet that pays 100 with probability 0.1 and 0 otherwise: 0.1 × 100 + 0.9 × 0 = 10. You never roll a 3.5, but over many rolls the average approaches it. Training losses are expected values: the average loss over the distribution of training examples.

Continuous distributions

Heights, returns and measurement errors can take any value, not just a list. For those, a curve called a density says where values are likely, and probabilities are areas under the curve. The famous bell-shaped normal distribution is described by its mean and standard deviation (chapter 8). Neural networks start training with weights drawn from such a curve (chapter 13).

In AI

• A classifier's output, such as "spam: 0.83", is meant to be a conditional probability: P(spam | this text). When those numbers match reality (of all messages scored 0.83, about 83% really are spam), the classifier is calibrated.
• A language model is a machine for P(next token | all the tokens so far). Its whole training objective is to make those conditional probabilities match real text (chapter 19).
• Every guardrail has a recall and a false positive rate, and the base rate of what it guards against decides how often its alarms are right.

A personal-data scrubber is measured in exactly these two directions. A planted set holds messages with personal data placed at known positions, and measures recall: of every email, phone number and ID number planted, how many were caught? A clean set holds messages with no personal data, and counts false masks. A release check can then demand that every email and phone number is caught and fail on even one false mask.

An intent filter that labels each question before the main model runs, turning abusive or off-topic questions away with a fixed reply, is judged the same way. If its labels come from a model, the same reasoning applies: if off-topic questions are rare in real traffic, a small rate of good questions wrongly labelled off-topic can outnumber the true catches.