Part I · Maths · Chapter 8

Mean, variance and normalisation

A handful of summary numbers describe a pile of data: where its middle is and how spread out it is. Normalisation uses them to put numbers on a common scale, which neural networks need in order to train, and error bars use them to say how far to trust a measured rate.

Why it matters

Layer normalisation, inside every transformer, is the z-score from this chapter applied to each token's vector. Evaluation numbers such as "the grader passed 92% of answers" are worthless without knowing how much they would wobble on a different sample. And a pipeline that collects figures from several web sources can choose between them with a median, for a reason this chapter explains.

The middle: mean and median

Five banks quote savings rates of 4.1, 4.4, 4.0, 4.3 and 4.2 percent.

The mean (average) adds them and divides by the count: (4.1 + 4.4 + 4.0 + 4.3 + 4.2) / 5 = 21.0 / 5 = 4.2.
The median sorts them (4.0, 4.1, 4.2, 4.3, 4.4) and takes the middle one: 4.2.

Now a sixth page shows 9.5, perhaps a typo, perhaps a promotional rate, perhaps text planted to mislead. The mean jumps to 30.5 / 6 = 5.08. The median, now the average of the two middle values 4.2 and 4.3, barely moves to 4.25. The median is robust: one wild value cannot drag it far. The mean is not.

μ = 1n Σ_i=1ⁿ x_iμ (mu) is the usual symbol for a mean.

The spread: variance and standard deviation

Two lists can share a mean and look nothing alike: (4, 4, 4) and (1, 4, 7) both average 4. To measure spread, look at how far each value sits from the mean, square those distances so negatives do not cancel positives, and average them. That is the variance. Worked with the five bank rates (mean 4.2):

value	4.1	4.4	4.0	4.3	4.2
value − mean	−0.1	0.2	−0.2	0.1	0
squared	0.01	0.04	0.04	0.01	0

Sum of squares 0.10, divided by 5 gives a variance of 0.02. Its square root is the standard deviation: √0.02 = 0.141 percentage points, back in the original units.

σ² = 1n Σ_i (x_i − μ)² σ = √σ²σ (sigma) is the standard deviation. When estimating from a sample, statisticians often divide by n − 1 instead of n, which gives 0.025 and 0.158 here; for large n the difference vanishes.

Z-scores and normalisation

A z-score says how many standard deviations a value sits from the mean: z = (x − μ) / σ. For the rate 4.4: (4.4 − 4.2) / 0.141 = 1.41. Converting every value this way is called standardising or normalising: the new list always has mean 0 and standard deviation 1, whatever units it started in.

Why bother? Picture a model with two inputs, salary in rupees (around 1,000,000) and age in years (around 35). Gradient steps that suit the age weight are absurdly small for the salary weight and vice versa, which is the lopsided valley of chapter 6. Normalising both inputs puts them on the same scale and makes the valley round.

A simpler alternative is min-max scaling: (x − min) / (max − min) squeezes values into 0 to 1. The rates become 0.25, 1, 0, 0.75 and 0.5. It is easy, but one outlier squashes everything else into a corner.

Try it

Normalise the vector (2, 4, 6) to mean 0 and standard deviation 1.

Answer

Mean 4. Deviations −2, 0, 2; squares 4, 0, 4; variance 8/3 = 2.667; standard deviation 1.633. Result: (−1.225, 0, 1.225). This is exactly what layer normalisation does to each token's vector, before it rescales with learned numbers (chapter 13).

Spotting outliers

In bell-shaped data about 68% of values lie within 1 standard deviation of the mean, about 95% within 2, and 99.7% within 3. So a value with |z| above 3 is unusual. But be careful with small samples: the outlier 9.5 itself inflates the standard deviation, so its own z-score is only 2.23. In fact with 6 values no z-score can ever exceed √5 ≈ 2.24. Robust checks (the median, or fixed limits set in advance) are safer when the data are few.

How sure is a measured rate?

An evaluation finds 8 bad answers in a sample of 100: a rate of 8%. Take another 100 and you might find 5 or 12. The standard error estimates that wobble. For a proportion p measured on n items:

SE = √( p(1 − p) / n ) = √(0.08 × 0.92 / 100) = 0.027

A rough 95% confidence interval is the rate plus or minus 1.96 standard errors: 8% ± 5.3%, or about 2.7% to 13.3%. With 1,000 items and the same 8% it shrinks to 6.3% to 9.7%. The width falls with √n: four times the data, half the width. Chapter 36 uses a better interval (Wilson's) that behaves properly for small counts and rates near 0.

Medians and fixed limits in a pipeline

Take a pipeline that collects a market figure, such as a savings rate, from several websites. A median is the robust way to settle on one value. A careful rule accepts a figure from sites outside a trusted list only when at least three independent sites give one, sorts the values and takes the middle. With an even count it takes the lower of the two middle values rather than their average, so the accepted number is always one a real source published. The reason is simple: one page can carry planted text or a typing error, and the median ignores it.

A second layer uses fixed limits rather than z-scores. Each kind of figure gets a band (a savings rate must lie between 0 and 8 percent, for example) and a jump limit from the last accepted value. Fixed limits suit a pipeline that sees a few figures at a time, where a standard deviation would be too noisy to trust.

The same rule helps when you present scenarios. A projection tool can make its middle scenario the median of several firms' central estimates, and its cautious and strong scenarios the lowest and highest.

Self-check

When would you report a median instead of a mean?

When the data may contain extreme values, for example incomes, response times or figures scraped from many websites. The median describes the typical value without being dragged by a few extremes.

A list has mean 50 and standard deviation 10. What is the z-score of 35?

(35 − 50) / 10 = −1.5. It sits one and a half standard deviations below the mean.

Why does normalising inputs help gradient descent?

It puts every input on a similar scale, so the loss surface is closer to a round bowl and one learning rate suits every weight.

An eval finds 2 failures in 50 answers. Is "4% failure rate" a solid claim?

No. The standard error is √(0.04 × 0.96 / 50) = 0.028, so the rough interval runs from about 0% to 9.4%. Small samples give wide intervals.