Part IV · Applied concepts · Chapter 39

Quantisation

A trained model is billions of numbers, usually stored with 16 bits each. Quantisation stores them with fewer bits, 8 or 4, so the model takes less memory and often runs faster, at the price of small rounding errors. This chapter shows exactly how a weight is rounded, why one large weight can spoil its neighbours, and how much memory each choice saves.

Number formats

Floating-point formats (fp, bf) keep their own exponent per number. Integer formats store whole numbers and one shared scale that turns them back into real values; that shared scale is where both the savings and the errors come from.

How a weight is quantised

Symmetric: pick a scale so the largest weight maps to the largest integer, divide, round, store the integer. To use it, multiply back.

Worked: weights −0.62, 0.10, 0.31, 1.27. For 8 bits the integers run from −127 to 127, so the scale is 1.27 / 127 = 0.01. Dividing gives −62, 10, 31, 127, all whole already, so nothing is lost. For 4 bits the integers run from −7 to 7 and the scale is 1.27 / 7 ≈ 0.1814:

Rounding is never off by more than half a step, so the worst possible error is scale / 2 ≈ 0.091.

Asymmetric: when the weights are not centred on zero, map the smallest weight to 0 and the largest to the top integer, and store a zero point as well. For the same four weights at 4 bits (integers 0 to 15): scale = (1.27 − (−0.62)) / 15 = 0.126, zero point = round(0.62 / 0.126) = 5, and the errors fall to 0.010, 0.026, 0.058 and 0.010: a mean of 0.026 instead of 0.053, because no levels are wasted below −0.62.

In symbols, symmetric with b bits: q_max = 2^b−1 − 1, s = max|w| / q_max, q = round(w / s), ŵ = q · s, and |w − ŵ| ≤ s / 2.

Outliers and groups

The scale is set by the largest weight. Put one outlier of 6.0 among the four weights above and the 4-bit scale becomes 6 / 7 ≈ 0.857: now 0.10 and 0.31 both round to 0 and −0.62 becomes −0.857. One large value has erased its neighbours. Real models do have such outliers, which is why practical methods:

• use a separate scale per small group of weights (often 32 to 128), so an outlier spoils only its own group;
• keep the few outlier features in 16 bits (the idea behind LLM.int8(), 2022);
• choose rounding to minimise the error of each layer's output, not each weight (GPTQ, 2022), or protect the weights that matter most for the activations (AWQ, 2023).

Memory arithmetic

Memory for weights = parameters × bits per weight / 8 bytes. For a 7-billion-parameter model:

Running the model needs more than the weights: the key and value cache grows with context length and with the number of requests in flight. Fewer bits also mean less data to move from memory for every token, which is often what limits speed.

Checking quality

Quantisation is a trade, so measure it: compare the quantised model against the original on your own tasks, not only on a general score such as perplexity. Small models and low bit counts lose more. A model quantised during training (quantisation-aware training) usually holds up better than one rounded afterwards.

Check the label against the file. A model name with "Q" or "int8" in it is a claim, not a measurement. Read the data type of the weight tensors and count bytes: the number of weights times the bits per weight, divided by 8, should match the file size. Worked: a file called "Q8" holds 100 million weights and is 200 MB. That is 2 bytes, or 16 bits, per weight; a true 8-bit file would be close to 100 MB. Check again whenever the library or the download source changes.

Ask what precision serves a hosted model. When you call a model through a provider, the provider decides how it is stored unless you say otherwise. Some gateways let you restrict a request to providers that serve at least a given precision (say 8-bit floating point or integer, or higher) and to refuse "unknown". A direct call to the model's maker usually carries no such setting, so there the provider decides.