Part III · How LLMs work · Chapter 19

How LLMs are trained

A chat model is made in stages. Pretraining teaches a transformer to predict the next token on a vast amount of text. Supervised fine-tuning teaches it the shape of a helpful reply. Preference training (RLHF, DPO and their relatives) teaches it which of two replies people prefer. Evaluation runs throughout. This chapter works through each stage with numbers, including the compute bill.

The objective: predict the next token

Pretraining uses one loss everywhere: at every position in a text, the cross-entropy between the model's predicted distribution and the token that actually came next (chapter 9). The text supplies its own labels, so no human labelling is needed. With a causal mask (chapter 18), one pass over a 2,048-token sequence yields 2,047 predictions and 2,047 loss terms at once.

Worked: the model gives the true next tokens of a short text probabilities 0.5, 0.25, 0.8 and 0.1.

The perplexity is e^1.151 = 3.16: on average the model was as unsure as if it were choosing uniformly among about 3 tokens. A model that knew nothing, spreading probability evenly over a 50,000-token vocabulary, would score ln 50,000 = 10.8 and a perplexity of 50,000.

Why does predicting the next token produce something that seems to understand? Because doing it well on varied text needs it. To continue "The capital of Australia is" you need a fact; to continue a proof you need the logic; to continue a dialogue you need to model the speakers. The loss rewards whatever internal machinery helps, and nothing else. It also rewards confident continuations whether or not they are true, which is one root of hallucination (chapter 36).

Data

Pretraining data is mostly public web text, plus books, code, scientific papers and other curated sources, measured in trillions of tokens. Raw web text is noisy, so most of the work is in preparation:

• Filtering for quality, with rules and with small classifiers trained to recognise good text.
• Deduplication, exact and near-exact. Repeated text wastes compute and makes models memorise.
• Removing unwanted content, such as personal data and toxic material.
• Mixing: deciding what share of training comes from code, maths, other languages and so on. The mix shapes what the model is good at.
• Holding out evaluation sets, and checking that benchmark questions did not leak into training data (contamination), which would inflate scores.

The data also sets the knowledge cutoff: the model knows nothing after its newest training text. That is why an application that needs current facts, such as today's market rates, must supply them in the prompt (chapter 24).

Compute: C ≈ 6ND

Training compute is well approximated by

Where does 6 come from? In the forward pass each parameter takes part in one multiply and one add per token: 2 operations. The backward pass needs about twice as many, because it computes gradients both for the weights and for the inputs of each layer: 4 more. 2 + 4 = 6 per parameter per token. (Attention's square term adds a little on top, small unless sequences are very long.)

Worked: a 7-billion-parameter model on 2 trillion tokens.

1. C = 6 × 7 × 10⁹ × 2 × 10¹² = 8.4 × 10²² operations.
2. Suppose each accelerator chip peaks at 3 × 10¹⁴ operations per second and you keep it 40% busy (real training rarely gets close to peak): 1.2 × 10¹⁴ useful operations per second.
3. 8.4 × 10²² / 1.2 × 10¹⁴ = 7 × 10⁸ chip-seconds ≈ 194,000 chip-hours.
4. On 1,024 chips: 7 × 10⁸ / 1,024 ≈ 684,000 seconds ≈ 7.9 days, if everything scales perfectly, which it never quite does.

How to split a budget between N and D

For a fixed compute budget, a bigger model sees fewer tokens. "Training Compute-Optimal Large Language Models" (2022) found that for the lowest loss per unit of compute, parameters and tokens should grow together, at roughly 20 training tokens per parameter. For the budget above, D = 20N gives 6 × N × 20N = 8.4 × 10²², so N ≈ 26 billion parameters and D ≈ 530 billion tokens.

Yet the worked model uses 286 tokens per parameter, far past 20. That is deliberate and common: compute-optimal means cheapest to train, but a model is trained once and used millions of times. A smaller model trained for longer costs more to make and less to run.

The memory lines use common rules of thumb. Training with Adam in mixed precision keeps about 16 bytes per parameter: 16-bit weights and gradients (2 + 2), a 32-bit master copy of the weights (4) and Adam's two running averages (4 + 4), the accounting used in "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (2019). Activations come on top. Just running a model in 16-bit numbers needs 2 bytes per parameter, less with quantisation (chapter 39).

Supervised fine-tuning

A pretrained "base" model continues text; ask it a question and it may continue with three more questions, as if writing a quiz. Supervised fine-tuning (SFT) trains it further on examples of conversations: an instruction and a good reply, often tens of thousands of them, written or checked by people. The loss is the same next-token cross-entropy, with one change: only the reply's tokens count. If an example has 6 prompt tokens and 4 reply tokens, the loss is the mean over the 4 reply positions; the prompt is context, not something to learn to write.

SFT also teaches the chat format: special tokens that mark where the system prompt, the user's turn and the assistant's turn begin, and how a tool call is written. After SFT the model answers questions, but it imitates its examples, including their mistakes, and has no notion of which of two plausible replies is better.

Learning from preferences: RLHF

People find it much easier to compare two replies than to write a perfect one. "Training language models to follow instructions with human feedback" (2022) described the recipe usually called RLHF, reinforcement learning from human feedback:

1. Collect comparisons. For many prompts, sample two or more replies and ask people which is better.
2. Train a reward model: a copy of the language model with a single number as output, trained so that the preferred reply scores higher. The usual loss treats the probability that reply A beats reply B as σ(r_A − r_B), the sigmoid of the score gap. Worked: r_A = 1.2, r_B = 0.4 gives σ(0.8) = 0.690; if people chose A, the loss is −ln 0.690 = 0.371, and training widens the gap.
3. Optimise the model against the reward model with reinforcement learning, typically the algorithm from "Proximal Policy Optimization Algorithms" (2017). The model generates replies, the reward model scores them, and the model is nudged towards higher-scoring replies, with a penalty on drifting too far from the SFT model, measured by KL divergence (chapter 9).

The KL penalty matters. A reward model is only an imperfect stand-in for human judgement, and a model optimised hard against it learns to exploit its blind spots, for example by writing longer or more flattering replies that score well without being better. This "reward hacking" is one source of the familiar tendency of assistants to agree with the user. A related approach, "Constitutional AI: Harmlessness from AI Feedback" (2022), has a model produce the comparisons by following written principles, reducing the human labelling needed.

DPO: preferences without a reward model

"Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023) showed that the same goal can be reached with a simple classification-style loss on the preference pairs, with no separate reward model and no reinforcement-learning loop. For a chosen reply y_w and a rejected reply y_l:

Worked with β = 0.1. The chosen reply's log-probability rose from −14.0 under the reference model to −12.0 now: +2.0. The rejected reply's fell from −14.0 to −15.0: −1.0. The bracket is 2.0 − (−1.0) = 3.0; times β is 0.3; σ(0.3) = 0.574; the loss is −ln 0.574 = 0.554. The loss falls as the model raises the chosen reply relative to the reference and lowers the rejected one. Because it is simple and stable, DPO and its variants are widely used, including for fine-tuning small open models.

Rewards you can check

For maths and code, the reward can come from a checker rather than a person or a reward model: did the final answer match, did the tests pass? Reinforcement learning against such verifiable rewards, at large scale, is how recent reasoning models learned to produce long chains of working before an answer (chapter 16), as described for example in "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025).

Evaluation

Every stage needs measurement, and no single number is enough.

• Held-out loss and perplexity track pretraining, but say little about usefulness.
• Benchmarks: fixed question sets with checkable answers, such as the 57-subject exam questions of "Measuring Massive Multitask Language Understanding" (2020). Cheap and comparable, but they saturate, and contamination can inflate them.
• Human preference: people compare two models' replies side by side, often blind. Expensive, but closest to what users feel.
• Model as judge: a strong model grades replies against a rubric, studied in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023). Scalable, but judges have biases (towards longer replies, towards their own style) and must themselves be checked against human labels.
• Your own task evals: questions from your domain, with the answers or properties you need. For an application, these matter far more than public leaderboards.

Your own task evals in practice

If you build on trained models rather than train them, the "your own task evals" of the list above are the evaluation you actually run. A typical set-up has a question bank, built by a script with a --check mode that exits with an error when the built file is out of date. A live run sends every question through the system and has a judge score each answer on every criterion of a written rubric. The judge itself is checked the way this chapter recommends: known defects are planted into good answers, and agreement between labellers, or between a judge candidate and human labels, is measured with Cohen's kappa (chapter 44).

Two training facts show up as practical limits in such a system. Models know nothing after their cutoff, so live facts such as deposit rates should come from a knowledge base and tools (chapter 27). And because preference-trained models aim to please, every figure in an answer should trace back to a tool result before it is shown (chapter 34).