Part IV · Applied concepts · Chapter 40

Fine-tuning and distillation

Fine-tuning continues training a model that already works, on your own examples, so it adopts a behaviour, a format or a style. Distillation trains a small model to imitate a large one. Both change the weights, which prompts and retrieval never do. This chapter covers when that is worth it, the arithmetic of full fine-tuning against LoRA, and how distillation's soft targets carry more than a label.

When to fine-tune

Try the cheaper changes first, in this order: a better prompt, examples in the prompt, retrieval of the facts the model lacks, tools for anything exact. Fine-tune when those plateau and the gap is about behaviour: always answering in a strict format, a house style, a narrow classification done thousands of times a day by a smaller, cheaper model. Fine-tuning is a poor way to add facts, especially facts that change: the model may blend them with what it already knows, and you cannot cite or update them. Retrieval does that job better.

Before any training, remember how much behaviour the cheaper levers already control. Versioned prompts (record the prompt version with every answer), retrieval and tools that compute every exact figure shape what a system does without touching a weight. Choosing among candidate models by measuring each on labelled answers and planted defects (chapter 44) matches a model to a standard without training it. Reach for training only when those run out.

Full fine-tuning and its memory bill

Full fine-tuning updates every weight. With the Adam optimiser in mixed precision, a common estimate is 16 bytes per parameter during training: 2 for the 16-bit weights, 2 for their gradients, 4 for a 32-bit master copy of the weights, and 4 + 4 for Adam's two running averages (this accounting comes from "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models", 2019). For a 7-billion-parameter model that is 7 × 10⁹ × 16 bytes = 112 GB, before activations: more than one large GPU holds.

It also risks forgetting: training hard on a new task can damage the old one. In this book's NumPy project, full fine-tuning on a turned version of a task took the original task's accuracy from 0.997 to 0.703 (chapter 41).

LoRA: train a small update instead

LoRA (from "LoRA: Low-Rank Adaptation of Large Language Models", 2021) freezes a weight matrix W of size d × k and learns an update made of two thin matrices: A (d × r) and B (r × k), with a small rank r. The layer uses

B starts at zero, so training starts exactly from the original model. Only A and B are trained, and after training A B can be added into W once, so the finished model runs at the original speed. Or keep the adapter separate and swap adapters per task over one shared base model.

Worked: a 4096 × 4096 matrix has 16,777,216 weights. With r = 8, LoRA trains 8 × (4096 + 4096) = 65,536 numbers: 0.39% of the matrix. Adapting four such matrices in each of 32 layers trains 128 × 65,536 = 8,388,608 numbers, about 17 MB at 16 bits.

The bet behind LoRA is that the change a task needs has low rank: it can be described by a few directions, even though the matrix is large. Often it can; when it cannot, raise r. QLoRA ("QLoRA: Efficient Finetuning of Quantized LLMs", 2023) goes further by holding the frozen base model in 4 bits (chapter 39) while the adapters train in 16 bits.

Distillation: a small model learns from a large one

A trained classifier outputs probabilities, not just a winner. Those probabilities say how the model sees the classes: "mostly a cat, a bit like a dog, nothing like a car". Distillation ("Distilling the Knowledge in a Neural Network", 2015) trains a small student to match a large teacher's probabilities, softened with a temperature T, instead of only the hard labels.

Worked: the teacher's logits are 4, 2 and 1. At T = 1, softmax gives 0.8438, 0.1142 and 0.0420 (chapter 9). Divide the logits by T = 2 first, giving 2, 1 and 0.5, and softmax gives 0.6285, 0.2312 and 0.1402. The ranking is the same, but the second and third classes now carry visible signal for the student to learn from.

The student's loss is the cross-entropy between the teacher's softened probabilities and its own at the same T. Its gradient at the logits is (p_student − p_teacher) / T, so it is usually multiplied by T² to keep the step size similar whatever T you choose.

The practical win is often data: the teacher can label as many unlabelled examples as you can afford to run it on. With language models, distillation usually means training a small model on text a large model generated; check the large model's licence and terms before you do.