Part IV · Applied concepts · Chapter 41

Projects: LoRA fine-tune and distillation

Two experiments, each a single NumPy script that runs in about a second: adapt a trained network to a new task with LoRA while its own weights stay frozen, and teach a 99-weight student from a 17,283-weight teacher. The results are reported as they came out, including where the textbook story does not hold on a problem this small.

What you need

• Python 3.10 or newer and NumPy. requirements.txt lists only numpy>=1.26.
• Any laptop. No GPU and no downloads: the data is generated by the scripts. Each run takes about a second on a laptop CPU.
• The files, in labs/finetune-distil/: ftcommon.py, lora_numpy.py, distil_numpy.py and requirements.txt. If you set up the workbench in Lab 0, use that environment; these scripts need only NumPy.

# from the repository root, in a fresh virtual environment
python3 -m venv .venv && . .venv/bin/activate
pip install -r labs/finetune-distil/requirements.txt
python labs/finetune-distil/lora_numpy.py
python labs/finetune-distil/distil_numpy.py

The shared pieces

Experiment 1: LoRA

lora_numpy.py pretrains a 2 → 64 → 64 → 3 network on task A (the spirals), then meets task B: the same spirals turned by 50 degrees, with only 30 labelled examples per class. It adapts in three ways. The Lora class keeps W1_A, W1_B, W2_A and W2_B; its grads method applies the chain rule, dA = scale · dW · B^T and dB = scale · A^T · dW, and the optimiser is handed only those four arrays.

$ python labs/finetune-distil/lora_numpy.py
pretrained on task A: accuracy A 0.997, B 0.625  (4,547 weights)
task B training set: 90 examples

method                              trained   acc B   acc A
full fine-tuning (every weight)       4,547   0.998   0.703
last layer only                         195   0.992   0.813
LoRA rank 2 on W1 and W2                388   0.995   0.752

base weights unchanged by LoRA training: True; merged model gives the same outputs: True
adapter removed again: task A accuracy 0.997 (the pretrained model is still there; keep one small adapter per task)

Reading it: all three methods learn task B. Full fine-tuning also damages task A most (0.997 to 0.703). LoRA trains 388 numbers, 8.5% of the network, and leaves the base untouched: remove the adapter and task A is back at 0.997, exactly. That reversibility, one base model with a small adapter per task, is LoRA's practical advantage.

Runs with other settings (seed 1 unless stated; accuracy on B / A):

An honest note: on a network this small, training only the last layer is as good as LoRA on task B and keeps more of task A, and with 5 examples per class it beats LoRA clearly. LoRA's savings grow with matrix size (chapter 40): for a 64 × 64 matrix, rank 2 is 1/16 of the weights; for 4096 × 4096 it is 1/1024. A toy cannot show that part.

Experiment 2: distillation

distil_numpy.py trains a teacher (2 → 128 → 128 → 3, 17,283 weights) on 1,200 labelled points of a tighter spiral (--twist 6). The student (2 → 16 → 3, 99 weights) gets only 5 labelled points per class. It is trained three ways, from the same starting weights for the same number of steps: (a) on the 15 labels; (b) on 3,000 random unlabelled points, each labelled with the teacher's top class; (c) on the same points with the teacher's full probabilities at temperature T, using the loss and T² scaling of chapter 40 (fit).

$ python labs/finetune-distil/distil_numpy.py
teacher: 17,283 weights, test accuracy 0.997
one transfer point: teacher probabilities at T = 4: 0.227, 0.022, 0.751

student: 2 -> 16 -> 3, 99 weights
student trained on                               examples  test acc
a. 15 hand-labelled points (hard labels)               15     0.697
b. teacher's top class on unlabelled points         3,000     0.916
c. teacher's probabilities at T = 4                 3,000     0.937

All three students share the same size, starting weights and number of steps.

Across three seeds and two temperatures (student test accuracy):

What the numbers say. The big gain is the teacher labelling unlabelled data: 53–70% becomes 92–96%. Soft targets add a little on top in five of the six runs (0.9 to 2.1 points) and lose 0.5 points in one. Temperature 4 against 1 makes no consistent difference here. With three classes and a confident teacher, there is little "dark knowledge" in the small probabilities; the original paper's large gains came from many classes. Try --labels 30: with 90 labels the hand-labelled student reaches 0.984 and beats both teacher-taught students (0.932 and 0.937), whose ceiling is set by how well a 99-weight network can copy the teacher.

Experiments to try

• python labs/finetune-distil/lora_numpy.py --rank 8 --shots 20: does more rank help? (Seed 1: LoRA reaches 0.997 on B with 1,552 numbers, but task A falls to 0.602 while the adapter is on.)
• In lora_numpy.py, adapt only W2: change ["W1", "W2"] in main.
• python labs/finetune-distil/distil_numpy.py --hidden 8: a student with 51 weights. (Here, seed 1, hard teacher labels beat soft ones: 0.890 against 0.851.)
• python labs/finetune-distil/distil_numpy.py --transfer 300: how much unlabelled data does the student need? (Seed 1: 0.878 hard, 0.906 soft.)

Scaling up to a real language model

The same ideas run on real models through Hugging Face's PEFT library (LoRA and QLoRA adapters) and TRL's SFT trainer, on small open models such as SmolLM2-135M or Qwen2.5-0.5B-Instruct (links in Further reading). Not run here: it needs a model download and, for speed, a GPU. Plan with the arithmetic from chapters 39 and 40: a 0.5-billion-parameter model needs about 1 GB for 16-bit weights, so LoRA on it fits a laptop with a recent GPU or Apple silicon; full fine-tuning at 16 bytes per parameter needs about 8 GB before activations. On CPU alone expect hours, not seconds.