Part II · ML foundations · Chapter 15

Project: train your own neural network

Write a neural network with nothing but NumPy: forward pass, cross-entropy, backpropagation, mini-batches and a validation split, all in about 200 lines you can read in one sitting. It trains in under a second on a laptop.

What you need

• Python 3.10 or newer and NumPy. The project's requirements.txt lists only numpy>=1.26.
• Any laptop. No GPU, no downloads: the data is generated by the script.
• The files: labs/numpy-neural-net/train.py and its requirements.txt, in this book's repository. Everything below runs from the repository root.

# from the repository root, in a fresh virtual environment
python3 -m venv .venv && . .venv/bin/activate
pip install -r labs/numpy-neural-net/requirements.txt
python labs/numpy-neural-net/train.py --gradcheck
python labs/numpy-neural-net/train.py --plot

A tour of train.py

The backward pass

This is the heart of the project. Compare each line with the hand calculation in chapter 12; the only difference is that every quantity is a matrix with one row per example.

def gradients(self, x, y):
    n = len(x)
    h, p = self.forward(x)
    d_out = ((p - y) / n)[:, None]   # sigmoid + cross-entropy: p - y, averaged
    g_W2 = h.T @ d_out                 # delta times the input each weight multiplied
    g_b2 = d_out.sum(axis=0)
    d_h = d_out @ self.W2.T            # back through the output weights
    d_z1 = d_h * (1.0 - h**2)          # back through tanh (slope 1 - tanh squared)
    g_W1 = x.T @ d_z1
    g_b1 = d_z1.sum(axis=0)
    return self.loss(p, y), [g_W1, g_b1, g_W2, g_b2]

What you should see

The gradient check prints backprop and numerical slopes side by side, then the worst relative gap. Anything below about 10⁻⁵ means the backward pass is right; the run here gave 1.8 × 10⁻¹⁰.

$ python labs/numpy-neural-net/train.py
data: 320 training and 80 validation points; network 2 -> 16 -> 1 (65 parameters)
epoch    1  train loss 0.6660  val loss 0.6666  train acc 0.613  val acc 0.600
epoch   50  train loss 0.5395  val loss 0.5671  train acc 0.694  val acc 0.700
epoch  100  train loss 0.3840  val loss 0.4766  train acc 0.850  val acc 0.762
epoch  150  train loss 0.1540  val loss 0.1671  train acc 0.975  val acc 0.975
epoch  200  train loss 0.0845  val loss 0.0996  train acc 0.994  val acc 0.988
epoch  300  train loss 0.0306  val loss 0.0396  train acc 1.000  val acc 1.000
lowest validation loss 0.0396 at epoch 300

(Some lines are left out.) Notice the plateau for the first 75 or so epochs, where the loss barely moves, then a sudden drop as the hidden neurons find useful directions. Plateaus like this are normal (chapter 6). With --plot the script draws the decision regions as text, with o and x for the two arms and . and : for the regions the network assigns to them; you can see the boundary winding between the arms.

Experiments

1. Skip standardisation. Run python labs/numpy-neural-net/train.py --raw. In the run here, accuracy stayed near 60% after 300 epochs, against 100% with standardised inputs. The raw coordinates are small, so the tanh neurons start nearly linear and the gradients for the first layer are tiny. One line of preprocessing made the difference.
2. Width. Try --hidden 2, 4 and 64. Where does the spiral become learnable? Does 64 learn faster or slower per epoch?
3. Learning rate. Try --lr 0.05 and --lr 5. Find the largest rate that still trains smoothly.
4. Overfitting. Make the data noisier by editing noise=0.04 in main to 0.15 and train for 2,000 epochs with --hidden 64. Watch training accuracy rise while validation accuracy stalls. Then add --l2 0.001.
5. ReLU. Replace np.tanh with np.maximum(0, ...) in forward, and fix the slope in gradients to (h > 0). Run --gradcheck first; if you forget the slope fix, it will tell you.
6. A second hidden layer. Add W3 and b3, extend forward and gradients, and keep the gradient check passing. This is the exercise that proves you understand backpropagation.

Where this leads

A deep-learning library does exactly this, with automatic gradients, GPU kernels and better optimisers. In PyTorch the whole gradients method becomes one call, loss.backward(). Chapter 20 uses PyTorch for a tiny language model, and you will recognise every part of its training loop.