Part I · Maths · Chapter 6

Gradient descent

Gradient descent finds the bottom of a valley by taking small steps downhill, using the slope from chapter 5. It is how almost every modern model, from a straight line to a language model, learns its parameters.

The update rule

You have a loss L(w) that measures how wrong a model is for a parameter value w. You want the w with the smallest loss. If the slope at your current w is negative, the loss falls to the right, so move right; if positive, move left. Move further when the slope is steep. In one line:

Worked by hand

Take L(w) = (w − 3)² + 1. Its lowest point is at w = 3, where the loss is 1. Its slope is 2(w − 3). Start at w = 0 with learning rate η = 0.1.

The steps shrink by themselves, because the slope shrinks as the ball nears the bottom. Each step closes 20% of the remaining distance to 3: the gap goes 3, 2.4, 1.92, 1.536 and so on, a factor of 0.8 each time.

The learning rate

For this bowl the gap to the bottom is multiplied by (1 − 2η) each step. That one factor explains every behaviour you will see in practice:

• η = 0.01: factor 0.98. Correct but slow; after 100 steps the gap is still 13% of where it started.
• η = 0.1: factor 0.8. Steady progress.
• η = 0.5: factor 0. One step lands exactly on 3. (This only happens on a perfect bowl.)
• η = 1.0: factor −1. The ball bounces between 0 and 6 forever.
• η = 1.1: factor −1.2. Each bounce is bigger than the last; the loss explodes. This is divergence, and in a real training run it shows up as the loss shooting to infinity or "NaN" (not a number).

Many parameters at once

With several parameters, use the gradient (chapter 5) and update every parameter by its own partial derivative, all at the same time:

Worked example: L(a, b) = (a − 1)² + 4(b − 2)², starting at (0, 0) with η = 0.1.

• Gradient: (2(a − 1), 8(b − 2)) = (−2, −16). Loss 17.
• Step: (0, 0) − 0.1 × (−2, −16) = (0.2, 1.6). Loss 1.28.
• Gradient now (−1.6, −3.2). Step to (0.36, 1.92). Loss 0.435.

Notice b moved much faster than a, because the loss is four times steeper in the b direction. When directions differ a lot in steepness, one learning rate is too big for one and too small for the other. Momentum and Adam (chapter 13) were invented to fix exactly this.

Valleys, saddles and plateaus

• Local minimum: a valley that is not the deepest. The bumpy curve in the widget has one. In very high dimensions true local minima turn out to be less of a problem than people feared, because a point has to curve upwards in every one of millions of directions to trap you.
• Saddle point: flat, but downhill in some directions and uphill in others, like the middle of a horse's saddle. The slope is zero, so plain gradient descent slows to a crawl there.
• Plateau: a wide flat region where the slope is tiny everywhere. Progress stalls until the ball reaches steeper ground.

Stochastic gradient descent

The real loss averages the error over every training example. Computing the exact gradient over millions of examples for every step would be far too slow. Instead, each step uses a small random batch, say 32 examples, and computes the gradient on just those. The gradient is noisy, but cheap, and on average points the right way. This is stochastic gradient descent (SGD). One pass through the whole training set is an epoch.

# the training loop of almost every model, in outline
for epoch in range(num_epochs):
    for batch in shuffled_batches(training_data):
        loss = average_loss(model, batch)      # forward pass
        grads = gradient_of(loss, model.params) # backward pass (chain rule)
        for p, g in zip(model.params, grads):
            p -= learning_rate * g              # the update rule

You will write this loop for real, with NumPy, in chapter 15.

The same loop outside training

The same "measure the error, then adjust" shape turns up outside training. A common example is a retry loop around a language model: when an output check blocks an answer, the system re-asks the question with the block reasons appended ("Revise the answer. Block reasons: …"), perhaps on a stronger model. Good systems allow a fixed number of retries. Like a learning rate, that cap is a deliberate limit: unbounded "try again" loops can bounce for ever or run up cost, just as a too-large step diverges.