Part I · Maths · Chapter 5
Derivatives and gradients
A derivative is the slope of a curve at one point: how fast the output changes when you nudge the input. Training a model is nothing more than following derivatives downhill, so this is the most important chapter of Part I.
Every model in Part II and Part III learns by asking, for each of its parameters, "if I nudge this number up a little, does the error go up or down, and by how much?" That question is a derivative. The chain rule lets a computer answer it for billions of parameters at once; the answer is called the gradient.
The slope of a curve
A straight line has one slope everywhere (chapter 2). A curve like f(x) = x2 is steeper in some places than others. To find its slope at x = 3, take a second point a small gap h to the right and compute rise over run, then shrink the gap:
| gap h | f(3 + h) | rise = f(3 + h) − 9 | rise / h |
|---|---|---|---|
| 1 | 16 | 7 | 7 |
| 0.1 | 9.61 | 0.61 | 6.1 |
| 0.01 | 9.0601 | 0.0601 | 6.01 |
| 0.001 | 9.006001 | 0.006001 | 6.001 |
The answers close in on 6. That limit is the derivative of x2 at 3. The line through the point with that slope just touches the curve; it is called the tangent.
Rules that save the limit
Nobody computes limits by hand every time. A handful of rules cover everything in this book:
| Function | Derivative | Worked check |
|---|---|---|
| a constant, c | 0 | A flat line has no slope. |
| xn (power rule) | nxn−1 | x2 → 2x, which is 6 at x = 3, as the table found. |
| c × f(x) | c × f′(x) | 5x2 → 10x. |
| f(x) + g(x) | f′(x) + g′(x) | x3 − 3x → 3x2 − 3. |
| ex | ex | The slope equals the height (chapter 3). |
| ln x | 1 / x | At x = 0.5 the slope is 2: small probabilities make ln change fast. |
| sigmoid σ(x) = 1 / (1 + e−x) | σ(x)(1 − σ(x)) | At 0: 0.5 × 0.5 = 0.25, the largest it ever gets. |
Where the derivative is zero the curve is flat: at the bottom of a valley, the top of a hill, or a plateau. For x3 − 3x, 3x2 − 3 = 0 at x = 1 (a valley) and x = −1 (a hill).
Find the derivative of 4x3 + 2x − 7, and its value at x = 1.
Answer
12x2 + 2. At x = 1 it is 14.
The chain rule
A neural network is a chain of functions (chapter 2). The chain rule says how slopes pass along a chain: multiply the slope of each link.
Worked example. Let h(x) = (2x + 1)2. Split it: the inner link is u = 2x + 1 and the outer link is u2. At x = 1:
- Inner value: u = 3. Inner slope: du/dx = 2.
- Outer slope at u = 3: 2u = 6.
- Multiply: 6 × 2 = 12.
Check with a small nudge: h(1.001) = 3.0022 = 9.012004, and (9.012004 − 9) / 0.001 = 12.004. It agrees. Intuition: the inner link doubles any nudge to x, and the outer link multiplies a nudge to u by 6, so the whole chain multiplies it by 12.
Longer chains just add more factors. A network with 50 layers has a chain of 50 links, and the slope of the error with respect to a weight in layer 1 is a product of about 50 factors. If many factors are small (like the sigmoid's 0.25 at best), the product shrinks towards zero: 0.255 ≈ 0.001 and 0.2510 ≈ 0.000001. That is the vanishing-gradient problem of chapter 14.
Find the slope of e3x at x = 0.
Answer
Inner u = 3x has slope 3. Outer eu has slope eu, which at u = 0 is 1. Product: 3.
Several inputs: partial derivatives
When a function has several inputs, nudge one at a time and hold the others still. That gives a partial derivative, written with a curly ∂.
Worked example: f(x, y) = x2y + 3y at (1, 2).
- Treat y as a constant: ∂f/∂x = 2xy = 2 × 1 × 2 = 4.
- Treat x as a constant: ∂f/∂y = x2 + 3 = 1 + 3 = 4.
The gradient
Collect all the partial derivatives into a vector and you have the gradient, written ∇f ("del f"). For the example, ∇f(1, 2) = (4, 4).
Picture standing on a hillside in fog. You cannot see the valley, but you can feel which way the ground slopes under your feet. The gradient is that feeling, in as many dimensions as the function has inputs.
In AI: the slope of the error
Here is the whole of Part II in miniature. A model predicts w × x. One training example has x = 2 and the right answer y = 6. The error, or loss, is the squared difference L(w) = (wx − y)2.
- At w = 1 the prediction is 2, the miss is 2 − 6 = −4, and the loss is 16.
- Chain rule: dL/dw = 2(wx − y) × x = 2 × (−4) × 2 = −16.
- The slope is negative, so increasing w lowers the loss. The right answer is w = 3, where the slope is 0.
Real models have many parameters, so they need the whole gradient. Backpropagation (chapter 12) is the chain rule applied from the loss backwards through every layer, reusing each link's slope so the whole gradient costs about as much as one extra pass through the network. Libraries such as PyTorch do this automatically ("autograd").
A model computes a = σ(wx) and has loss (a − 1)2. With w = 0.5 and x = 2, find dL/dw. You need σ(1) = 0.7311.
Answer
Three links. Loss: 2(a − 1) = 2 × (−0.2689) = −0.5378. Sigmoid: a(1 − a) = 0.7311 × 0.2689 = 0.1966. Inside: d(wx)/dw = x = 2. Product: −0.5378 × 0.1966 × 2 = −0.2115. Negative, so raising w lowers the loss. A numerical nudge gives the same −0.2115.
Most people who build on hosted models never take a derivative in their own code. Every model they call was trained elsewhere with exactly this maths. Knowing how those models learned still helps, because it explains how they fail (chapter 36).
In chapter 15 you write backpropagation yourself in NumPy and check every gradient against a numerical slope.
Self-check
What does a derivative of zero tell you about a curve at that point?
The curve is flat there: it could be the bottom of a valley, the top of a hill or a flat stretch. Training looks for valleys of the loss.
Using the chain rule, what is the derivative of (3x − 2)2?
Outer 2u times inner 3: 2(3x − 2) × 3 = 6(3x − 2).
What is the gradient of f(x, y) = 3x + 5y?
(3, 5). Each partial derivative is a constant because the function is a flat plane.
Why do long chains of sigmoids make learning slow?
The chain rule multiplies one slope per link, and the sigmoid's slope is at most 0.25. Many such factors multiply to a number near zero, so early layers barely change.
Further reading
- Essence of calculus (3Blue1Brown playlist) video beginner
Visual introduction to derivatives as rates of change, building intuition before any formula work. - Visualizing the chain rule and product rule | Chapter 4, Essence of calculus video beginner
Makes the chain rule concrete with nudges flowing through nested functions, the core of backpropagation. - Multivariable calculus (Khan Academy) video intermediate
Course whose derivatives unit covers partial derivatives and gradients for functions of many inputs. - The gradient vector (Khan Academy article) article intermediate
Defines the gradient as the vector of partial derivatives with worked 2D and 3D examples.
Free, checked on 2026-10-21. All resources for this chapter.