Part II · ML foundations · Chapter 12

Neural networks and backpropagation

A neural network is logistic regression stacked in layers, with a bend between each layer. This chapter builds one from a single neuron, runs it forward and backward by hand with real numbers, and then lets you train one in the browser.

Why it matters

Backpropagation is the algorithm that trains every model in Part III. If you can follow the twenty or so multiplications in the worked example below, you understand what a GPU does for weeks when a language model is trained; it just does it for billions of weights at once.

One neuron

A neuron takes inputs, computes a weighted sum plus a bias, and passes the result through an activation function:

a = f(w₁x₁ + w₂x₂ + b)

Worked: inputs (0.5, 1.0), weights (0.1, 0.4), bias 0. The sum is 0.05 + 0.4 = 0.45. With the sigmoid as f, the output is σ(0.45) = 0.611. A single sigmoid neuron is exactly logistic regression (chapter 10).

Activation functions

Name	Formula	f(−2), f(0), f(2)	Slope
sigmoid	1 / (1 + `e`^−z)	0.119, 0.5, 0.881	a(1 − a), at most 0.25
tanh	(`e`^z − `e`^−z) / (`e`^z + `e`^−z)	−0.964, 0, 0.964	1 − a², at most 1
ReLU	max(0, `z`)	0, 0, 2	0 for negative z, 1 for positive

Modern language models use smooth cousins of ReLU (named GELU and SwiGLU) in their MLP layers. Chapter 14 explains why ReLU-style activations replaced the sigmoid.

Why the bend is essential

Without activations, two layers compute W₂(W₁x), which equals (W₂W₁)x: one matrix, so one linear layer. Worked: if layer 1 doubles its input and layer 2 triples it, together they multiply by 6, a single straight line. However many linear layers you stack, you only ever get a straight decision boundary. The activation between layers is what lets a network bend its boundary.

Layers

Put several neurons side by side and they form a layer; in matrix form a whole layer is h = f(Wx + b) (chapter 4). Layers between the input and the output are hidden layers. A network of fully connected layers is called a multilayer perceptron, or MLP.

Counting parameters. A layer from n inputs to m neurons has m × n weights and m biases. A network 2 → 4 → 1 has (4 × 2 + 4) + (1 × 4 + 1) = 12 + 5 = 17 parameters. A transformer layer with width 4,096 has tens of millions.

A remarkable result, the universal approximation theorem, says that one hidden layer with enough neurons can approximate any reasonable function as closely as you like. It promises that a network exists, not that gradient descent will find it or that it will be small; in practice deeper networks learn the same functions with far fewer neurons.

A forward pass by hand

Our network: 2 inputs, 2 hidden sigmoid neurons, 1 sigmoid output. The task is to output 1 for the input x = (0.5, 1.0).

Parameter	Value
hidden neuron 1: weights, bias	(0.1, 0.4), 0
hidden neuron 2: weights, bias	(−0.3, 0.2), 0.1
output neuron: weights, bias	(0.5, −0.6), 0.2

Hidden sums: z₁ = 0.1×0.5 + 0.4×1.0 + 0 = 0.45; z₂ = −0.3×0.5 + 0.2×1.0 + 0.1 = 0.15.
Hidden outputs: h₁ = σ(0.45) = 0.6106; h₂ = σ(0.15) = 0.5374.
Output sum: z_out = 0.5×0.6106 − 0.6×0.5374 + 0.2 = 0.1829.
Output: a = σ(0.1829) = 0.5456.
Loss (binary cross-entropy, target 1): −ln 0.5456 = 0.6059.

Backpropagation by hand

Backpropagation applies the chain rule (chapter 5) from the loss backwards, one layer at a time. Call the slope of the loss with respect to any quantity its "delta".

Output delta. For sigmoid with cross-entropy the slope with respect to z_out is a − y = 0.5456 − 1 = −0.4544.
Output weights. z_out = w₁h₁ + …, so ∂L/∂w₁ = delta × h₁ = −0.4544 × 0.6106 = −0.2775, and for w₂: −0.4544 × 0.5374 = −0.2442. The bias gets the delta itself: −0.4544.
Back into the hidden layer. Each hidden output reached the loss through its output weight: ∂L/∂h₁ = −0.4544 × 0.5 = −0.2272; ∂L/∂h₂ = −0.4544 × (−0.6) = 0.2726.
Through the hidden sigmoids. Multiply by each sigmoid's slope h(1 − h): 0.6106 × 0.3894 = 0.2378 and 0.5374 × 0.4626 = 0.2486. Hidden deltas: −0.2272 × 0.2378 = −0.0540 and 0.2726 × 0.2486 = 0.0678.
Hidden weights. Delta times the input that weight multiplied. Hidden neuron 1: (−0.0540 × 0.5, −0.0540 × 1.0) = (−0.0270, −0.0540). Hidden neuron 2: (0.0339, 0.0678). The biases get the deltas.

Check by nudging. Raise hidden neuron 1's second weight from 0.4 to 0.400001 and rerun the forward pass: the loss falls by 0.000000054, a slope of −0.0540. The chain rule was right. This "gradient check" is how people test hand-written backprop code.

Update. With learning rate 0.5, every parameter moves by −0.5 × its gradient. For example the output weight 0.5 becomes 0.5 + 0.5 × 0.2775 = 0.6387, and hidden neuron 1's second weight becomes 0.4 + 0.5 × 0.0540 = 0.4270. Running the forward pass again gives an output of 0.6408 and a loss of 0.4451, down from 0.6059. One step of learning, done by hand.

Notice

Each delta was computed once and reused for every weight that feeds it. That reuse is why backpropagation costs only about twice a forward pass, instead of one forward pass per weight. Libraries such as PyTorch record the forward computation and run this backward sweep automatically.

Try it

In the worked example, what is the gradient for hidden neuron 2's bias, and which way does a gradient step move it?

Answer

Its delta, 0.0678. A gradient step subtracts 0.5 × 0.0678, so the bias falls from 0.1 to 0.066. That lowers h₂, which helps, because h₂ reaches the output through a negative weight.

Train one yourself

The widget runs plain full-batch gradient descent with exactly the backpropagation above, written out in book/assets/widgets/nntrain.js. Chapter 15 has you write the same thing in Python with NumPy.

Forward passes in everyday systems

Most applications built on trained models only ever run the forward pass: no backward pass, no training. Some call hosted language models over the network. Others also run small networks on their own machine. A local embedding model turns text into vectors; a named-entity model, such as the spaCy model inside the Presidio library, spots names and places in a message so a scrubber can mask them. When you store vectors, check the configured dimension against the model's real output size at start-up. A stored vector of the wrong length would make every search silently wrong.

Self-check

How many parameters does a 3 → 5 → 2 network have?

(5 × 3 + 5) + (2 × 5 + 2) = 20 + 12 = 32.

Why does a network with no activation functions behave like one linear layer?

A product of matrices is one matrix, so W₂(W₁x) = (W₂W₁)x. Without a bend between layers, depth adds nothing.

In backprop, how do you get the gradient for a weight from its neuron's delta?

Multiply the delta by the input that the weight multiplied in the forward pass.

What is a gradient check, and why use it?

Nudge one parameter by a tiny amount, rerun the forward pass and compare the change in loss with the computed gradient. If they disagree, the backprop code has a bug.