Part II · ML foundations · Chapter 14

From perceptron to transformer: the bottlenecks

The road from a single artificial neuron to the transformer is a chain of bottlenecks, each removed by one idea. Knowing the chain explains why today's models look the way they do, and which limits they still have.

The chain at a glance

The perceptron and XOR

The perceptron of 1958 is one neuron with a step activation: output 1 if w₁x₁ + w₂x₂ + b > 0, else 0. Its learning rule is simple: when it is wrong, add (target − prediction) × input to the weights. For AND it finds a line within a few passes. For XOR, which outputs 1 when exactly one input is 1, it never settles.

Why XOR is impossible for one line: we would need (0,1) and (1,0) above the line, so w₂ + b > 0 and w₁ + b > 0, and (0,0) and (1,1) below, so b ≤ 0 and w₁ + w₂ + b ≤ 0. Add the first two: w₁ + w₂ + 2b > 0. But the last two give w₁ + w₂ + 2b ≤ b ≤ 0. A contradiction. The 1969 book "Perceptrons" made limits like this famous, and funding for neural networks dried up for years.

A hidden layer solves it

Two hidden neurons are enough. Let h₁ compute OR (fire if x₁ + x₂ − 0.5 > 0), let h₂ compute AND (fire if x₁ + x₂ − 1.5 > 0), and let the output fire if h₁ − h₂ − 0.5 > 0: "OR but not AND".

The hidden layer moves the points into a new space where one line works. But a step function has zero slope, so these weights cannot be learned by gradient descent. The 1986 paper "Learning representations by back-propagating errors" popularised smooth activations and backpropagation (chapter 12), so hidden layers could be trained. The trainer in chapter 12 learns XOR this way.

Vanishing gradients and ReLU

With backprop, people tried deeper networks and found the early layers barely learned. The chain rule multiplies one factor per layer, and a sigmoid's slope is at most 0.25. Through 10 sigmoid layers the gradient shrinks by up to 0.25¹⁰ ≈ 10⁻⁶. The early layers, which should learn the basic features everything else builds on, receive almost nothing. The opposite problem, exploding gradients, happens when the factors are larger than 1.

The fixes arrived over a decade:

• ReLU (widely adopted around 2010 to 2012): its slope is exactly 1 for positive inputs, so it does not shrink the gradient where it is active.
• Initialisation that keeps signal sizes steady (chapter 13).
• Normalisation layers (2015 and 2016) that keep each layer's inputs well scaled.
• Residual connections, from "Deep Residual Learning for Image Recognition" (2015), which add a direct path for the gradient around each layer. Networks of over a hundred layers became trainable.

Two outside changes mattered as much: large labelled datasets such as ImageNet, and GPUs. In 2012 a deep convolutional network trained on GPUs won the ImageNet image-recognition competition by a wide margin, and the field turned towards deep learning.

Sequences: recurrent networks and LSTMs

Text is a sequence. A recurrent neural network reads it one token at a time and carries a hidden state forward, a vector that is meant to summarise everything so far:

Training it means backpropagating through time, and the same multiplication problem returns, now along the sequence. If each step back multiplies the gradient by about 0.5, the signal from 20 tokens ago arrives scaled by 0.5²⁰ ≈ 0.000001. In "The money I lent to my sister, who lives in Lisbon and works as a nurse, has not been paid back", a plain RNN struggles to connect "paid back" to "money".

The LSTM ("Long Short-Term Memory", 1997) adds a separate cell state updated by addition rather than repeated multiplication, with learned gates that decide what to forget, what to write and what to read. When the forget gate is near 1, the gradient passes back almost unchanged, the same trick as a residual connection. LSTMs powered machine translation and speech recognition through the mid-2010s.

Two bottlenecks remained. Every step depends on the previous one, so an RNN cannot process the tokens of a sentence in parallel, and training on huge text collections is slow. And in sequence-to-sequence translation ("Sequence to Sequence Learning with Neural Networks", 2014), the encoder squeezes the whole source sentence into one fixed-size vector before the decoder begins. A 5-word sentence and a 50-word sentence get the same few hundred numbers.

Attention

"Neural Machine Translation by Jointly Learning to Align and Translate" (2014) let the decoder look back at every encoder state when producing each word, weighting them by relevance. To translate one word, the model scores each source position, turns the scores into weights with a softmax (chapter 9), and takes the weighted average of the source states. Nothing has to be squeezed into one vector; the model learns where to look.

The transformer

"Attention Is All You Need" (2017) removed recurrence entirely. Every token attends to every other token directly, and a whole sequence is processed in parallel, which suits GPUs perfectly. Residual connections and layer normalisation, the fixes for deep training, are built into every block. This made it practical to train on far more text than before, and the GPT line of models followed: "Language Models are Few-Shot Learners" (2020) showed a model that could do new tasks from a few examples in its prompt, and "Training language models to follow instructions with human feedback" (2022) described the tuning that turned such models into assistants (chapter 19).

The transformer has its own bottleneck. Attention compares every token with every other, so its cost grows with the square of the input length: doubling the input from 1,000 to 2,000 tokens quadruples the attention work from 1 million to 4 million comparisons. That is why context windows have limits, why long prompts cost more and slow down, and why systems retrieve a few relevant passages instead of pasting in every document (chapter 24).

Keeping inputs bounded

Almost every model a developer calls today is a transformer: the hosted language models and the embedding models alike. Well-built applications keep each call's input bounded, which keeps that quadratic cost small. They reject or shorten an overlong question (a cap of 2,000 characters is typical for a chat box), and they retrieve a handful of passages, three to five, rather than reading whole documents.