Part II · ML foundations · Chapter 10

What learning means: linear and logistic regression

Machine learning means choosing a function's parameters from examples instead of writing its rule by hand. Linear regression predicts numbers, logistic regression predicts yes or no, and together they contain every idea a neural network uses.

Why it matters

Every model in this book, up to the largest language model, is built from four parts: a function with parameters, a loss that scores its mistakes, an optimiser that lowers the loss, and data. This chapter assembles all four in the simplest possible setting, where you can check every number by hand.

The four-part recipe

Data: examples of inputs with the right outputs, called labels. Learning from labelled examples is supervised learning.
Model: a function with adjustable parameters (chapter 2).
Loss: one number that says how wrong the model is on the data (loss).
Optimiser: a method for changing the parameters to lower the loss, usually gradient descent (chapter 6).

The inputs are often called features. For a house price model they might be area, bedrooms and distance to the station; for a spam filter, counts of certain words.

Linear regression

The model is a straight line, ŷ = wx + b, where ŷ ("y-hat") is the prediction. With several features it is ŷ = w₁x₁ + … + w_nx_n + b, a dot product plus a bias (chapter 4).

The loss: mean squared error

Three examples: (x, y) = (1, 2), (2, 4) and (3, 5). Try the line w = 1.5, b = 0.667:

x	y	prediction 1.5x + 0.667	error	error squared
1	2	2.167	0.167	0.028
2	4	3.667	−0.333	0.111
3	5	5.167	0.167	0.028

The mean squared error (MSE) is the average of the last column: 0.167 / 3 = 0.056.

L(w, b) = 1n Σ_i (wx_i + b − y_i)²

Squaring makes every error count as positive and punishes big errors far more than small ones: an error of 3 costs 9, an error of 1 costs 1. That is also its weakness, as the widget shows: one far-off point drags the whole line towards it.

Finding the best line

For a straight line there is a formula. The best slope is the sum of (x − mean x)(y − mean y) divided by the sum of (x − mean x)². Here mean x = 2 and mean y = 3.667, so the slope is ((−1)(−1.667) + 0 + (1)(1.333)) / (1 + 0 + 1) = 3 / 2 = 1.5, and b = 3.667 − 1.5 × 2 = 0.667. That is the line in the table; no other line has a lower MSE on these points.

Gradient descent reaches the same answer step by step, and unlike the formula it works for any model. Start at w = 0, b = 0, where every prediction is 0:

∂L/∂w = (2/n) Σ (prediction − y) × x = (2/3)(−2×1 − 4×2 − 5×3) = (2/3)(−25) = −16.67.
∂L/∂b = (2/n) Σ (prediction − y) = (2/3)(−11) = −7.33.
With learning rate 0.05: w = 0 + 0.05 × 16.67 = 0.833 and b = 0 + 0.05 × 7.33 = 0.367. Already closer.

Logistic regression: predicting yes or no

Now the label is a class: spam or not, will default or not. We want a probability between 0 and 1. Take the same linear score z = wx + b and squash it with the sigmoid:

p = σ(z) = 11 + e^−zσ(0) = 0.5, large positive z gives nearly 1, large negative z gives nearly 0.

The loss is cross-entropy for two classes (chapter 9), also called binary cross-entropy: −ln p when the label is 1, and −ln(1 − p) when it is 0.

Worked example with w = 2, b = −3 and one input x = 2 whose label is 1:

Score: z = 2 × 2 − 3 = 1. Probability: σ(1) = 0.731.
Loss: −ln 0.731 = 0.313.
Gradient: with sigmoid and cross-entropy together, the slope of the loss with respect to z is simply p − y = 0.731 − 1 = −0.269 (the two-class case of chapter 9's result). So ∂L/∂w = −0.269 × x = −0.538 and ∂L/∂b = −0.269.
A step with learning rate 0.5 raises w to 2.269 and b to −2.866, pushing p for this example up towards 1.

The model says "yes" when p > 0.5, which is when z > 0. With two features the line w₁x₁ + w₂x₂ + b = 0 splits the plane into a yes side and a no side. This decision boundary is always straight, which is logistic regression's great limit: it cannot separate classes that need a curved border. The trainer in chapter 12 shows it failing, and a hidden layer fixing it.

With more than two classes, give each class its own score and use softmax instead of sigmoid. This is softmax regression, and it is exactly the last layer of a language model, with one class per token in the vocabulary.

Try it

Same model (w = 2, b = −3). An input x = 0.5 has label 0. Find p, the loss and ∂L/∂w. You need e² = 7.389.

Answer

z = 1 − 3 = −2, so p = 1 / (1 + 7.389) = 0.119. Label 0, so loss = −ln(1 − 0.119) = 0.127. Gradient: (p − y) × x = 0.119 × 0.5 = 0.060. Positive, so w goes down a little.

Choosing a loss

Predicting	Output layer	Usual loss
a number (a price, a temperature)	none (raw score)	mean squared error, or mean absolute error when outliers matter
yes or no	sigmoid	binary cross-entropy
one of many classes (including the next token)	softmax	cross-entropy

Why not train on accuracy, the share of answers that are right? Because accuracy is flat almost everywhere: nudging a weight slightly usually changes no prediction, so its slope is zero and gradient descent has nothing to follow. Cross-entropy changes smoothly with every nudge. We train on the smooth loss and report accuracy.

A classifier made of words

Not every classification decision needs a trained classifier. Systems built on language models often ask a model with written criteria instead. A short instruction plus one criterion per class (say simple, standard and complex for incoming questions) goes to a model, which returns a label and perhaps a confidence or probabilities. If the call fails, or the label is not one of the allowed classes, the system falls back to a safe default.

Compared with training a logistic regression, this needs no labelled data and can be changed by editing a sentence. The costs are a model call per item and probabilities that nobody has checked for calibration. Evaluation (chapter 44) is the place to measure how well it sorts.

Build it

In Lab 2 you build a test bench that runs a prompt over labelled cases and scores the replies, which is how you measure a classifier made of words.

Self-check

Name the four parts of the learning recipe.

Data with labels, a model with parameters, a loss that scores mistakes, and an optimiser that lowers the loss.

Predictions 3 and 5 against true values 4 and 5. What is the MSE?

Errors −1 and 0, squares 1 and 0, mean 0.5.

Why can logistic regression not separate classes arranged in a ring?

Its decision boundary is where a linear score equals zero, which is always a straight line (or flat plane). A ring needs a curved boundary.

Why train on cross-entropy but report accuracy?

Accuracy has zero slope almost everywhere, so gradient descent cannot use it. Cross-entropy changes smoothly with every parameter, so it can be minimised; accuracy is easier for people to read.