Part I · Maths · Chapter 3

Exponentials and logarithms

Repeated multiplication gives exponentials; asking "how many times?" gives logarithms. Money grows by the first, and every AI loss function is written with the second.

Repeated multiplication

Put 1,000 in an account paying 7% a year. Each year the balance is multiplied by 1.07 (chapter 2):

• After 1 year: 1,000 × 1.07 = 1,070.
• After 2 years: 1,070 × 1.07 = 1,144.90.
• After 3 years: 1,144.90 × 1.07 = 1,225.04.

After n years you have multiplied by 1.07 a total of n times, which is a power:

A function where the variable sits in the power is an exponential. Compare it with simple growth, where you earn 7% of the original 1,000 each year and nothing on the interest: after 10 years that is 1,000 × (1 + 0.07 × 10) = 1,700. The gap widens every year.

Power rules you will use

• a^m × aⁿ = a^m+n. Worked: 2³ × 2² = 8 × 4 = 32 = 2⁵.
• (a^m)ⁿ = a^mn. Worked: (2²)³ = 4³ = 64 = 2⁶.
• a⁰ = 1 and a⁻ⁿ = 1 / aⁿ. Worked: 2⁻³ = 1/8 = 0.125.
• A fractional power is a root: a^1/2 = √a, and a^1/20 is the twentieth root.

Decay

Multiplying by a number below 1 shrinks things exponentially. Halve something three times and you have 0.5³ = 0.125 of it. Halve it 100 times and you have 0.5¹⁰⁰ ≈ 0.000…0008, with 30 zeros after the decimal point. Keep that number in mind for the next section.

The number e

Suppose a bank pays 100% a year but adds the interest in smaller and smaller pieces. Paid once: 1 becomes 2. Paid monthly, 1/12 each time: (1 + 1/12)¹² = 2.613. Paid daily: (1 + 1/365)³⁶⁵ = 2.7146. The more often you compound, the closer you get to a fixed number:

You will meet e mostly through the function exp(x) = e^x. It has two properties that make it the favourite of AI: it is always positive, whatever x is (exp(−5) = 0.0067, exp(0) = 1, exp(3) = 20.1), and its slope at every point equals its own value (chapter 5). Softmax (chapter 9) uses exp to turn any list of scores into positive numbers.

Logarithms: counting the multiplications

A logarithm answers the question "what power do I need?"

• log₁₀ 1,000 = 3, because 10³ = 1,000.
• log₂ 8 = 3, because 2³ = 8.
• log₂ (1/8) = −3, because 2⁻³ = 1/8.

So log undoes exp, and exp undoes log: ln(e³) = 3 and e^{ln 5} = 5.

The rules that make logs useful

Check the first with base 10: log 100 + log 1,000 = 2 + 3 = 5, and log(100 × 1,000) = log 100,000 = 5. Logs turn multiplication into addition. Before calculators, engineers multiplied large numbers with printed log tables for exactly this reason.

The second rule solves "how long until it doubles?". We need 1.07ⁿ = 2. Take ln of both sides: n × ln 1.07 = ln 2, so n = 0.6931 / 0.0677 = 10.24 years. The "rule of 72" is a shortcut for the same thing: 72 / 7 = 10.3.

Logs of numbers below 1 are negative

Probabilities sit between 0 and 1, so their logs are zero or negative: ln 1 = 0, ln 0.9 = −0.105, ln 0.5 = −0.693, ln 0.1 = −2.303, ln 0.001 = −6.91. The closer the probability is to zero, the larger the negative number. Flip the sign and you get a "surprise" score: an event you gave probability 0.9 surprises you by 0.105; one you gave 0.001 surprises you by 6.91. That surprise is exactly the loss used to train language models (chapter 9).

In AI: adding logs instead of multiplying probabilities

A language model gives a probability to each next token. The probability of a whole sentence is the product of the step probabilities. Three steps with probabilities 0.9, 0.8 and 0.7 give 0.9 × 0.8 × 0.7 = 0.504. A thousand steps of about 0.5 each would give 0.5¹⁰⁰⁰ ≈ 10⁻³⁰¹. The 32-bit floats that models compute in round anything below about 10⁻⁴⁵ to exactly zero, and even 64-bit floats give up below about 10⁻³⁰⁸, which a few more steps would reach.

Take logs instead: ln 0.9 + ln 0.8 + ln 0.7 = −0.105 − 0.223 − 0.357 = −0.685, and exp(−0.685) = 0.504, the same answer. A thousand steps simply add up to about −693, which is easy to store. That is why models report "log probabilities" (often shortened to logprobs) and why training minimises a sum of negative logs.

Growth numbers in practice

Banks usually add interest monthly, not yearly. Divide the yearly rate by 12 and multiply that many times: growth = (1 + rate / 12) ** months. For 10,000 at 7% over 10 years that is 1.005833¹²⁰ = 2.00966, so the lump sum grows to 20,096.61. Yearly compounding would give 10,000 × 1.07¹⁰ = 19,671.51, a little less, because the interest starts earning sooner.

Saving a fixed amount every month adds a geometric series, and the sum has a closed form: contribution * (growth - 1) / monthly_rate. Saving 200 a month for those 10 years gives 200 × (2.00966 − 1) / 0.005833 = 34,616.96. At a 0% rate the formula divides by zero, so a careful calculator needs a separate branch for that case.

Money code should use a decimal type, not binary floats. Floats cannot hold 0.1 exactly: adding 0.1 ten times in Python gives 0.9999999999999999, and rounding errors like that show up in cents. Python's Decimal rounds the way a person expects.

Two more habits. To express a future amount in today's money, divide by (1 + inflation) ** years. And to turn a start and end value into one average yearly return, take a root: (end / start) ** (1 / years) - 1. If 10,000 grows to 26,533 in 20 years, the average return is 2.6533^1/20 − 1 = 5.0% a year.