Part III · How LLMs work · Chapter 16

Language models, tokens and sampling

A large language model does one thing: given some text, it gives a probability to every possible next piece of text. Everything else, from chat to tool calls, is built by running that one step in a loop. This chapter covers the pieces of text (tokens), how they are chosen (tokenisers), how the next one is picked (sampling), and the limits and extras that follow: context windows, reasoning tokens and structured output.

The next-token machine

Give a model the text "An emergency fund should cover six". It returns a logit, a raw score, for every token in its vocabulary, often 30,000 to 200,000 of them. A softmax (chapter 9) turns the scores into probabilities. Suppose five tokens matter:

A token is picked (more on how below), appended to the text, and the whole thing goes back in to get the next token. This loop is called autoregressive generation. A 300-token answer is 300 trips through the model, which is why output tokens are slower and dearer than input tokens: the input is read in one parallel pass, the output is written one token at a time.

Tokens: why not words or letters?

A model needs a fixed list of symbols, its vocabulary. Whole words make a huge list that still misses new words, names and typos. Single letters make a tiny list but very long sequences, and attention costs grow with the square of the length (chapter 14). Tokens sit between: common words are one token, rarer words are a few pieces, and anything at all can still be spelled out from small pieces.

Byte-pair encoding, worked by hand

Byte-pair encoding (BPE) builds the vocabulary from data. Start with single characters, then repeatedly merge the most frequent adjacent pair into a new symbol. Take a tiny corpus: "rate" 5 times, "rates" 2, "late" 3, "later" 2 and "eat" 4. Mark the end of each word with _ so the tokeniser can tell "eat" at the end of a word from "eat" inside "eaten".

1. Count adjacent pairs, weighted by word counts. "a t" appears in all five words: 5 + 2 + 3 + 2 + 4 = 16. "t e" appears in four: 5 + 2 + 3 + 2 = 12. Merge a + t → at. Now "rate" is r at e _.
2. Recount. "at e" = 12 is now top. Merge at + e → ate.
3. "ate _" = 8 (rate 5, late 3). Merge ate + _ → ate_.
4. "r ate_" = 5. Merge r + ate_ → rate_: "rate" is now a single token.
5. "at _" = 4 and "e at" = 4 tie; the rule here takes the one that sorts first, at + _ → at_.
6. "e at_" = 4. Merge e + at_ → eat_.

After six merges the corpus went from 80 tokens to 31. A word the tokeniser never saw still works: "rated" becomes r | ate | d | _, four tokens, built from learned pieces plus single letters. Real tokenisers do exactly this on billions of words and stop after 30,000 to 200,000 merges. Most work on bytes rather than letters, so any text in any language, emoji included, can be encoded.

Other schemes follow the same idea with different merge rules. WordPiece, used by BERT-style models, picks merges by a likelihood score instead of a raw count and marks pieces that continue a word with ##. SentencePiece treats the space as an ordinary character, so it works for languages written without spaces.

What tokenising does to cost

Two questions of exactly 44 characters each, run through the WordPiece tokeniser of BAAI/bge-small-en-v1.5, a small open embedding model whose vocabulary has 30,522 tokens:

Plain English words are mostly one token each. Currency codes, numbers with separators and tickers break into many pieces. Rules of thumb such as "about four characters per English token" are only rough; text full of figures, code or non-Latin scripts usually costs more tokens per character. When cost matters, count with the provider's own tokeniser or token-counting endpoint.

Splitting numbers into arbitrary pieces is one reason models are unreliable at arithmetic: "15,000" arrives as three unrelated symbols. That is a strong argument for doing sums in code and letting the model call that code as a tool (chapter 28).

Choosing the next token: sampling

With the probabilities in hand, a decoding rule picks one token.

• Greedy: always take the most likely token ("months"). Repeatable, but long greedy outputs tend to loop and sound flat.
• Sampling: draw at random in proportion to the probabilities, so "months" 73.1% of the time and "salaries" 2.2%.
• Temperature T divides every logit before the softmax. At T = 0.5 the logits double and the softmax sharpens: "months" rises to 0.958. At T = 2 they halve and it flattens: "months" falls to 0.463 and "salaries" climbs to 0.080. As T approaches 0, sampling becomes greedy.
• Top-k: keep only the k most likely tokens and renormalise. With k = 2: months 0.731 / 0.852 = 0.858 and weeks 0.142. The long tail of odd tokens can never be picked.
• Top-p (nucleus sampling): keep the smallest set of top tokens whose probabilities add up to at least p. With p = 0.9: months (0.731), weeks (0.852 so far), years (0.933, past 0.9, stop). Renormalised: 0.784, 0.130 and 0.087. Unlike top-k, the set grows when the model is unsure and shrinks when it is confident.

Even at temperature 0, providers do not promise identical outputs on every call: batching, hardware and numeric rounding can change the last digits of a logit and flip a close choice. Treat repeatability as a goal you test for, not a guarantee.

Context windows

A model reads a bounded number of tokens at once: its context window. Everything counts towards it: the system prompt, the conversation so far, tool descriptions and results, retrieved documents, and the answer being written. Three facts follow.

1. Cost and speed grow with length. Every input token is billed and processed on every call, and attention work grows with the square of the length (chapter 18).
2. A bigger window is not a better memory. In "Lost in the Middle: How Language Models Use Long Contexts" (2023), models answered best when the needed fact was near the start or end of a long input and noticeably worse when it sat in the middle. Putting less, better-chosen text in front of the model often beats putting in everything; that is the subject of chapter 22.
3. Nothing persists between calls. The model has no memory of a previous request. Anything it should know, including earlier turns of the same chat, has to be sent again (chapter 37).

Reasoning tokens

Some models can "think" before answering: they generate a stretch of working tokens, then the answer. The working gives the model more computation per question, since each extra token is another pass through the network, and it helps most on multi-step problems. Providers call these thinking or reasoning tokens. Three practical points:

• They are billed, usually at the output price, even when the provider does not show them to you.
• They add latency, because they are generated one at a time before the first word of the answer.
• Most APIs let you set how much thinking to do, as a token budget or an effort level. More is not always better: for lookups and short tasks it mostly adds cost.

Structured output

Code cannot act on "Sure! Here is your answer:". It needs data. Structured output asks the model to reply in a fixed shape, usually JSON that matches a JSON Schema. There are two levels of strength:

• Asked for: the schema is in the prompt or request, and the model usually complies. The reply must still be validated, because "usually" is not "always".
• Enforced: the decoder is only allowed to sample tokens that keep the output valid against the schema (constrained decoding). At each step, tokens that would break the JSON get probability zero. The shape is then guaranteed, though the values inside can still be wrong.

Either way, the safe pattern is: parse, validate against the schema, and on failure send the error back and ask for a corrected reply, a bounded number of times. Well-built systems do exactly this.

In practice

• Tokens are the unit of cost. Record input_tokens, output_tokens and, where the provider reports them, reasoning_tokens for every call. Keep a table of prices per million tokens. To keep a budget safe, reserve the worst case before the call: its maximum output tokens times the output price.
• Reasoning is set per model. Some models accept a thinking setting, others an effort level, and some reason by their own default. On current Claude models the thinking counts towards max_tokens, so a small limit can leave no room for the answer.
• Often there are no sampling settings. Many applications set no temperature, top-k or top-p and use the provider's defaults, and some current models reject a temperature outright. Repeatability is handled by checking the output instead: take figures from tools, block digits in prose, and reject references that do not resolve to a tool result (chapter 34).
• Structured output everywhere. Have the model reply with a typed answer: a list of segments that are either prose or references to a field in a tool's result. Validate the reply. On failure, send the whole exchange back with a message such as "The previous JSON did not validate (…). Return corrected JSON only." and stop after a fixed number of tries.