Part III · How LLMs work · Chapter 17

The embedding layer and positions

A tokeniser turns text into whole numbers, token ids. A network needs vectors. The embedding layer turns each id into a vector of a few hundred or thousand numbers, and a positional scheme adds where in the sequence each token sits. Together they make the input that every transformer block works on.

From token id to vector: a table lookup

The embedding layer is a matrix E with one row per token in the vocabulary and d columns, where d is the model width. Token id 2 gets row 2. That is all.

Worked with a five-token vocabulary and d = 3:

The sentence "save invest fund" is ids (1, 2, 4), and its input is a 3 × 3 matrix made of rows 1, 2 and 4. The same lookup can be written as a matrix product: the one-hot vector (0, 0, 1, 0, 0) times E picks out row 2 (chapter 4). Thinking of it as a product matters for training: the gradient of the loss reaches E like any other weight, but only the rows of tokens that appeared in the batch receive an update.

Nobody writes these numbers. They start random and are learned with everything else. Training pushes tokens used in similar contexts towards similar rows, so in the table "save" and "invest" (cosine similarity 0.92) sit close together and "spend" points away from both (−0.88 with "save"). The arithmetic is in chapter 4.

How big is it?

Rows times columns. A 50,000-token vocabulary at d = 768 is 38.4 million numbers. At d = 4,096 and a 128,000-token vocabulary it is 524 million. In small models this is a large share of all parameters, which is one reason vocabularies are not made arbitrarily large. Many models reuse the same matrix at the other end, to turn the final vector back into a score for every token ("tied embeddings").

Attention is blind to order

The next chapter's attention compares every token with every other token and takes a weighted average. Nothing in that calculation knows which token came first: shuffle the input rows and each output row is the same, just shuffled. "Bank lends to customer" and "customer lends to bank" would look alike. So position has to be added to the input.

Learned positions

The simplest scheme is a second table, one row per position: position 0 has its own vector, position 1 another, up to some maximum. The input for a token is its token row plus its position row. The vectors are learned like any other weights.

The catch is the hard limit: a model trained with 512 position rows has no row for position 512. Inputs must be cut to fit, which is why many embedding models have a strict token limit.

Sinusoidal positions

"Attention Is All You Need" (2017) used fixed vectors made from sine and cosine waves, one frequency per pair of dimensions:

Worked for d = 4. Pair 0 uses frequency 1, pair 1 uses 1/10000^2/4 = 0.01:

The first pair spins fast and tells neighbours apart; the second barely moves and tells near from far over long ranges, like the second hand and hour hand of a clock. Their key property: the dot product of two positions' vectors depends only on the distance between them, since sin a sin b + cos a cos b = cos(a − b). The model can learn "two tokens back" once and use it everywhere.

Rotary positions (RoPE)

Most recent language models use rotary position embedding, from "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021). Instead of adding a vector to the input, it rotates the query and key vectors inside attention (chapter 18) by an angle proportional to the position, a different speed for each pair of dimensions. Attention compares queries and keys by their dot product, and the dot product of two rotated vectors depends only on the difference of their angles. So the score depends on how far apart two tokens are, never on where they sit.

Worked in two dimensions with angle 0.5 per position. Take q = (1, 0.5) and k = (0.8, 0.2); unrotated, q · k = 0.9.

• Query at position 1, key at position 3: rotate q by 0.5 and k by 1.5. The dot product equals q · (k rotated by 1.5 − 0.5 = 1.0) = (1)(0.8 cos 1 − 0.2 sin 1) + (0.5)(0.8 sin 1 + 0.2 cos 1) = 0.264 + 0.391 = 0.655.
• Query at 11, key at 13: the angles are 5.5 and 6.5, the difference is still 1.0, and the score is still 0.655.
• Same position: difference 0, score 0.9 as unrotated.

Rotary positions have no table with a last row, so a model can at least run past its training length, though quality drops unless the rotation speeds are rescaled. Methods that stretch the angles are how many models extend their context windows after training. Another approach, from "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (2021), adds no position vectors at all and instead subtracts a penalty from each attention score in proportion to distance.

The start of the residual stream

After the lookup and positions, each token is one vector of size d. That vector then passes through every block of the model, and each block adds something to it rather than replacing it (residual connections, chapter 13). It helps to picture a stream per token that starts as "which token, where" and gathers context block by block until, at the end, it holds enough to score the next token.

Both ideas at real size

A small open embedding model, BAAI/bge-small-en-v1.5, shows both ideas at real size. Its public config.json gives a vocabulary of 30,522 tokens and a width of 384, so its token table holds 30,522 × 384 = 11.7 million numbers. It uses learned absolute positions with 512 rows, which is why it truncates its input at 512 tokens.

That model's output is not the token table itself. It runs the whole transformer and pools the final vectors into one 384-number vector per text, the kind of embedding that search uses (chapter 23). If you store such vectors, check the configured dimension against the model's own vector size, so that a mistyped setting fails loudly instead of producing vectors of the wrong length.