Part IV · Applied concepts · Chapter 23

Embeddings and similarity

An embedding model turns a whole piece of text into one vector, a list of a few hundred numbers, arranged so that texts with similar meaning get vectors pointing in similar directions. Comparing meaning becomes comparing directions, which a computer can do millions of times a second. That is the foundation of semantic search and of retrieval-augmented generation.

From token vectors to one text vector

A transformer gives one vector per token (chapter 18). An embedding model is a transformer, usually one that reads without a causal mask, plus a pooling step that turns those many vectors into one: take the vector of a special first token, or average all of them. It is then trained so that the pooled vectors of related texts are close.

The training is usually contrastive: show the model pairs that belong together (a question and the passage that answers it, two paraphrases) and push their vectors together, while pushing apart pairs that do not. Worked with three unit vectors in two dimensions: a question q = (0.6, 0.8), its answer a = (0.8, 0.6) and an unrelated passage u = (1, 0). The similarities are q·a = 0.96 and q·u = 0.6. A softmax over these two scores (chapter 9), scaled by a temperature of 0.1, gives the right answer probability e^9.6 / (e^9.6 + e⁶) = 0.973, and the loss is −ln 0.973 = 0.027. Training lowers that loss over millions of pairs, each batch using the other examples as the "unrelated" ones.

Measuring similarity

Three measures are common (chapter 4):

• Cosine similarity: the cosine of the angle between two vectors, from −1 to 1. It ignores length and compares direction only.
• Dot product: the sum of products. For vectors of length 1 it equals the cosine.
• Euclidean distance: the straight-line distance. For unit vectors, distance² = 2 − 2 × cosine, so it ranks results in exactly the same order as cosine.

Most embedding models return unit-length vectors, which makes the three equivalent for ranking. Databases often store cosine distance, 1 − cosine, so that smaller means closer.

Real embeddings, real scores

The widget below uses real vectors. Twelve short personal-finance questions, and three search queries, were embedded with BAAI/bge-small-en-v1.5, a small open embedding model that runs on a laptop, giving 384 numbers each. The first eight numbers of "How much should I keep in an emergency fund?" are −0.041, −0.041, −0.044, −0.029, 0.026, 0.001, 0.025, 0.008. No single number means anything on its own; meaning lives in the direction of all 384 together. The vectors were computed once for this book from the model's full-precision weights and stored in the widget rounded to three places.

Three lessons from these numbers:

1. Meaning beats wording. "I was laid off. How many months can I live on my savings?" finds "How long would my savings last without a salary?" first (0.790), with no key words in common beyond "savings".
2. Scores are relative. Across all the pairs here, the lowest similarity between two different sentences is 0.444; nothing comes near 0. A score of 0.6 is not "60% relevant". Thresholds must be set per model, from data.
3. Abbreviations hurt. "best FD rates" ranks "Which bank has the best term deposit rate?" first (0.680), but puts the emergency-fund question (0.639) just above "What rate does a one-year fixed deposit pay?" (0.637). The small model does not reliably know that FD means fixed deposit. Fixes include query rewriting, a bigger model and combining with keyword search (chapter 25).

Choosing and changing an embedding model

• Dimensions. More numbers can hold finer distinctions but cost storage and search time: 384 numbers at 4 bytes is 1.5 KB per passage, 1,024 numbers is 4 KB.
• Maximum input. Text past the model's limit is cut off (512 tokens for this small model), so passages must be split to fit (chapter 24).
• Query and passage modes. Some models embed questions and passages differently, because a short question and a long passage look different. Use the right mode for each.
• Never mix models. Vectors from two models, or two versions of one model, live in different spaces; comparing them is meaningless. Changing the model means re-embedding every stored passage. Record the model name and vector size beside the index, and refuse to search when the setting no longer matches what the index was built with.
• Hosted or local. A hosted embedding service is convenient and sends your text to someone else, so send only text you are willing to share. Retry briefly on rate-limit and server errors, wait longer when indexing offline than when a user is waiting, and check the length of every vector that comes back.
• Evaluate on your data. Public benchmarks are a starting point. What matters is whether the right passage comes first for your questions (chapter 26).
• Scores in a database. Vector databases usually store cosine distance and sort ascending; turn it back into similarity (1 minus the distance) before you show a score or apply a threshold.