Part IV · Applied concepts · Chapter 22

Context engineering and choosing models

Prompt engineering is about the words of the instructions. Context engineering is about everything else that goes into the window on each call: which history, which documents, which tool results, in what order and how much of each. The second half of the chapter covers the other big lever you control: which model gets the call.

The window is a budget

A model sees only what is in its context window on this call (chapter 16). A typical call of an assistant that uses tools might hold:

At $2 per million input tokens and $10 per million output tokens, with a 600-token answer, one call costs 13,250 × 2 / 10⁶ + 600 × 10 / 10⁶ = $0.0265 + $0.006 = $0.0325, or $32.50 per thousand questions. Notice that the question is 0.4% of the input. Almost everything you pay for is context you chose to send.

Five things to do with context

1. Select. Send what this question needs, not everything that might help. Retrieve the top few passages (chapter 24) instead of pasting whole documents; give each step only the tools it needs.
2. Order. Stable material first, the changing material after it, and the question last. Models use the start and end of a long input better than the middle: in "Lost in the Middle: How Language Models Use Long Contexts" (2023), accuracy dropped when the needed fact sat mid-window. Put what matters most near an edge.
3. Compress. Summarise or drop old turns, trim tool results to the fields the model needs, and remove duplicates. Long agent sessions often "compact" their history: replace the oldest part with a summary written by the model, keeping decisions and open questions.
4. Isolate. Give a side task its own fresh context and take back only its conclusion. That is the main idea behind subagents (chapter 32): a search that reads 40 files should not leave those 40 files in the main window.
5. Persist outside the window. Facts that must survive across chats belong in storage your code reads back deliberately, such as a memory store (chapter 37) or notes files, not in an ever-growing history.

Two cheap limits go with these. Cap how much chat history is sent, for example the last three exchanges, rebuilt from stored text. And cap the length of the question a user may submit, so one huge paste cannot fill the window. Both put a ceiling on the size, and so on the cost, of every call.

A useful test for any piece of context: if a careful human expert were given this window, would this piece help them answer, or would they have to skip over it?

Prompt caching

Providers can cache the processed form of a prompt's opening, so a later call that starts with exactly the same tokens skips that work and pays less for it. On Anthropic's API, as its documentation described it when this book was checked:

• The cached prefix is built in the order tools, then system prompt, then messages. You mark where it ends with a cache_control breakpoint, up to 4 per request.
• A cached prefix lives 5 minutes by default, refreshed each time it is used; a 1-hour lifetime costs more to write.
• Writing to the 5-minute cache costs 1.25 times the input price; reading from it costs a small fraction, 0.1 times for most models.
• Prompts below a minimum length are not cached at all (512 tokens for the newest models, more for older ones), silently.

The design rule that follows: put everything stable at the start and keep it byte-identical. A timestamp or a user's name near the top of the system prompt breaks the cache on every call. In the example, the 3,500 stable tokens read from cache at 0.1 times the price bring the call from $0.0325 to $0.0262, a saving of about 19% (before the cost of the occasional cache write).

Choosing a model

There is no best model, only the best model for a step. Compare candidates on the things that matter for that step:

Routing: small models for easy questions

If most questions are easy, sending all of them to the largest model wastes money and time. A router classifies each request and sends it to a model tier. Worked with prices per million tokens (input/output) like those of the three Claude models used in the labs: a small model at $0.10/$0.50, a medium one at $2/$10, a large one at $4/$20, and a call of 10,000 input and 500 output tokens:

Against $0.025 for sending everything to the medium model, that is 52% cheaper, before the router's own cost. The catch is misrouting: a hard question sent to the small model gets a worse answer. Routers should fail towards the safer tier, keep a floor under categories where a weak answer is costly, and let a failed answer retry one tier up.