Part IV · Applied concepts · Chapter 21

Prompt engineering

A prompt is the program you hand a language model, written in plain language. Prompt engineering is writing that program so the model does the same right thing every time, including on the inputs you did not think of. It is less about magic phrases and more about the habits of a good specification: say the job, give the reasons, show examples, fix the output shape and keep data apart from instructions.

What a prompt is made of

An API call to a chat model sends more than one block of text:

• A system prompt: standing instructions for the whole conversation, such as the role, rules and output format.
• Messages: the conversation so far, alternating user and assistant turns, with tool calls and tool results in between (chapter 28).
• Tool definitions: names, descriptions and input schemas of the tools the model may call. These are prompt text too, and their wording matters.
• An output schema, when structured output is used (chapter 16).

All of it becomes tokens in one context window. The model has no other knowledge of your situation: what is not in the prompt or in its training does not exist for it.

Seven habits that work

1. Say the job, the audience and why it matters

"Summarise this" leaves the model to guess length, reader and purpose. "Summarise this policy for a customer deciding whether to return a book; keep only what affects that decision" does not. Giving the reason behind a rule helps the model apply it to cases the rule did not list.

2. Be specific where it counts

Name the edge cases you care about and what to do in each. Vague rules ("be careful with numbers") change nothing; precise ones ("never state a price or a date from memory; quote it from the document") do. If two rules could conflict, say which wins.

3. Structure the prompt

Use headings or tags to separate role, rules, data and the request. Put long reference material before the question, and the question last. Clear sections also make the prompt easier for people to review and change.

4. Show examples

A few worked input-and-output pairs, called few-shot examples, often teach a format or a judgement call faster than paragraphs of rules. Two cautions: models copy surface features of examples (length, wording, even the order of labels), so vary them; and examples must agree with the rules, or the model will follow the examples.

5. Fix the output shape

If code will read the answer, ask for JSON matching a schema, enforce it where the provider allows, and validate it anyway (chapter 16). If a person will read it, say how long and in what form.

6. Keep data apart from instructions

Anything pasted into a prompt, a document, a web page, an email or a tool result, may contain text that looks like an instruction. Mark it clearly as data and tell the model never to follow instructions inside it. This reduces, but does not remove, the risk of prompt injection; the real defences are in chapter 35.

7. Do not let the model do arithmetic it can get wrong

Models predict digits as tokens (chapter 16). They are good at choosing which calculation to do and poor at guaranteeing its result. Give them a tool that computes, and tell them to use it for every figure.

A worked improvement

Suppose a prompt labels customer messages, and on a test set of 40 messages it gets 31 right (77.5%). Looking at the 9 failures is the work; guessing is not.

The numbers are illustrative, but the method is the point: a fixed test set, one change at a time, the score after each change, and every failure read. With only 40 questions, a gain of one or two is within noise (chapter 36 shows how to put an interval around a score), so keep the test set growing and rerun the whole set after every change, because a fix for one case can break another.

Split big jobs into steps

One prompt that must classify, retrieve, calculate and write tends to do each part worse. Splitting the job into steps, each with its own short prompt and checked output, makes each step testable on its own, lets cheaper models handle the easy steps, and shows you which step failed. The cost is more calls and more latency. A support assistant, for example, can be built this way: a first step labels the message and how hard it is, a second step answers it with the right tools, and checks run before anything is shown to the customer (chapter 29 lists the common shapes).

Prompts are code

• Keep them in files, under version control, not in string literals scattered through the code.
• Version them. When a prompt changes in a way that changes behaviour, give it a new version and record the version with every call, so any answer can be traced to the exact text that produced it.
• Test them on a fixed set of cases before and after every change, and look at the failures, not just the score (chapter 44).
• Aim a change at one failure. A prompt change is often one sentence written for one observed failure: when two answers contradict each other, add the rule that says which one wins. Then rerun the whole test set, because the sentence can shift other cases.
• Fill fixed wording in from code. Text that must not vary, such as a legal notice or an approved disclaimer, belongs in code that copies it into a placeholder in the prompt or onto the answer, not in the model's own words.
• Re-test when the model changes. A prompt tuned for one model is not guaranteed to behave the same on another, or on a newer version of the same one.