Part V · Build it

Lab 5: tools and an agent loop

Give a model four tools and a loop, and it can answer a question no single call could: look up two parts, add their prices exactly, work out a delivery date and, once a person says yes, place an order. In this lab you write the tools, the checks around them and the loop itself in plain Python, and watch every step go by.

What you will build

• Four tools over an invented bike shop's catalogue, each described by a JSON Schema the model reads.
• A calculator that does exact decimal arithmetic and cannot be talked into running code.
• A check on every tool request, with every failure sent back as an error result the model can act on.
• An approval step: the order tool runs only after a person types y.
• The tool loop: call the model, run the tools it asks for, send the results back, repeat, and stop at a step limit.
• A scripted stand-in model, so the whole run works offline and in tests.

Before you start

• Your workbench from Lab 0. Lab 1 explains the fields of a single reply (stop_reason, token counts, cost), which this lab reads at every step.
• The ideas: chapter 28 (tools and function calling) and chapter 29 (agents). Steps 3 and 4 lean on chapter 33 (harness), chapter 34 (guardrails) and chapter 35 (security).
• The files, in labs/05-agent/: agent.py (the tools, the checks, the loop and the stand-in model, about 500 lines), catalogue.json (twelve invented parts with prices in GBP) and test_agent.py.
• Cost: nothing on the fake backend or Ollama; about a tenth of a cent a run on claude-haiku-5-5 (worked out below).

Step 1: describe the tools

A tool is two halves. The model sees a name, a description and an input schema; your code holds the function, which the model never sees. In agent.py each tool is a small Tool record that keeps the two halves together:

@dataclass
class Tool:
    name: str
    description: str                 # the only manual the model gets
    input_schema: dict[str, Any]     # JSON Schema; checked by check_input before the tool runs
    run: Callable[..., dict[str, Any]]
    side_effect: bool = False        # True means a person must approve every call

    def spec(self):
        """The definition sent to the model with every call."""
        return {"name": self.name, "description": self.description, "input_schema": self.input_schema}

Three habits from chapter 28 show up in the definitions. Each description says what the tool returns and when to use it, and names its limits: add_working_days says "It does not know about public holidays", so the model can warn the customer. Prices travel as decimal strings such as "6.49", never as JSON numbers. And every schema sets "additionalProperties": false, so a made-up argument is an error, not something silently ignored.

The definitions are not free. The program's first line estimates them at about 495 tokens, using the rough rule of four characters per token, and they go with every call: a five-call run sends them five times.

Step 2: a calculator that cannot run code

The tempting calculator is one line: eval(expression). Do not write it. eval runs any Python, and the expression is written by a model that may have read text planted by someone else (chapter 35). __import__('os').system(...) is a valid "expression" that runs a shell command with your permissions.

Instead, calculate asks Python's ast module to parse the text into a syntax tree, without running it, and then walks the tree itself, accepting only three kinds of node: a number, an arithmetic operator from a fixed table, and a leading plus or minus. Anything else, a name, a function call, an attribute, a piece of text, is refused with a message saying which piece:

def _evaluate(node, source):
    if isinstance(node, ast.Constant) and type(node.value) in (int, float):
        # Read the digits as written, so 6.49 stays exactly 6.49.
        return Decimal(node.value) if type(node.value) is int else Decimal(ast.get_source_segment(source, node))
    if isinstance(node, ast.BinOp) and type(node.op) in _BINARY:
        left, right = _evaluate(node.left, source), _evaluate(node.right, source)
        if isinstance(node.op, ast.Pow) and (right != right.to_integral_value() or abs(right) > MAX_EXPONENT):
            raise ToolInputError(f"a power must be a whole number no bigger than {MAX_EXPONENT}")
        return _BINARY[type(node.op)](left, right)
    if isinstance(node, ast.UnaryOp) and type(node.op) in _UNARY:
        return _UNARY[type(node.op)](_evaluate(node.operand, source))
    raise ToolInputError(f"only numbers and + - * / // % ** are allowed, and {ast.unparse(node)!r} is neither")

The numbers are Python Decimals built from the digits as written. A binary float cannot hold 0.1 exactly, so in plain Python 0.1 + 0.2 prints 0.30000000000000004; this calculator returns 0.3. These are real results from the module:

Two limits keep the work small whatever the model sends: an expression of at most 200 characters, and whole powers no bigger than 100. Chapter 28 explains why arithmetic belongs in code at all.

Step 3: check every request before it runs

The model writes the arguments, so they are untrusted input, however well it usually does. check_input checks them against the tool's schema: types, required fields, unknown fields, lengths and number ranges, the parts of JSON Schema these four tools use. Its messages name the field and the problem, such as input.items[0].quantity must be at most 10, but never repeat the value, which in a real system could be someone's details. (Claude can also be asked to keep its arguments schema-valid, but a check in your own code costs nothing and protects every backend.)

run_tool is where a request becomes a result. Every way it can fail becomes a result marked is_error, so one bad call never ends the loop:

def run_tool(call, approve=ask_person):
    ...
    tool = REGISTRY.get(call.name)
    if tool is None:
        return result(f"there is no tool called {call.name!r}; the tools are {', '.join(REGISTRY)}", True)
    try:
        check_input(tool.input_schema, call.input)
    except ToolInputError as exc:
        return result(f"invalid input for {call.name}: {exc}", True)
    if tool.side_effect and not approve(call):
        return result("the person did not approve this, so it did not run. Do not try again; "
                      "ask what they would like to change.", True)
    try:
        output = tool.run(**call.input)
    except ToolInputError as exc:   # a mistake the model can fix
        return result(str(exc), True)
    except Exception as exc:        # a bug in the tool: report it and keep the loop alive
        return result(f"{call.name} failed: {type(exc).__name__}: {exc}", True)
    return result(json.dumps(output, ensure_ascii=False))

Two kinds of failure are kept apart. A ToolInputError is a mistake the model made and can fix, so its message is written for the model. Any other exception is a bug in your tool; the model still gets a result, so it can tell the customer something went wrong, and you get the exception's type in the trace. MCP servers draw the same line (Lab 6). Chapter 28 and chapter 33 describe these guards in general.

Step 4: a person approves side effects

place_order is marked side_effect=True. Before it runs, the loop, not the tool and not the model, shows the exact request and asks:

def ask_person(call):
    """Show the exact request and let a person decide. Anything but y or yes means no."""
    print(f"   ?? the model wants to run {call.name} with {json.dumps(call.input)}")
    try:
        answer = input("   ?? allow it? [y/N] ")
    except EOFError:  # no one at the keyboard: the safe answer is no
        answer = ""
    return answer.strip().lower() in ("y", "yes")

Three choices matter. The default is no: Enter alone, a typo, or a run with no keyboard at all does not place an order. A refusal goes back to the model as an error result that says not to try again, so it asks the customer instead of retrying. And --auto-approve exists only for tests and demos, where nobody is there to ask.

The approval lives in code because a description is only advice. "Use it only when the customer asks you to order" steers a model that is working well; a model that misread the question, or was steered by injected text, can still ask for an order, but it cannot type y. This is "the model proposes, code disposes" from chapter 29, and the permissions part of a harness (chapter 33).

Step 5: the loop

This is the whole agent. Everything else in the file is tools, checks and the offline stand-in:

for step in range(1, max_steps + 1):
    reply = llm.chat(messages, system=system, tools=specs, max_tokens=max_tokens, effort=effort)
    ...  # add up calls, tokens and cost, print the step

    if reply.stop_reason != "tool_use" or not reply.tool_calls:
        # The model answered (end), ran out of room (max_tokens) or declined (refusal).
        run.answer, run.stop = reply.text, reply.stop_reason
        return run
    if step == max_steps:
        # No call is left to read the results, so do not run anything, side effects least.
        break

    # The assistant turn goes back exactly as it came, then one turn with every result.
    messages.append(reply.message())
    results = [run_tool(call, approve) for call in reply.tool_calls]   # printed as they run
    messages.append(llm.tool_results(results))

run.stop = "step_limit"
run.answer = f"I stopped after {max_steps} model calls without finishing, ..."

• How it ends. Any stop_reason other than tool_use ends the loop. end is an answer. max_tokens means the reply was cut off; on Claude the model's thinking counts toward max_tokens, which is why the loop asks for 4,096 rather than the adapter's default of 2,048.
• One turn of results. A reply can ask for several tools at once; step 1 of the demo asks for two lookups. They all run, and all their results go back together in one turn.
• The step limit. max_steps (8 by default) counts model calls. When the last allowed call still asks for tools, the loop does not run them: no call is left to read their results, and a side effect that nobody reads is the worst kind. It stops with stop set to step_limit and a plain message for the customer. Chapter 28 describes another choice, one last call that must answer from what it has; it is in Going further.
• Growing input. The model keeps nothing between calls, so each call resends the whole conversation. In the demo run below, the fake backend's input counts go 156, 484, 564, 640, 729: the arithmetic series of chapter 28, which is why the step limit is also a cost limit.

This is the observe, decide, act loop of chapter 29, with the model doing the deciding and your code doing everything else.

Why the adapter replays the provider's raw reply

The loop appends reply.message(), not a turn it builds itself from reply.text and reply.tool_calls. For Claude, reply.message() carries the reply's content blocks exactly as they arrived (the _raw field), and the adapter sends those blocks back on the next call instead of rebuilding them.

The reason is thinking. Current Claude models think before they answer, and a reply can open with thinking blocks before the tool calls. Each carries a signature, and under the default setting its text comes back empty, so it is easy to think it can be dropped. It cannot. When you send tool results, the assistant turn that asked for the tools must go back with its thinking blocks unchanged: the API rejects thinking blocks whose content was modified, and dropping them breaks the turn, because they carry the model's reasoning across the tool call. Rebuilding the turn from text and tool calls would drop them. The Ollama and fake backends keep no raw content, so their turns are rebuilt from the shared format, which is all they need. The test test_the_assistant_turn_is_replayed_as_it_came checks that the loop passes the raw content through untouched.

Step 6: a stand-in model for offline runs

The fake backend has two ways to play a model. llm.fake.script(...) queues fixed replies, which suits tests:

llm.fake.script(llm.ToolCall("x1", "weather", {"city": "York"}), "I cannot check the weather.")
run = agent.run_agent("Will it rain?", say=quiet)
assert results_sent(run)[0]["is_error"]          # an unknown tool came back as an error result

llm.fake.responder takes a function that chooses each reply from the request. pretend_model is one: for the two demo questions it decides each step from the conversation so far, reading the prices from the lookup results and today's date from the system prompt, as a real model would. It is a script, not a model, and its first sum has a deliberate slip (x for times) so you can watch an error result go back and be corrected. main() installs it only when the backend is the fake one; any other question gets a note saying to use a real model.

Checklist

• I ran the demo on the fake backend and read every step.
• I can say what the model sees of a tool and what it never sees.
• I tried to make calculate run code, and it refused.
• I watched an error result go back to the model and be corrected.
• I declined an order and saw that it did not run.
• I ran with --max-steps 2 and saw what happens at the limit.
• I can explain why the assistant turn goes back to Claude unchanged.
• python -m pytest 05-agent passes in labs/.
• I asked a question of my own on Claude or Ollama.

Run it

# from the repository root
python labs/05-agent/agent.py --today 2026-11-02      # the demo question
python labs/05-agent/agent.py --demo order             # an order: you are asked y/N
python labs/05-agent/agent.py --max-steps 2            # the step limit
cd labs && python -m pytest 05-agent && cd ..

--today fixes the date so your run matches this one; leave it out and the agent uses your clock. On the fake backend the demo prints exactly this:

backend: fake; 4 tools, about 495 tokens of definitions sent with every call
question: I need two inner tubes for 700x28c tyres and a pair of disc brake pads. What do they cost altogether, and if I order today, when would they arrive?

step 1: 156 tokens in, 1 out, stop: tool_use
   -> lookup_part {"query": "inner tube 700x28"}
   <- {"query": "inner tube 700x28", "currency": "GBP", "matches": [{"sku": "TUB-700-28", "name": "Inner tube 700x28-32c, Presta valve", "price": "6.49",...
   -> lookup_part {"query": "disc brake pads"}
   <- {"query": "disc brake pads", "currency": "GBP", "matches": [{"sku": "PAD-DISC-R", "name": "Disc brake pads, resin, one pair", "price": "18.50", "st...
step 2: 484 tokens in, 1 out, stop: tool_use
   -> calculate {"expression": "2 x 6.49 + 18.50"}
   <- ERROR '2 x 6.49 + 18.50' is not arithmetic: use numbers, + - * / // % ** and brackets
step 3: 564 tokens in, 1 out, stop: tool_use
   -> calculate {"expression": "2 * 6.49 + 18.50"}
   <- {"expression": "2 * 6.49 + 18.50", "value": "31.48"}
step 4: 640 tokens in, 1 out, stop: tool_use
   -> add_working_days {"start": "2026-11-02", "days": 3}
   <- {"start": "2026-11-02", "working_days": 3, "date": "2026-11-05", "weekday": "Thursday"}
step 5: 729 tokens in, 41 out, stop: end

answer (end): Two inner tubes (TUB-700-28, 6.49 each) and a pair of disc brake pads (PAD-DISC-R, 18.50) come to 31.48 GBP. Ordered today, they would arrive on Thursday 2026-11-05.
total: 5 model calls, 2573 input and 45 output tokens, $0.000000
the same tokens on claude-haiku-5-5 (adding 5 x 495 tokens of tool definitions) would cost $0.000527, plus the thinking a real model writes

Read it as the model would. Step 1 asks for two lookups at once. Step 2's sum is not Python, comes back as an error, and step 3 fixes it. Step 4 gets the date from code rather than from the model's sense of the calendar. Every figure in the answer appears in a tool result above it. The fake's token counts come from the four-characters-per-token rule and leave out the tool definitions; its "1 out" for a tool request is not what a real model writes.

With --max-steps 2 the run ends after the second call, and the sum it asked for is reported but not run:

step 2: 484 tokens in, 1 out, stop: tool_use
   not run: calculate {"expression": "2 x 6.49 + 18.50"}

answer (step_limit): I stopped after 2 model calls without finishing, so I have no answer yet. Try a narrower question, or allow more steps.

With --demo order the loop stops to ask. Typing n gave this (the first lines are left out):

step 2: 324 tokens in, 1 out, stop: tool_use
   -> place_order {"items": [{"sku": "TUB-700-28", "quantity": 2}]}
   ?? the model wants to run place_order with {"items": [{"sku": "TUB-700-28", "quantity": 2}]}
   ?? allow it? [y/N] n
   <- ERROR the person did not approve this, so it did not run. Do not try again; ask what they would like to change.
step 3: 415 tokens in, 22 out, stop: end

answer (end): I have not placed the order, because it was not approved. What would you like to change?

Typing y instead prints [side effect] pretend order ORD-7565EF: 1 line(s), 12.98 GBP and the answer names that reference. The tests run in well under a second: 48 passed.

On a real model

LLM_BACKEND=anthropic python labs/05-agent/agent.py
LLM_BACKEND=anthropic python labs/05-agent/agent.py "Do you have an 11-speed chain, and what would two cost?"
LLM_BACKEND=ollama python labs/05-agent/agent.py "Do you have an 11-speed chain, and what would two cost?"

A real model's run differs from the scripted one, and from one run to the next: it usually writes the sum correctly the first time, so the demo often takes four calls, and its wording and token counts are its own. The output tokens include its thinking, so expect far more than the fake's handful. What to check is the same every time: each price in the answer appears in a lookup result, each total is a calculate value, and each date came from add_working_days. Small local models call tools less reliably; watch for an answer that skips the tools and guesses.

What it costs

Nothing on the fake backend or Ollama. For claude-haiku-5-5 ($0.10 per million input tokens, $0.50 per million output), worked out from the fake run above:

• Input. The fake counted 2,573 tokens of messages over 5 calls. Each call also carries the tool definitions, about 495 tokens, which the fake does not count: 5 × 495 = 2,475. Together, 5,048 input tokens.
• Output. The fake writes almost nothing. A real model writes its thinking as well as each tool request; allow 300 tokens a call, 1,500 in all.

5,048 × $0.10 / 1,000,000 + 1,500 × $0.50 / 1,000,000 = $0.000505 + $0.000750 ≈ $0.0013 a run

So a hundred runs of the demo cost about 13 cents. The ceiling is every call using its whole max_tokens of 4,096: 20,480 output tokens, about $0.011 a run. Anthropic also adds a short system prompt of its own whenever tools are sent, which this estimate leaves out; on the anthropic backend the totals the program prints are the provider's own counts, so trust those over this estimate. The order demo makes three calls and costs less.

Experiments

1. Count the limit. Predict which request is left unrun with --max-steps 3, then run it. Then find the smallest limit at which the demo finishes.
2. Attack the calculator. In a Python shell, import the module (import sys; sys.path.insert(0, "labs/05-agent"); import agent) and try to make agent.calculate read a file or print something. Then, in a scratch copy only, replace it with eval and try __import__('os').getcwd(). That one line is why Step 2 exists.
3. Remove the approval. Set side_effect=False on place_order and run --demo order. The order goes through with no question asked. Put it back, and run the tests: test_a_side_effect_waits_for_a_yes would have caught it.
4. Descriptions steer. On a real model, delete "Use it for every sum instead of working it out yourself" from calculate's description and ask the demo question a few times. Does the model still call it, or add up in its head?
5. Effort. On Claude, run the demo with --effort low and --effort high. Compare the output tokens, the cost line and the number of steps.
6. A fifth tool. Add one, such as converting kilometres to miles for a customer planning a ride. Write its schema, add it to TOOLS and a test, and watch the definitions estimate on the first line grow.

Going further

• One last call at the limit. Instead of a fixed message, make one more call that asks the model to answer from what it has already read (chapter 28). Decide how you will stop that call from asking for tools again.
• Repeated calls. Answer a repeated tool call, the same tool with the same arguments, with the earlier result and a note to answer now, so a stuck model is pushed on.
• Run requests in parallel. Step 1's two lookups are independent. Run them in threads and keep their results in request order.
• Trace it. Lab 9 uses llm.add_listener to record every call with its tokens and cost; point it at this loop.
• Serve the tools over MCP. Lab 6 builds an MCP server, the standard way to offer tools like these to any host.
• Guard the answer too. Lab 7 adds tested checks on what goes in and out.