Part IV · Applied concepts · Chapter 42

Running AI in production

A demo answers one question well. Production answers thousands a day within a time budget and a money budget, keeps working when a provider is slow or down, and tells you when quality slips. This chapter does the arithmetic of latency, cost and caching, and lists the engineering that keeps an AI service up.

Latency: first token and the rest

A model call has two speeds: the time to the first token (reading the prompt, queueing) and the rate at which tokens follow. Total time ≈ time to first token + output tokens / tokens per second.

Worked: first token after 0.8 s, then 60 tokens per second, for a 400-token answer: 0.8 + 400 / 60 ≈ 7.5 s. Streaming does not make that faster, but the customer sees words after 0.8 s instead of a blank screen for 7.5. Long outputs are the main cost in time, so asking for shorter, structured answers is a speed fix as well as a cost fix.

Streaming over HTTP usually means server-sent events: respond with text/event-stream and Cache-Control: no-cache, and add X-Accel-Buffering: no so a reverse proxy does not hold the stream back. Send a status event as each stage starts, then the answer. Work that does not affect the answer, such as extracting memory facts, can run after the answer is sent and arrive as a last event, so it never adds to the time the customer waits (chapter 37).

Watch percentiles, not the average. Sorted response times of 2, 2, 3, 3, 3, 4, 4, 5, 9 and 30 seconds have a mean of 6.5 s, but a median (p50) of 3.5 s, and the slowest 10% take 30 s. The p95 or p99 is what your unhappiest customers feel.

Cost per answer

Cost = input tokens × input price + output tokens × output price, with prices quoted per million tokens. Worked, at $2 per million input tokens and $10 per million output tokens: an answer that sends 6,000 tokens and receives 500 costs 6,000 × 2 / 10⁶ + 500 × 10 / 10⁶ = $0.012 + $0.005 = $0.017. At 10,000 answers a day that is $170 a day. An answer that makes five calls (classify, two lookups, merge, check) costs the sum of all five.

Record the cost of every call as it happens: collect the costs of all calls in a run, then write one row per call with model, provider, prompt version and token counts, and put each call's cost and tokens on its trace span so they sum up the tree to the whole answer (chapter 45). Judge calls belong in the same table, against the message they judged.

Prompt caching

Many calls start with the same long prefix: tool definitions, system prompt, reference documents. Providers can cache that prefix so later calls skip re-reading it. With Anthropic's API you mark the end of the reusable part with cache_control; the prefix is built in the order tools, then system, then messages, so put stable content first and anything that changes per request last. A cached prefix lives 5 minutes by default, refreshed each time it is used, or 1 hour if you ask. Prefixes below a minimum length (512 tokens on the newest models) are simply not cached.

Pricing is relative to the normal input price: writing a 5-minute cache entry costs 1.25×, a 1-hour entry 2×, and reading from the cache 0.1× for most models (0.05× for Opus 5.5 and Sonnet 5.5).

Worked, at $2 per million input tokens with a 0.05× read price: a 5,000-token prefix costs $0.010 per call uncached. Cached, the first call writes it for 5,000 × 2.50 / 10⁶ = $0.0125 and each later call reads it for 5,000 × 0.10 / 10⁶ = $0.0005. For 100 calls within the cache lifetime: $1.00 uncached against $0.0125 + 99 × $0.0005 ≈ $0.062 cached. Caching also cuts time to first token, because the prefix is not processed again.

Two places it pays most. In a tool loop, each round resends every earlier round, which is exactly a prefix, so marking the request for caching means later rounds read the earlier ones cheaply. And a fixed breakpoint on the system prompt of a repeated task lets the next conversation on that task read it too. Keep the table of prices in one dated place and check it against the provider's pricing page whenever you touch it: a cache read that is priced wrongly makes every cost chart wrong.

Rate limits, retries and batching

• Rate limits. Providers cap requests and tokens per minute. Know yours, and queue rather than fail when you approach them.
• Retries with backoff. Retry only errors that can succeed later (timeouts, overload, rate limits), waiting longer each time (1, 2, 4, 8 seconds) with a little randomness ("jitter") so many clients do not retry in step. Cap the attempts.
• Timeouts on everything. A call with no timeout can hold a request open for minutes.
• Retry budgets follow who is waiting. A person watching a spinner gets one quick retry after a couple of seconds. A background job building an index can retry the same transient errors (429, 500, 502, 503, 504 or a dropped connection) up to several times, 20 seconds apart. A call still rate-limited after the client's own quick retries can wait a few seconds and try once more.
• Batch what can wait. Nightly evaluations or document processing can go through a provider's batch API, which trades hours of delay for a lower price.

Staying up

• Fallbacks. If one provider fails, route the same model through another, or drop to a different model, and record which one answered.
• Circuit breakers. After repeated failures, stop calling a dependency for a while instead of piling up slow requests; try again later.
• Degrade gracefully. Decide what the customer sees when a step fails: a shorter answer, a cached one, or an honest "let me hand you to a person".
• Budgets as errors. A spend limit that refuses calls turns a runaway bill into a visible error.

Budgets that cannot be overrun

A budget checked after a call only tells you that you overspent. Book the worst case before the call instead: the worst-case cost is the maximum output tokens times the output price. Worked: with a cap of 4,096 output tokens on a model priced at $10 per million output tokens, one call reserves 4,096 × 10 / 10⁶ = $0.04096. Refuse the call when the amount already spent, plus the amounts reserved by calls in flight, plus this worst case, would reach the limit. Round each recorded cost up to the nearest millionth of a dollar so the meter never undercounts. And make the limit a required setting: a path that spends money should refuse to start when no limit is set, because an optional budget is the one that gets left off. If the counters live in the process's memory they reset on restart and do not add up across servers; that is acceptable for a prototype if you write it down.

What to watch