Part IV · Applied concepts · Chapter 38

Frameworks and model providers

You can call a model with a plain HTTP request. Frameworks exist because real systems need more around that request: switching providers, running tool loops, validating structured output, tracing, and wiring many steps into a flow. This chapter sorts the options into layers, says what each layer buys and costs, and covers running open models on your own hardware.

The layers

Most systems use two or three layers. The trade is always the same: each layer saves code and adds something you must learn, pin and debug through.

Gateways and routing

A gateway puts many models behind one API. You change a model by changing a string, and you can say which providers may serve a request, in what order, and whether to fall back. That is also where data rules live: some providers keep prompts and some do not, and a gateway can be told to use only those that keep nothing (zero data retention). The cost is one more company in your data path, so its own terms matter as much as the model's.

Graphs for multi-step flows

When one answer takes several steps (check the input, plan, call several tools or sub-agents, merge, check the output), a graph keeps the flow readable. Each node is a function that reads the shared state and returns updates; edges, some conditional, say what runs next; parallel branches run together and their results are combined by a reducer.

Worked: three parallel lookups take 4, 6 and 5 seconds. Run one after another they take 4 + 6 + 5 = 15 seconds. Run in parallel the answer waits only for the slowest, max(4, 6, 5) = 6 seconds. Parallel branches cost the same tokens; they save time.

Running open models yourself

vLLM is a serving engine built for throughput. Its PagedAttention manages the attention key and value memory in pages, the way an operating system manages memory, which lets it pack many requests onto one GPU, and continuous batching adds new requests to a running batch instead of waiting. vllm serve <model> starts an OpenAI-compatible server, on http://localhost:8000 by default.

Ollama is built for convenience on one machine. ollama run <model> downloads and runs a model; its API is at http://localhost:11434/api, with an OpenAI-compatible one at /v1, and a Modelfile (starting with FROM) describes a custom model. Neither is run in this chapter, because both need model downloads; Ollama is one of the backends the labs can use (Lab 0).

The first question is memory. Worked: a 7-billion-parameter model at 16 bits per weight needs 7 × 10⁹ × 2 bytes = 14 GB for the weights alone, before the key and value cache that grows with every token in flight. At 4 bits it is about 3.5 GB (chapter 39).

Choosing

• Can you see the request? You should be able to log exactly what was sent and what came back.
• Can you control the loop? Turn limits, timeouts and spend caps must be yours to set.
• How fast does it change? Young frameworks change their APIs often. Pin exact versions and upgrade on purpose.
• Can you leave? Keep your own types for answers and tools, so a framework is an adapter at the edge, not the shape of your whole system.

Patterns that keep you in control

Whichever layers you pick, a few patterns keep the choice reversible:

• Name jobs, not models. Code asks for a job ("classify", "write", "check"). Settings map each job to a model reference whose prefix picks the route, for example a gateway, a direct provider call or a local server. Moving a job to another model or provider is then a settings change and a restart, with no code edit.
• Keep provider rules in one route object. Gateways accept per-request preferences: zero data retention, which quantisations are allowed (chapter 39), a price ceiling, speed targets, and an optional provider order and fallback rule. Build that object in one place from the job's settings.
• Read the cost from the response and record it per call. When calls overlap on one client, carry each call's route and cost in a per-call context (a context variable in Python), so overlapping calls each keep their own route and each cost is counted once. Put the provider's generation id on the trace span.
• Mind how a provider handles structured output. Some models are sent plain JSON mode with the schema pasted into the prompt; others accept a strict json_schema response format. Prefer the strict format where it exists, and then add no schema text to the prompt.
• Keep one gateway under the graph. A graph decides the order of steps; model calls inside its nodes should still go through the same single gateway (chapter 33), so scrubbing, caps and cost records apply to every call.
• Use a heavy library only where it is needed. If one tool needs an extra client library (an indexer that expects a particular unified client, say), keep it in that tool and out of the request path.
• Pin exact versions, and wait before upgrading. Pin the model, graph, scrubber, embedding and protocol packages exactly, and let fast-moving packages resolve only to releases at least two weeks old. Upgrade on purpose, with the tests.