Part IV · Applied concepts · Chapter 43

AI in a regulated domain

Finance, health, insurance and law are regulated domains: what a system tells people can affect their money, wellbeing or rights, and the organisation must be able to show how each answer came about. This chapter is about the engineering habits that make an AI system fit for that setting. It makes no claims about what any law requires; that is a question for your compliance and legal colleagues, and the habits below are what lets you answer them.

Every answer traceable

Record enough with each answer to rebuild it later:

• the input, as stored (with personal data masked);
• every tool call with its arguments and result, and every document passage used, each with its as-of date;
• the model ids, the prompt versions and the code version (a git commit);
• the checks that ran and their verdicts, including the reasons for any block;
• what the customer finally saw.

Make the record automatic and complete. A trace that depends on someone remembering to log will be missing exactly when it is needed.

Explainable by construction

Design so that the explanation is part of the answer, not reconstructed afterwards. Figures come from tools and carry their source and an as-of date, and a check blocks any answer whose source has no date. Statements about documents cite the section. Fixed wording that must appear (disclosures, risk statements) is owned by code and checked, not left to the model to remember: code picks the right wording from what the tools actually did, and when the line is the only thing missing it can append the line rather than fail the answer (chapter 34). A customer, an auditor and an engineer should all be able to see where each number came from.

People stay in charge of actions

A model may suggest; it should not act. Anything that moves money, changes an account or files a document goes through an explicit confirmation by the customer, or through a person, with the suggestion and the confirmation recorded separately. A suggested action can be a button that describes what it will do, sized by code rather than by the model, which asks for confirmation before anything happens; tell the checking model so, and let it check the button's content. When the system is unsure or a check fails, hand over to a human ("let me connect you to a person") instead of guessing.

Data handling

• Minimise. Send each model only what the task needs; mask personal data before any call.
• Know where data goes. List every service that sees customer data, what it keeps and for how long, and record who approved each one. Make the approvals part of configuration, so the system refuses to run without them.
• Use made-up data until the approvals exist. Synthetic personas let you build and test everything without real customers' details. If an outside service has not yet been approved for real data, only synthetic data goes through it.
• Let people see and delete what the system remembers about them.

Change control

A prompt edit can change behaviour as much as a code change. Treat prompts, model choices and thresholds as versioned artefacts: reviewed, tested against a fixed evaluation set before release, and recorded with each answer. Keep a written list of known shortcuts and gaps, with what would fix each one, and check that the code and the list agree. Describe each model you rely on, its intended use, how it was evaluated and where it fails, in a short model card ("Model Cards for Model Reporting", 2018), and each dataset's origin and limits in a datasheet ("Datasheets for Datasets", 2018).

Testing across customers

An answer can be right for one kind of customer and wrong for another. Test with personas that differ in the ways that matter (age, risk appetite, country, currency) and compare outcomes such as block rates and judge scores between groups, with confidence intervals (chapter 36), before concluding that a difference is real.

How many answers to review

Reviewing every answer by hand is rarely possible, so review a random sample. How big? If a fault affects a fraction p of answers, the chance that a random sample of n contains none is (1 − p)ⁿ. To be 95% sure of seeing at least one, you need (1 − p)ⁿ ≤ 0.05, that is n ≥ ln 0.05 / ln(1 − p).

Worked: for a fault in 1% of answers, n ≥ −2.996 / −0.01005 ≈ 298.1, so review 299 answers. This is the rule of three turned around: 3 / 0.01 = 300.

A floor for a prototype

Before any real customer touches the system, agree a short list of fixed rules and enforce them in code. A reasonable floor has four: no secrets in the code or the prompts; synthetic data only; no action triggered by model output alone; and every known shortcut written in a list, with a script that scans the code for shortcut markers and checks that the list and the code agree. The list is how a known gap stays a known gap instead of a surprise.

Retrieval fields help too: for each answer that read documents, record how its passages were found (vector search or a fused ranking) and, when that search fell back to something simpler, why. A fallback that nobody can see is a change in behaviour that nobody approved.

The same habits in three domains

None of this tells you what a regulator requires. It makes the questions a regulator, an auditor or a customer asks answerable, and a lawyer or compliance colleague can then decide what is required.