Build AI handbook

Build AI, from first principles to working code

A free, open-source, hands-on book about how AI works and how to build with it. It starts from school algebra, explains how neural networks and language models learn, covers the ideas behind real AI applications, and has you build them yourself in eleven labs: a first model call, a prompt test bench, retrieval with citations, an agent, an MCP server, guardrails, a judge, tracing and a small assistant.

Who this is for

Anyone who wants hands-on experience building AI. You need no background in AI or advanced maths: Part I starts from equations you met at school and works every idea with real numbers before any symbols. For the labs you need to be able to run a Python script; Lab 0 sets up everything else. Each lab runs without an account or a key on a built-in fake model, on a free local model through Ollama, or on Claude with your own key.

How to use this book

The book is free and needs no account. You can read it in three ways, and the pages are the same in each:

  • Online at handbook.aggarwal.dev, on a computer or a phone. The site sets no cookies, runs no analytics and loads nothing from other sites.
  • Offline, from a copy. Clone the repository, or download it as a ZIP, and double-click book/index.html. There is no build step and no network access: every page, diagram and widget is plain HTML, CSS and JavaScript in book/.
  • Served on your own machine. Run python3 tools/serve.py from the repository root and open http://127.0.0.1:8765/. Progress tracking then works in every browser. Chapter 1 says what the server does and refuses.

Each of the three keeps its own progress; chapter 1 compares them and says how to move your progress from one to another.

The book is open source. Its pages, labs and tools are at github.com/aggarwaldev/build-ai-handbook. The text is under CC BY 4.0 and the code under the MIT License, so you may share and adapt both with credit (how to credit it). Corrections and contributions are welcome, from a typo to a new lab: Help improve the book says where to start.

The labs run on your own computer once Lab 0 has set it up, or in the browser with nothing installed: every lab page has buttons to open it in Colab, Binder or Codespaces.

Inside the book:

  • Navigation. The contents are on the left (behind the menu button on a phone). Every page ends with previous and next links. Press / to search chapter titles, headings and glossary terms.
  • Dotted words are glossary terms, like token. Hover or focus one to see a short definition; click it for the full entry.
  • Boxes. A green Why it matters box says what problem an idea solves. An amber Careful box marks a common mistake. A blue Note is a side remark. A teal Build it box sends you to the lab where you build the idea yourself.
  • Interactive pieces carry a Try it label. Move the sliders and type into the boxes; nothing is sent anywhere.
  • Self-checks close every chapter. Answer in your head, then click to open the answer.
  • Ask AI. Select any passage and press Ask AI to ask Claude about it, in a new tab, or in the page with your own Anthropic API key. Chapter 1 explains how.

Reading paths

Read straight through if you like a firm foundation. Or pick a path: each one links back to the chapters it needs, so you can fill gaps as you meet them.

Build now

Start with working code and read the concepts each lab points to.

  1. Lab 0: set up your workbench: Python, a virtual environment, and a backend (fake, Ollama or Claude).
  2. Lab 1: your first model call, with tokens and sampling and prompt engineering.
  3. Lab 2: a prompt test bench, with prompt engineering and evaluating LLM output.
  4. Lab 3: RAG from scratch, with embeddings and retrieval and RAG.
  5. Lab 4: measure your retrieval, with vector search internals and evaluating RAG.
  6. Lab 5: tools and an agent loop, with tools and function calling and agents.
Foundations

How the machinery works inside, in order. Do every Try it; they are short.

  1. Part I, Maths: chapters 2 to 9, the maths everything later needs.
  2. Part II, ML foundations: chapters 10 to 14, then train your own neural network (chapter 15).
  3. Part III, How LLMs work: chapters 16 to 19, then train a tiny language model (chapter 20).
Shipping

What it takes to run an AI system other people rely on.

  1. Chapters 33 to 37: harness engineering, guardrails, security, hallucination and memory.
  2. Chapters 38 to 43: frameworks, quantisation, fine-tuning, production and regulated domains.
  3. Chapters 44 and 45: evaluating output with judges, and observability, tracing and cost.
  4. Lab 7, Lab 8 and Lab 9: guardrails, a judge and tracing you can measure. Then the capstone.

Your progress

The book keeps track of what you have read and built, in this browser only: no account, no cookies, nothing sent anywhere.

  • Every chapter and lab has a Mark this chapter done box and a My notes box above its previous and next links. Notes save as you type.
  • Each lab ends its steps with a checklist. Tick the steps as you finish them; when every step is ticked, the lab is marked done.
  • The sidebar shows a tick beside pages you have finished, a dot beside pages in progress, and how many pages of each part are done.
  • Your progress shows the whole picture, where to continue, your notes and recent activity, and lets you export your progress to a file, import it on another device, or reset it.

Contents

Start here

Chapter 1How to read this bookWhere to read it, the parts, the boxes and widgets, progress tracking, Ask AI, reusing the book and helping to improve it. Start hereYour progressWhat you have read, built and noted, with export, import and reset.

Part I: Maths

A self-contained maths track. Worked numbers first, then the symbols, then an exercise with its answer.

Chapter 2Algebra, functions and graphsVariables, equations, functions as machines, straight lines and slope. Chapter 3Exponentials and logarithmsPowers, the number e, growth, logs as the inverse, and why AI uses logs. Chapter 4Vectors and matricesArrows and lists of numbers, dot products, and matrix multiplication by hand. Chapter 5Derivatives and gradientsSlope at a point, the chain rule, partial derivatives and the gradient. Chapter 6Gradient descentWalking downhill on a loss curve, and how the learning rate matters. Chapter 7Probability and BayesDistributions, conditional probability and Bayes' rule with counts. Chapter 8Mean, variance and normalisationCentre, spread, standard scores and why networks normalise. Chapter 9Softmax, cross-entropy and KL divergenceTurning scores into probabilities and measuring how wrong they are.

Part II: ML foundations

Chapter 10What learning meansLinear regression from scratch, loss functions, logistic regression. Chapter 11GeneralisationTrain, validation and test sets; overfitting; regularisation. Chapter 12Neural networks and backpropagationNeurons, activations, backprop with real numbers, and a live trainer. Chapter 13Training deep networks wellInitialisation, SGD, momentum, Adam, normalisation layers, dropout, residuals. Chapter 14From perceptron to transformerEach bottleneck and the idea that removed it. Chapter 15 · ProjectTrain your own neural networkA NumPy-only network you write and train yourself.

Part III: How LLMs work and are trained

Chapter 16Language models, tokens and samplingNext-token prediction, BPE worked by hand, temperature, context windows. Chapter 17The embedding layer and positionsFrom token ids to vectors, and how order gets in. Chapter 18The transformer blockAttention with real numbers, multi-head, the MLP, residuals, layer norm. Chapter 19How LLMs are trainedThe objective, data, scaling and the 6ND rule, SFT, RLHF, DPO, evaluation. Chapter 20 · ProjectTrain your own tiny language modelA bigram model in pure Python, then a tiny character transformer.

Part IV: Applied concepts

Chapter 21Prompt engineeringRoles, instructions, examples, delimiters, templates, cache-friendly order and injection-resistant prompts. Chapter 22Context engineering and choosing modelsWhat goes in the window and in what order, token budgets, and picking a model on cost, speed and quality. Chapter 23Embeddings and similarityFrom token vectors to one text vector, cosine similarity, and choosing an embedding model. Chapter 24Retrieval and RAGChunking, putting passages in the prompt, citations, and how RAG fails. Chapter 25Vector search internalsExact search cost, HNSW, IVF, BM25 worked by hand, hybrid search and re-ranking. Chapter 26Evaluating RAGPrecision@k, recall@k, MRR, nDCG, groundedness and golden sets. Chapter 27The knowledge base and maintaining itDocuments, official texts, figures and news: provenance, licences and keeping them current. Chapter 28Tools and function callingSchemas, the tool loop and its cost, structured output, and calculators that keep the numbers in code. Chapter 29Agents and agentic flowReAct, planning, graphs versus free agents, stopping conditions. Chapter 30MCP, basic and advancedThe Model Context Protocol from handshake to security, with a tiny server you can run. Chapter 31SkillsA folder with SKILL.md that an agent loads on demand, with a worked example. Chapter 32SubagentsDelegating to workers with their own context, and writing a good brief. Chapter 33Harness engineeringEverything around a model that turns it into a working agent or app. Chapter 34GuardrailsInput and output guards, required wording owned by code, and measuring a guard's precision and recall. Chapter 35LLM securityInjection, jailbreaks, leakage, excessive agency, cost abuse, supply chain. Chapter 36HallucinationWhy it happens, how to measure it with intervals, and how to reduce it. Chapter 37MemoryConversation history and its cost, long-term facts, recall, and giving people control. Chapter 38Frameworks and model providersThe layers, gateways and routing, graphs for multi-step flows, and running open models yourself. Chapter 39QuantisationNumber formats, scale and zero point, GPTQ, AWQ, GGUF and QLoRA. Chapter 40Fine-tuning and distillationFull versus LoRA, the W + BA maths, soft targets and the KL loss. Chapter 41 · ProjectsLoRA fine-tune and distillationHands-on with small open models, with hardware notes. Chapter 42Running AI in productionLatency, cost per answer, prompt caching, retries, rate limits and budgets that cannot be overrun. Chapter 43AI in a regulated domainTraceable answers, people in charge of actions, data handling and change control. Chapter 44Evaluating LLM output: judges and agreementTest sets, deterministic checks, a model as judge, Cohen's kappa against people, and judge biases. Chapter 45Observability, tracing and costWhat to record per call, traces and spans, latency percentiles, cost from tokens, and alerts.

Part V: Build it

Runnable labs in labs/. Each one says what it costs on Claude; Ollama and the built-in fake model cost nothing.

Lab 0Set up your workbenchPython, a virtual environment, the requirements, and choosing a backend: fake, Ollama or Claude. Lab 1Your first model callOne message, a system prompt and a conversation, and every field of the reply: tokens, stop reason, time and cost. Lab 2A prompt test benchRun prompt versions over labelled cases, score every reply and compare runs case by case. Lab 3RAG from scratchChunk a small set of help pages, embed them, retrieve, and answer with citations that code checks. Lab 4Measure your retrievalHit@k, recall@k, MRR and nDCG for TF-IDF, BM25 and their fusion, with an honest interval. Lab 5Tools and an agent loopA small agent with a catalogue search, exact arithmetic, dates and an order tool, and the loop that runs them. Lab 6Build an MCP serverTools, a resource and a prompt over stdio, seen through the official client and on the raw wire. Lab 7Guardrails you can testMask personal data, flag instructions hidden in passages, check output, and measure each guard on labelled data. Lab 8An LLM judge you can trustA judge for a help desk's answers, tested against human labels and planted defects. Lab 9Tracing and costTraces and spans for every model and tool call, with time and cost added up per run. Lab 10Capstone: a small assistantThe pieces from the earlier labs, put together into one small assistant you can test and measure.

Reference

ReferenceGlossaryEvery term the book uses, in plain words. ReferenceResourcesFree videos, articles, docs and papers for every chapter.
A note on accuracy

Model names, prices and APIs change quickly. Every number the labs print comes from running their code, and each lab says which model and prices it assumed. If something here has gone out of date, the code in labs/ and the provider's own documentation win. Chapter 1 says how to report it or fix it.