{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Module 30: Production Evals" ], "id": "2b820984-7a5e-4678-a5e7-8a7550b1e59c" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "[Module 17](../m17_evaluation/lesson.qmd) measured *models* on\n", "*benchmarks*: exact match, pass@k, judges, leaderboards, error bars. A\n", "product team asks a different question. Its system is a model wrapped in\n", "prompts, retrieval, tools and code, serving one kind of user with one\n", "kind of request, and the question is: **is this system doing what we\n", "meant, on the inputs we actually get, and did the change we just made\n", "make it better or worse?** No public benchmark answers that. The team\n", "has to build the instrument itself. That instrument is called an\n", "**eval**.\n", "\n", "A common history repeats across production write-ups. A team gets a\n", "working prototype quickly. A conversational assistant for job seekers,\n", "for example, reached about 80% of its target experience in the first\n", "month, then took four more months to get past 95%. In the second phase\n", "the fixes become whack-a-mole: a prompt change repairs one complaint and\n", "silently breaks another, the prompt grows a paragraph per edge case, and\n", "nobody can say whether last week’s release was an improvement. The model\n", "did not stop improving. The team ran out of ways to *see*.\n", "\n", "This module builds that sight from first principles. It starts with the\n", "smallest unit (reading one trace and writing down what went wrong) and\n", "ends with the full loop that lets a system improve from its own\n", "production traffic.\n", "\n", "### What You’ll Learn\n", "\n", "After this module, you can:\n", "\n", "- State the **mental model**: an eval is an instrument built for one\n", " product, and the product improves only as fast as the instrument lets\n", " you see, explain and verify a change.\n", "- Run **error analysis**: read traces, code failures, rank failure\n", " modes, and know when to stop reading.\n", "- **Define correctness before tuning**, including for labels where\n", " errors differ in cost.\n", "- Replace one score with an **instrument panel** that says *what*\n", " regressed.\n", "- Choose a grader from the **grader ladder** and validate a **model\n", " judge** as a classifier, then **correct** the pass rate it reports.\n", "- Build **ground truth** five ways, from real failures to label-free\n", " pooling.\n", "- **Audit** an eval for the bugs that make a score mean nothing.\n", "- Explain how these pieces combine for **agents**, **multi-agent\n", " systems** and **long-running harnesses**, and run a simulation of the\n", " **improvement loop** that ties them together.\n", "\n", "### Prerequisites\n", "\n", "- [Module 17: Evaluation](../m17_evaluation/lesson.qmd): error bars,\n", " paired differences, LLM-as-judge bias.\n", "- [Module 18: Tool Use & Agents](../m18_agents/lesson.qmd): the agent\n", " loop and structured tool calls.\n", "- [Module 29: Agentic Infrastructure](../m29_agentic_infra/lesson.qmd):\n", " spans and traces, and the infrastructure layers whose failures this\n", " module measures." ], "id": "5f3aa31b-4075-4b58-8c95-55dad2dbd158" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Intuition: Three Capabilities, One Loop\n", "\n", "Every team that ships an LLM product needs three capabilities:\n", "\n", "1. **Measure** quality: a repeatable way to say how good the system is.\n", "2. **Debug**: a way to look at what the system did on a given input and\n", " why.\n", "3. **Change** the system: edit prompts, code, retrieval, tools, or\n", " fine-tune.\n", "\n", "Most teams invest almost entirely in the third. Changing a prompt is\n", "easy and feels like progress. Without the first two, though, each change\n", "is a guess judged by a glance at a few outputs (a “vibe check”). Vibe\n", "checks are useful early and never stop being useful. They stop being\n", "*sufficient* once the easy failures are gone, because the remaining\n", "failures are rare, scattered, and easy to reintroduce.\n", "\n", "Step through the loop. Each step names what stalls when it is missing." ], "id": "0457c758-6f06-45fb-ab1a-d7b9eddcb153" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Key Insight**\n", ">\n", "> The rate at which an LLM product improves is set by the slowest arm of\n", "> the loop. A team that can change the system in minutes but needs a day\n", "> to know whether the change helped improves at the speed of a day.\n", "\n", "### Vocabulary\n", "\n", "Production eval teams have converged on a small vocabulary. [Module\n", "29](../m29_agentic_infra/lesson.qmd#observability-traces-in-eval-data-out)\n", "built the layer that records it as spans. A **task** is one input with\n", "success criteria. A **trial** is one attempt at it. A **grader** scores\n", "some aspect of a trial. The **transcript** (or trace) records everything\n", "the system did. The **outcome** is the final state of the world, not the\n", "system’s claim about it. An **eval harness** runs tasks and grades them.\n", "It is not the same thing as the **agent harness**, the code that turns a\n", "model into an agent. An eval of an agent always measures the model and\n", "its harness together.\n", "\n", "Two kinds of suite behave differently. A **capability** suite asks “can\n", "it do this yet?” and starts with a low pass rate. A **regression** suite\n", "asks “did we break anything?” and should stay near 100%. When a\n", "capability suite saturates, its tasks graduate into the regression\n", "suite.\n", "\n", "## Principle 1: Look at the Data First\n", "\n", "The failure categories that matter for a product are not in any generic\n", "metric list. They are in the product’s traces. The first eval activity\n", "is not writing a metric. It is **reading**.\n", "\n", "The procedure has four steps:\n", "\n", "1. **Sample** traces from real or realistic traffic.\n", "2. **Open-code**: for each trace, write one short note on the *first*\n", " thing that went wrong. The first error matters most, because later\n", " errors are often consequences of it. Mark the trace pass or fail.\n", " Binary verdicts come before any 1–5 scale. A scale invites\n", " disagreement about what a 3 means and hides the decision you\n", " actually need, which is ship or fix.\n", "3. **Axial-code**: group the notes into a small set of named **failure\n", " modes**.\n", "4. **Count and rank.** Fix the largest mode first.\n", "\n", "Keep reading in batches until new batches stop producing new modes. That\n", "point is called **saturation**. Reading never stops entirely, because\n", "traffic changes. But saturation tells you when another batch is unlikely\n", "to teach you anything new about the *current* distribution.\n", "\n", "Friction decides whether this happens at all. Teams that read traces\n", "routinely almost always build a small purpose-made viewer. It shows the\n", "input, the output, the retrieved context and the tool calls on one\n", "screen, with a pass/fail button and a notes field. A generic log search\n", "tool is rarely enough." ], "id": "b09396a3-f927-4394-b27e-00ac1a0e861e" }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "wrong-date-range 9 36% cum 36%\n", "missed-constraint 6 24% cum 60%\n", "hallucinated-attendee 4 16% cum 76%\n", "tool-error-hidden 3 12% cum 88%\n", "tone 2 8% cum 96%\n", "leaked-internal-id 1 4% cum 100%\n", "new modes per batch: [4, 1, 0, 0, 1, 0]" ] } ], "source": [ "from errors import build_taxonomy, demo_traces, pareto, saturation_curve\n", "\n", "traces = demo_traces() # 60 reviewed traces from a scheduling assistant\n", "for r in pareto(build_taxonomy(traces)):\n", " print(f\"{r.mode:22s}{r.count:3d} {r.share:5.0%} cum {r.cumulative:5.0%}\")\n", "print(\"new modes per batch:\", saturation_curve(traces, 10))" ], "id": "c236dced" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The top two modes account for 60% of the failures. A team that had been\n", "tuning tone, because a stakeholder forwarded one curt reply, was working\n", "on 8% of the problem.\n", "\n", "> **Try This**\n", ">\n", "> In the figure, set the batch slider to 5 and then 20. A small batch\n", "> makes the saturation curve noisy, and a late new mode can arrive after\n", "> several zeros. That is why “stop at saturation” means “stop for now”.\n", "\n", "## Principle 2: Define Correctness Before Tuning\n", "\n", "Eyeballing outputs and editing prompts builds intuition, but it does not\n", "build anything you can repeat. If you change the prompt before deciding\n", "what “right” means, every result looks like progress. Write the grader\n", "first.\n", "\n", "Exact match is the simplest definition, and it is often too blunt.\n", "Consider a classifier that maps free-text product descriptions onto a\n", "taxonomy with about 10,000 leaves, eight levels deep. Getting the\n", "top-level department wrong (“Electronics” for a pair of boots) is far\n", "worse than getting the eighth-level leaf wrong. Exact match scores both\n", "as 0.\n", "\n", "A **hierarchical score** charges for the *first* level that goes wrong,\n", "with a weight that shrinks with depth:\n", "\n", "$$\n", "\\text{score} = 1 - w_{\\ell^\\ast}, \\qquad\n", "w = (1.0,\\ 0.7,\\ 0.5,\\ 0.4,\\ 0.3,\\ 0.2,\\ 0.15,\\ 0.1),\n", "$$\n", "\n", "where $\\ell^\\ast$ is the first mismatched level. A wrong root scores 0.\n", "A wrong eighth level scores 0.9." ], "id": "3a5b3873-bccf-4513-bee4-16d435e59012" }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "1.00 root=1 Apparel > Shoes > Boots > Ankle boots\n", "0.60 root=1 Apparel > Shoes > Boots > Chelsea boots\n", "0.50 root=1 Apparel > Shoes > Sandals > Slides\n", "0.00 root=0 Electronics > Phones > Cases > Folio" ] } ], "source": [ "from graders import hierarchical_score, root_only\n", "\n", "gold = [\"Apparel\", \"Shoes\", \"Boots\", \"Ankle boots\"]\n", "for pred in (gold,\n", " [\"Apparel\", \"Shoes\", \"Boots\", \"Chelsea boots\"],\n", " [\"Apparel\", \"Shoes\", \"Sandals\", \"Slides\"],\n", " [\"Electronics\", \"Phones\", \"Cases\", \"Folio\"]):\n", " score, root = hierarchical_score(pred, gold), root_only(pred, gold)\n", " print(f\"{score:.2f} root={root:.0f} {' > '.join(pred)}\")" ], "id": "e4f17881" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The story continues past the scorer, and the continuation is the reason\n", "to build more than one evaluator. Most errors were deep and cheap. A\n", "second evaluator that scored **only the root level** isolated the\n", "expensive ones. Filtering failures by it showed that many were not\n", "product descriptions at all. They were audience-targeting labels that\n", "only made sense to the person who wrote them, such as\n", "“purchase-prediction segment, shopping, 90 days”. The model was forcing\n", "a product category onto them. No prompt fix was right. The fix was\n", "architectural: detect this input class and label it “uncategorizable”." ], "id": "b7c9a500-df33-4e8a-b18f-76659201db4a" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Key Insight**\n", ">\n", "> A second evaluator is a lens, not a duplicate. Each grader you add\n", "> lets you filter failures by a different property, and filtering is how\n", "> an undiscovered input class becomes visible.\n", "\n", "## Principle 3: One Score Says *That*, a Panel Says *What*\n", "\n", "A single aggregate score is like a film’s percentage on a review\n", "aggregator. It tells you how many critics liked the film, not whether it\n", "is funny or frightening or both. When one eval score drops from 0.87 to\n", "0.81 after a prompt change, the team knows that something regressed. It\n", "does not know what, and the usual response is an hour of re-running the\n", "pipeline and guessing.\n", "\n", "The fix is to score **dimensions**, each on a simple scale, and to score\n", "**each LLM call** in the pipeline separately (“shrink-wrapping” the\n", "call). A pipeline-level score alone is an integration test with no unit\n", "tests. A support-ticket summarizer, for example, produces a category, a\n", "priority and a draft reply. It gets three graders, not one:\n", "\n", "| Dimension | Grader | Scale |\n", "|----------------------|------------------------------|-------------|\n", "| Category accuracy | exact match against a label | 0 / 1 |\n", "| Priority calibration | within one step of the label | 0 / 0.5 / 1 |\n", "| Reply relevance | model judge with a rubric | 0 / 1 |\n", "\n", "The panel below runs a four-dimension pipeline twice. The second run\n", "degrades one LLM call." ], "id": "cce02f57-de13-452e-8591-a5eef776c8f9" }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "aggregate: 0.925 -> 0.885\n", " citation grounding 0.88 -> 0.72 (-0.16)\n", " extraction precision 0.94 -> 0.94 (+0.00)\n", " policy compliance 0.97 -> 0.97 (+0.00)\n", " tone 0.91 -> 0.91 (+0.00)" ] } ], "source": [ "from graders import demo_panel_runs\n", "\n", "runs = demo_panel_runs()\n", "a, b = runs[\"aggregate_before\"], runs[\"aggregate_after\"]\n", "print(f\"aggregate: {a:.3f} -> {b:.3f}\")\n", "for dim, delta in runs[\"diff\"].items():\n", " before, after = runs[\"before\"][dim], runs[\"after\"][dim]\n", " print(f\" {dim:22s} {before:.2f} -> {after:.2f} ({delta:+.2f})\")" ], "id": "149115da" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Two side effects make the panel worth its cost. First, writing a\n", "dimension forces the team to define it. A team cannot write a “realism”\n", "grader without agreeing on what realism means, so building evals doubles\n", "as requirements discovery. Second, the panel separates *evals* from\n", "*validation*. An eval is an instrument you read to make a decision. A\n", "validation check runs on every production response and acts on it (see\n", "[Runtime validation](#runtime-validation-is-not-an-eval)).\n", "\n", "## Principle 4: The Grader Ladder\n", "\n", "Graders come in three families, ordered by cost:" ], "id": "eb8a0d1c-c0c4-4d92-a10b-4d3f5eee1d52" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. **Deterministic** checks: schema validation, exact or tolerant\n", " numeric match, regular expressions, set membership, checks on the\n", " environment’s final state, checks on which tools were called. They\n", " are fast, free and reproducible. Their weakness is brittleness. Use\n", " them for everything they can express. A single generic assertion can\n", " cover a whole class of failures, for example a regex that fails any\n", " reply that shows the user a raw internal ID.\n", "2. **Model-based** graders: a rubric, a natural-language assertion\n", " (“the reply does not promise a refund”), a pairwise comparison, or a\n", " comparison against a reference answer. They handle nuance. They are\n", " non-deterministic, they cost money, and they must be validated (next\n", " section).\n", "3. **Human** review: expert grading, spot checks, A/B tests with real\n", " users. Humans are the reference that calibrates everything else.\n", " They are also the slowest and most expensive rung.\n", "\n", "Run each rung at the cadence its cost allows. Assertions run on every\n", "change. Judges and human review run on a schedule. A/B tests run after\n", "significant changes to a mature product." ], "id": "fdbc8375-3251-4f40-8a51-5c6eeefadbd1" }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "schema fail wrong type: priority\n", "no-internal-ids fail found '3f2b1c9a-1d2e-4f5a-8b9c-0d1e2f3a4b5c'\n", "exact fail '96.12' != '96.124991'\n", "numeric_close pass " ] } ], "source": [ "from graders import exact, numeric_close, regex_absent, schema_check\n", "\n", "UUID = r\"[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}\"\n", "schema = {\"category\": str, \"priority\": int}\n", "reply = \"Booked: 3f2b1c9a-1d2e-4f5a-8b9c-0d1e2f3a4b5c\"\n", "checks = [schema_check({\"category\": \"billing\", \"priority\": \"high\"}, schema),\n", " regex_absent(reply, UUID, name=\"no-internal-ids\"),\n", " exact(\"96.12\", \"96.124991\"),\n", " numeric_close(\"96.12\", 96.124991, rel=1e-3)]\n", "for r in checks:\n", " print(f\"{r.name:16s} {r.verdict.value:5s} {r.detail}\")" ], "id": "b123c299" }, { "cell_type": "markdown", "metadata": {}, "source": [ "A pass rate is a **product decision**, not automatically 100%. A team\n", "decides which failures it can tolerate, at what rate, and writes that\n", "threshold down.\n", "\n", "## Principle 4, continued: A Judge Is a Classifier\n", "\n", "A model judge reads an output and returns pass or fail. Before you trust\n", "its verdicts you have to know how often it is right, which means\n", "treating the judge like any other binary classifier.\n", "\n", "The procedure that works in practice:\n", "\n", "1. A domain expert labels a sample of outputs pass/fail and writes a\n", " short **critique** explaining each verdict.\n", "2. The judge prompt is written from those critiques. A few labelled\n", " examples go into the prompt as demonstrations.\n", "3. The remaining labels are split into a **dev** set, used to iterate\n", " on the prompt, and a held-out **test** set, used once to report the\n", " judge’s accuracy.\n", "4. Report the **true-positive rate** (of outputs the expert passed, the\n", " share the judge passed) and the **true-negative rate** (of outputs\n", " the expert failed, the share the judge failed) separately.\n", "\n", "Step 4 is where most teams go wrong. Most production outputs pass, so a\n", "judge that passes everything has high raw agreement with the expert and\n", "catches nothing." ], "id": "3225eabf-afe2-4218-a5a0-37d3cda048f9" }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "raw agreement 90%\n", "TPR 1.00 TNR 0.00 kappa 0.00" ] } ], "source": [ "from judge import confusion, cohen_kappa, raw_agreement, tpr_tnr\n", "\n", "human = [True] * 90 + [False] * 10 # 90% of outputs are good\n", "lazy = [True] * 100 # a judge that always says PASS\n", "c = confusion(lazy, human)\n", "tpr, tnr = tpr_tnr(c)\n", "print(f\"raw agreement {raw_agreement(lazy, human):.0%}\")\n", "print(f\"TPR {tpr:.2f} TNR {tnr:.2f} kappa {cohen_kappa(c):.2f}\")" ], "id": "4aee497a" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Further practices: use one judge per dimension rather than one judge for\n", "everything; let the judge answer “unknown” rather than forcing a guess;\n", "prefer binary verdicts. Iterating on a judge prompt with a single expert\n", "over three rounds has been reported to reach more than 90% agreement on\n", "a balanced set. Without a separate evaluator, a system grading its own\n", "work is lenient. A separate judge tuned toward skepticism is easier to\n", "fix than a generator asked to criticize itself.\n", "\n", "A judge also has known biases. It tends to prefer the first of two\n", "answers it is shown (**position**), the longer answer (**verbosity**)\n", "and answers written in its own style (**self-enhancement**), and it can\n", "fail on reasoning it cannot do itself. Strong judges agree with human\n", "raters more than 80% of the time, about as often as humans agree with\n", "each other. That figure is a ceiling for a well-validated judge, not a\n", "default for an unvalidated one. Running a pairwise comparison in both\n", "orders, and counting a win only when both verdicts agree, controls the\n", "position effect at twice the cost.\n", "\n", "### The Math: Correcting the Pass Rate a Judge Reports\n", "\n", "Suppose the judge has measured rates TPR and TNR, and on unlabelled\n", "production traffic it passes a share $q$ of outputs. The true pass rate\n", "$p$ is unknown. Each output is either truly good (probability $p$), in\n", "which case the judge passes it with probability TPR, or truly bad, in\n", "which case the judge passes it with probability $1 - \\text{TNR}$:\n", "\n", "$$\n", "q = \\text{TPR}\\cdot p + (1 - \\text{TNR})(1 - p).\n", "$$\n", "\n", "Solve for $p$:\n", "\n", "$$\n", "\\hat p = \\frac{q + \\text{TNR} - 1}{\\text{TPR} + \\text{TNR} - 1}.\n", "$$\n", "\n", "The denominator $\\text{TPR} + \\text{TNR} - 1$ measures how much\n", "information the judge carries. When it is 0, the verdict is independent\n", "of the truth and no correction is possible. When it is small, small\n", "errors in TPR, TNR or $q$ become large errors in $\\hat p$. The error bar\n", "comes from resampling both the labelled split (uncertainty in TPR and\n", "TNR) and the production verdicts (uncertainty in $q$)." ], "id": "94e97b2d-2f24-485f-a006-d8c29bbc6a0c" }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "TPR 0.95 TNR 0.64 (on 200 labels)\n", "judge reports 77.9%\n", "corrected 70.8% 95% CI 60.9%-78.9%\n", "true rate 68.7%" ] } ], "source": [ "from judge import demo_judge\n", "\n", "d = demo_judge()\n", "print(f\"TPR {d['tpr']:.2f} TNR {d['tnr']:.2f} (on 200 labels)\")\n", "print(f\"judge reports {d['q']:.1%}\")\n", "print(f\"corrected {d['point']:.1%} 95% CI {d['lo']:.1%}-{d['hi']:.1%}\")\n", "print(f\"true rate {d['true_production_rate']:.1%}\")" ], "id": "2e9746e6" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Try This**\n", ">\n", "> Set TNR to 0.55 and TPR to 0.5. The interval explodes, because the\n", "> judge barely separates good from bad. Then raise the number of labels.\n", "> More labels narrow the interval only up to a point. A judge near\n", "> chance cannot be rescued by measuring it more precisely.\n", "\n", "## Where Ground Truth Comes From\n", "\n", "A grader compares an output with something. Labels are the expensive\n", "part of every eval. Five sources, ordered from most to least direct:\n", "\n", "1. **Real failures.** 20–50 tasks drawn from actual problems are enough\n", " to start. Early changes tend to have large effects. One research\n", " assistant’s success rate went from 30% to 80% on about 20 queries\n", " after one prompt change, so a small set detects them.\n", "2. **Historical records.** A closed ticket paired with the merged code\n", " change that resolved it is a solved example with no annotation\n", " needed. One team built several hundred labelled cases this way for a\n", " system that suggests API endpoints for integration requests. Each\n", " case asked two questions: are the needed endpoints in the top five\n", " suggestions, and is the most-needed one ranked first?\n", "3. **Synthetic by construction.** Generate the facts first, then render\n", " them into documents. The label set is exhaustive because it existed\n", " before the document did. Render the same facts into many layouts,\n", " then add degradation: character noise at the levels of a clean scan,\n", " a light fax and a heavy fax, or a column header that appears only on\n", " page one. Synthetic *user inputs* work the same way: enumerate\n", " features × scenarios (a search that returns zero, one or many\n", " results) and generate inputs for each cell.\n", "4. **Partial expert labels.** Experts label only the facts that matter.\n", " This measures recall on what matters. It cannot measure precision,\n", " because unlabelled facts might be correct.\n", "5. **No labels.** Run several systems, pool their outputs and treat the\n", " union as pseudo-truth. This gives a *relative* ranking only. Facts\n", " that only one system produced are **singletons** and need review. A\n", " system that produces many singletons without winning on recall is\n", " probably hallucinating.\n", "\n", "### Healthy proxies, broken system\n", "\n", "For extraction from scanned documents, **recall** is the share of the\n", "page’s facts the system captured, and **precision** is the share of\n", "extracted facts the page supports. Both need exhaustive labels. One team\n", "reported an extractor that, on a badly degraded page, returned about\n", "forty well-typed, quoted facts per page. Its true recall was 9%. The\n", "values were built from misread characters. Fact counts, schema\n", "conformance and quote coverage all looked healthy. Only hand labels\n", "showed the problem.\n", "\n", "The sweep below reproduces the shape with a facts-first synthetic\n", "generator and a layout-parsing extractor." ], "id": "64af99f7-3965-4e0f-aae0-cf3a8dc0eb2b" }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " CER facts/page schema precision recall\n", " 0.0% 12.0 100% 100% 100%\n", " 0.7% 11.8 100% 94% 92%\n", " 4.5% 10.8 100% 55% 49%\n", "13.3% 9.3 100% 18% 14%\n", "25.0% 8.5 90% 5% 3%" ] } ], "source": [ "from extraction import degradation_sweep\n", "\n", "print(\" CER facts/page schema precision recall\")\n", "for r in degradation_sweep():\n", " cer, n, ok = r[\"cer\"], r[\"facts_per_page\"], r[\"schema_rate\"]\n", " p, rec = r[\"precision\"], r[\"recall\"]\n", " print(f\"{cer:5.1%} {n:9.1f} {ok:6.0%} {p:9.0%} {rec:6.0%}\")" ], "id": "595adfb8" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At a heavy-fax error rate the extractor still returns three quarters as\n", "many facts as on a clean page, and every one of them is schema-valid.\n", "Recall is 14%." ], "id": "fbeb11ed-30db-4286-a7f6-c5065471f74d" }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " pooled singletons | true recall true precision\n", "A: 67% 3 | 83% 100%\n", "B: 47% 0 | 58% 100%\n", "C: 73% 5 | 50% 55%" ] } ], "source": [ "from extraction import demo_pooling\n", "\n", "p = demo_pooling()\n", "print(\" pooled singletons | true recall true precision\")\n", "for k in \"ABC\":\n", " print(f\"{k}: {p['pooled_recall'][k]:6.0%} {p['singletons'][k]:10d} | \"\n", " f\"{p['true_recall'][k]:11.0%} {p['true_precision'][k]:14.0%}\")" ], "id": "d99cbc9a" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pooling ranks C first because pooling rewards volume. C’s singleton\n", "count is the warning. Its extra facts are inventions, and its true\n", "recall is the lowest of the three.\n", "\n", "> **A high score can hide the failure that matters**\n", ">\n", "> A 95% recall says nothing about *which* 5% is missing. If the missing\n", "> facts are the allergy and the dose, the extractor is unusable. Partial\n", "> expert labels on the critical facts exist to catch exactly this case.\n", "\n", "## Runtime Validation Is Not an Eval\n", "\n", "Some checks run on **every production response** and act on the result:\n", "they reject, retry, repair or escalate. They borrow eval machinery but\n", "play a different role.\n", "\n", "- **Verbatim quotes.** Each extracted fact must carry a quote that\n", " occurs on the page and contains the value. Facts whose quote matches\n", " nothing are discarded. This is an upper bound on hallucination, not a\n", " measurement of it, because OCR noise and line breaks also break\n", " matches.\n", "- **Tolerant parsing.** One assistant found about 10% of its structured\n", " responses malformed. Re-prompting would have added latency and load. A\n", " parser that recognized and repaired the common mistakes, plus hints in\n", " the prompt about those mistakes, brought the rate to about 0.01%.\n", "- **Typed boundaries.** In a multi-agent workflow, every message between\n", " agents and every tool call is validated against a schema, and an agent\n", " may return only one of a fixed set of actions (request more\n", " information, assign, close as duplicate, no action). A violation is a\n", " contract failure: retry, repair or escalate before bad state\n", " propagates." ], "id": "04ccb564-1d8f-4868-b814-dc5e30688d55" }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "kept ['5 mg'] | rejected ['iv']" ] } ], "source": [ "from extraction import Fact, quote_validate\n", "\n", "page = \"DOSE: 5 mg\\nROUTE: oral\"\n", "kept, rejected = quote_validate([Fact(\"dose\", \"5 mg\", \"DOSE: 5 mg\"),\n", " Fact(\"route\", \"iv\", \"ROUTE: iv\")], page)\n", "print(\"kept\", [f.value for f in kept], \"| rejected\", [f.value for f in rejected])" ], "id": "aa144c0c" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Principle 5: Suspect the Eval First\n", "\n", "A surprising score is a bug in the task, the grader or the harness until\n", "the transcripts show otherwise. Published audits of agent benchmarks\n", "have found all of the following:\n", "\n", "- A grader required the string `96.124991…` and rejected `96.12`. After\n", " this and similar grader fixes, plus a less constrained harness, one\n", " model’s score on the benchmark went from 42% to 95%. The model did not\n", " change.\n", "- A task penalized a system for following the task’s stated instruction.\n", "- A test assumed a file path that the task description never stated.\n", "- Trials shared state, and an agent read a previous trial’s history to\n", " solve the next one.\n", "- An agent found a loophole in a booking policy that served the user\n", " better, and the eval as written marked it failed.\n", "\n", "A 0% pass rate over many trials for a capable system usually means a\n", "broken task, not an incapable system." ], "id": "3b97710b-8b09-4fa8-959b-0ca456308c1c" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "no-ref no reference solution\n", "impossible reference solution fails its own graders\n", "trial pass rate: isolated 33%, shared state 68%" ] } ], "source": [ "from suite import Task, audit, demo_leakage\n", "\n", "tasks = [\n", " Task(\"ok\", \"2+2\", reference=\"4\", graders=[lambda o: o == \"4\"]),\n", " Task(\"no-ref\", \"summarize\", reference=None, graders=[lambda o: len(o) < 200]),\n", " # the spec never names the output path the grader expects\n", " Task(\"impossible\", \"path?\", reference=\"out.txt\", graders=[lambda o: o == \"data/out.txt\"]),\n", "]\n", "for f in audit(tasks):\n", " print(f\"{f.task_id:10s} {f.problem}\")\n", "leak = demo_leakage()\n", "print(f\"trial pass rate: isolated {leak['isolated_trial_rate']:.0%}, \"\n", " f\"shared state {leak['shared_trial_rate']:.0%}\")" ], "id": "6acfbc40" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The audit checklist:\n", "\n", "1. Every task has a reference solution, and the reference passes its\n", " own graders.\n", "2. Two domain experts reading the task would reach the same verdict.\n", "3. Trials start from a clean environment.\n", "4. Someone reads transcripts before the score is reported.\n", "5. Saturated suites are retired to regression duty and replaced.\n", "\n", "> **Explanations are not evidence**\n", ">\n", "> Asking a model to explain its answer is tempting to read as a window\n", "> into its decision. Controlled tests on classification tasks found that\n", "> adding an explanation before or after the label changed accuracy by\n", "> zero to three points and sometimes lowered it. When a hint changed the\n", "> answer, the model produced a matching justification for the new\n", "> answer. Treat an explanation as a plausible story. If you want to know\n", "> whether reasoning helps your task, measure it on your task.\n", ">\n", "> The same caution applies to unexplained gains. One team fine-tuned a\n", "> moderation classifier on generated rationales and saw a large\n", "> improvement. They then found that a bug had left the post text out of\n", "> the training prompts. A gain you cannot explain is a reason to check\n", "> the eval, not to ship.\n", "\n", "## Feedback Speed\n", "\n", "A team that waits a day for its eval signal improves at the speed of a\n", "day. Production teams typically run three loops at different speeds:" ], "id": "eceec781-3601-4fb2-9f45-a91523d3b1c2" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- **Engineer checks**: minutes, coarse, on a handful of examples.\n", "- **Annotator review**: about a day. One team scaled this to a group of\n", " linguists reviewing up to 500 conversations per day, scored on overall\n", " quality, hallucination, policy violations, coherence and style. That\n", " signal became its main indicator of production readiness.\n", "- **User feedback**: three or more days per change.\n", "\n", "Automated graders and validated judges exist to move signal from the\n", "slow loops into the fast one. Latency also belongs on the panel. One\n", "team bounded time to first token but not the time between tokens. With a\n", "200-token reasoning step, a 10 ms increase per token adds 2 seconds per\n", "response, and a public launch triggered timeout alerts.\n", "\n", "## Worked Examples: The Principles on Real Systems\n", "\n", "The principles above come from teams that shipped LLM products and wrote\n", "about what broke. This part walks through five of those systems. Each\n", "one is anonymized and, where the source gave no numbers, simulated so\n", "the figure can be stepped through. Each follows the same arc: **the\n", "system**, **what went wrong**, **which principle applied and how**, and\n", "**what changed**. The figures label any number the source did not report\n", "as illustrative." ], "id": "ff73bfa3-89c8-4eb7-8fc5-c8488168c9e5" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Worked Example 1: A Job-Search Assistant That Plateaued\n", "\n", "**The system.** A conversational assistant answers questions about jobs,\n", "companies and posts. A **router** sends each question to one of several\n", "specialized agents. A **retrieval** step calls internal search services.\n", "A **generation** step writes the answer with a larger model.\n", "\n", "**What went wrong.** The first month reached about 80% of the target\n", "experience. The next four months were spent getting past 95%. Three\n", "separate problems were hiding inside one overall number:\n", "\n", "1. **Quality plateaued.** Overall quality is the *product* of the stage\n", " pass rates, because a request succeeds only when every stage\n", " succeeds. Once the router and retrieval were good, nearly all\n", " remaining failures were in generation, mostly hallucination. That\n", " was the hardest stage to measure and fix.\n", "2. **About 10% of structured outputs were malformed.** Each agent\n", " emitted parameters for the next step in a structured format.\n", " Re-prompting on a failure would have added latency and load.\n", "3. **Responses timed out at launch.** The team had bounded time to\n", " first token but not the time between tokens. A reasoning step\n", " multiplies every per-token delay.\n", "\n", "**Applying the principles.**\n", "\n", "- *Decompose (Principle 3):* score each stage separately, so the plateau\n", " shows up as one weak stage rather than a flat overall line.\n", "- *Deterministic first (Principle 4):* a parser that recognizes and\n", " repairs the common format mistakes, plus prompt hints about those\n", " mistakes, took malformed outputs from about 10% to about 0.01% without\n", " a second model call.\n", "- *Feedback speed:* a group of linguists reviewed up to 500\n", " conversations a day on quality, hallucination, policy, coherence and\n", " style. That became the main readiness signal. Model-based\n", " hallucination detection was the first automated grader to work.\n", "- *Latency on the panel:* time between tokens was added as a tracked\n", " dimension next to time to first token." ], "id": "44cf83fb-0d0e-490c-b3b1-5a251de999ec" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Try This**\n", ">\n", "> Move from month 1 to month 5. The router gains 2.5 points and\n", "> generation gains 9, while overall quality gains 15, because it is the\n", "> product of the three stages. Generation is the weakest stage in every\n", "> month. Then set the time between tokens to 30 ms with 200 reasoning\n", "> tokens. The extra 10 ms per token adds 3.5 s per response, and the\n", "> response (about 11 s) crosses the 10 s timeout.\n", "\n", "### Worked Example 2: A CRM Assistant Stuck in Whack-a-Mole\n", "\n", "**The system.** An assistant inside a real-estate CRM finds listings,\n", "creates contacts, schedules showings and drafts emails in natural\n", "language.\n", "\n", "**What went wrong.** Prompt changes made fast early progress and then\n", "stalled. Each fix for a reported problem broke something nobody was\n", "watching, and the prompt grew a paragraph per edge case. The team\n", "checked each change on the two flows it demoed every week, so the\n", "breakages were invisible until users hit them.\n", "\n", "**Applying the principles.**\n", "\n", "- *Look at the data (Principle 1):* reading traces showed that failures\n", " clustered by **scenario** (no results, many results, an ambiguous\n", " name, a missing field) more than by feature.\n", "- *Define correctness (Principle 2):* one scoped assertion per feature ×\n", " scenario cell. For example, “when a search returns zero listings, the\n", " reply says so and offers to widen the search”. Synthetic inputs were\n", " generated for every cell. Generic assertions, such as a regex that\n", " fails any reply showing a raw internal ID, ran on every cell.\n", "- *Cadence (Principle 4):* the assertion suite ran on every change.\n", " Trace review and an aligned judge ran on a schedule. A/B tests were\n", " saved for large changes.\n", "\n", "Step through five prompt versions. A star marks the two cells the weekly\n", "demo covered." ], "id": "97b676f8-a94c-4215-8a87-d640fa3f8708" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Version 2 is the lesson. The demo passes, the targeted fix works, and\n", "the overall pass rate *falls*, because two cells nobody demoed broke.\n", "With the matrix in place, versions 4 and 5 fix exactly the red cells and\n", "cannot silently break green ones.\n", "\n", "### Worked Example 3: Aligning a Judge With One Expert\n", "\n", "**The system.** An assistant turns a plain-language question into a\n", "query in a product’s query language. Whether a query is *right* depends\n", "on whether it answers the question asked, which only a domain expert can\n", "judge reliably. No rule captures that.\n", "\n", "**What went wrong.** The first judge prompt was a generic rubric. It\n", "agreed with the expert on most outputs, and it passed almost every query\n", "that ran, including queries that answered a different question. Raw\n", "agreement looked fine because most outputs were good.\n", "\n", "**Applying the principles.**\n", "\n", "- *A judge is a classifier (Principle 4):* the expert labelled outputs\n", " pass or fail and wrote a one-paragraph critique for each. The labels\n", " were split into train (examples for the prompt), dev (for iterating)\n", " and a held-out test split. TPR and TNR were reported separately.\n", "- Each round folded the expert’s critiques into the judge prompt as\n", " explicit rules and examples. After three rounds, agreement with the\n", " expert was reported above 90%." ], "id": "e9626e88-2f90-4c18-8734-98ec7c53ac4e" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first round is the trap: raw agreement in the mid-70s, while the\n", "judge catches fewer than a third of the bad queries. Only TNR shows the\n", "problem. The rounds that follow raise TNR from 30% to 90% while TPR\n", "stays between 90% and 100%, because the critiques describe *why* a query\n", "fails.\n", "\n", "### Worked Example 4: Ground Truth From History\n", "\n", "**The system.** Engineers who build integrations receive feature\n", "requests as tickets. A recommender pre-fills each ticket with the API\n", "endpoints it will probably need, plus documentation links.\n", "\n", "**What went wrong.** Nothing yet, and that was the problem. The team\n", "could not tell whether the recommendations were good without hundreds of\n", "labelled tickets. Labelling them by hand would have cost a great deal of\n", "engineer time.\n", "\n", "**Applying the principles.**\n", "\n", "- *Where ground truth comes from:* every closed ticket that added a new\n", " capability and links to a merged code change is a solved example. The\n", " change shows which endpoints were actually used. A model extracted the\n", " endpoints from each code change, and the suite was built from those.\n", "- *Define correctness (Principle 2):* two questions per case. Are the\n", " needed endpoints in the top five suggestions? Is the most-needed\n", " endpoint ranked first?" ], "id": "003e20b2-f539-4b82-b3bb-b6ead9f86ff2" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The funnel’s stage counts are illustrative; the source reported only\n", "that the filters left “a few hundred” cases. The filters matter more\n", "than the count. Excluding *fixes* keeps only tickets where the change is\n", "the whole answer. Requiring a merged change guarantees that the “answer”\n", "was accepted by a reviewer.\n", "\n", "### Worked Example 5: A Long-Running Coding Agent That Declared Victory\n", "\n", "**The system.** An agent builds a web application over many sessions.\n", "Each session starts with an empty context window.\n", "\n", "**What went wrong.** Without structure, the agent tried to build too\n", "much at once and marked features done after a unit test or a single\n", "request. It never checked them end to end. A later session saw a mostly\n", "finished project and declared the whole job complete. The *claimed*\n", "progress and the *true* progress drifted apart, and nothing in the loop\n", "measured the gap.\n", "\n", "**Applying the principles.** The fix turns the eval into an artifact the\n", "agent reads every session. That artifact is a feature list of over 200\n", "items, each with verification steps, all initially failing:\n", "\n", "- *Define correctness first (Principle 2):* every feature has its\n", " verification steps before any code is written.\n", "- *Grade the outcome (Designs):* a feature is marked passing only after\n", " an end-to-end check in a real browser.\n", "- *Suspect the state (Principle 5):* each session starts by running the\n", " app and a smoke test, which reopens anything a later change broke." ], "id": "0ca42230-265a-4b7e-9dc9-891a1552087f" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Session counts and break rates here are simulated. The pattern, a\n", "widening claimed-versus-true gap and an early stop, is the reported one.\n", "\n", "## Designs: How the Pieces Fit\n", "\n", "Everything so far applies to a single LLM call. Agents, multi-agent\n", "systems and long-running harnesses use the same principles, combined\n", "differently.\n", "\n", "### Agents: grade the outcome\n", "\n", "An agent changes the world, so grade the **final state**: the\n", "reservation exists in the database, the test suite passes, the file is\n", "on disk. The agent’s claim that it finished is not evidence. Check the\n", "path as a supplement (it should not have deleted the database on the\n", "way), give partial credit, and add checkpoints for long workflows. Run\n", "several trials per task and report both pass@k and pass^k (next\n", "subsection): a 75% per-trial rate is only 42% reliable across three\n", "consecutive runs.\n", "\n", "When the agent retrieves before it answers, one metric separates\n", "“answered” from “supported”:\n", "\n", "$$\n", "\\text{faithfulness} = \\frac{\\#\\ \\text{claims in the response supported by the retrieved context}}{\\#\\ \\text{claims in the response}}.\n", "$$\n", "\n", "A correct answer with low faithfulness came from the model’s memory, not\n", "from the sources it cited, and it will fail when the sources and the\n", "memory disagree.\n", "\n", "No single method catches everything. Production teams layer automated\n", "suites, production monitoring, A/B tests, user feedback, transcript\n", "review and periodic human studies. Each layer catches failures the\n", "others miss.\n", "\n", "### Reliability: pass@k versus pass^k\n", "\n", "An agent’s output varies from run to run, so one trial per task measures\n", "luck. Two summaries of $k$ trials are easy to confuse and mean opposite\n", "things. **pass@k** is the probability that *at least one* of $k$\n", "attempts succeeds, and it rises with $k$. **pass^k** is the probability\n", "that *all* $k$ attempts succeed, and it falls. With $p$ the per-trial\n", "success rate and independent trials,\n", "\n", "$$\n", "\\text{pass@}k = 1 - (1 - p)^k, \\qquad \\text{pass}^k = p^k .\n", "$$\n", "\n", "pass@k answers “can it ever do this?”, the right question when a person\n", "or a verifier picks the best of $k$ attempts. pass^k answers “will it do\n", "this every time?”, the right question for an agent that runs unattended.\n", "Both equal $p$ at $k = 1$ and then move apart. At $p = 0.9$, ten\n", "consecutive successes still happen only $0.9^{10} \\approx 35\\%$ of the\n", "time. A published benchmark of tool-using agents in simulated\n", "customer-service domains found that the best function-calling agents of\n", "the time solved fewer than half of the tasks, and that pass^8 was below\n", "25% in its retail domain." ], "id": "7f326694-db2d-43bc-a593-8993d0fcff4f" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Independence is the figure’s assumption. Real failures are correlated: a\n", "task that fails for a structural reason fails every time, which makes\n", "pass@k lower than the formula says and pass^k higher.\n", "\n", "### Reading public agent benchmarks\n", "\n", "Public agent benchmarks are useful as a reference point and misleading\n", "as a product eval. Their reported results make two lessons visible.\n", "First, read the **gap to human**, not the ranking. At release, people\n", "solved 92% of one multi-step web-and-tool question set against 15% for a\n", "leading model with plugins; on real desktop tasks the figures were 72%\n", "against 12%; on realistic web tasks, 78% against 14%. Second, the\n", "**harness and environment are part of the score**. An issue-fixing\n", "benchmark graded by each repository’s own tests measures the agent\n", "harness, the test suite and the model together, and a grader bug can\n", "move the score as much as a model change (Principle 5). A product suite\n", "built from your own traces answers the question a leaderboard cannot:\n", "does it work on *your* traffic?\n", "\n", "### Multi-agent systems: failures are coordination failures\n", "\n", "A research assistant built from an orchestrator and parallel worker\n", "agents illustrates the failure-and-fix pattern at system scale:\n", "\n", "| Failure | Fix |\n", "|------------------------------------|------------------------------------|\n", "| The orchestrator spawned 50 workers for a simple question | Explicit effort rules: one worker and 3–10 tool calls for a fact; 2–4 workers for a comparison; more than 10 only for broad research |\n", "| Vague delegation (“research the shortage”) produced duplicated work | Every delegation states an objective, an output format, tool guidance and boundaries |\n", "| Agents went down wrong paths because tool descriptions were poor | An agent exercised each tool and rewrote its description; later tasks finished 40% faster |\n", "| Users reported “it can’t find obvious information” | Undiagnosable until full tracing was added |\n", "| Agents preferred low-quality sources | Caught by human testers, not by the automated judge |\n", "\n", "The judge for this system worked best as **one call** returning a 0–1\n", "score and a pass/fail verdict over a rubric: factual accuracy, citation\n", "accuracy, completeness, source quality, tool efficiency." ], "id": "2cce6862-da48-4117-af94-3ee7a8c7d1f3" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A multi-agent workflow behaves more like a distributed system than a\n", "conversation. Treat it like one: typed messages at every boundary, one\n", "valid action per step, logged intermediate state, retries and partial\n", "failure designed in from the start.\n", "\n", "### Long-running agents: the harness is an eval machine\n", "\n", "An agent that works across many sessions forgets everything between\n", "them. The harnesses that work turn the eval into an artifact the agent\n", "reads at the start of every session:\n", "\n", "- A **feature list** in a machine-readable file: over 200 features for\n", " one web-app build, each with step-by-step verification, all initially\n", " marked failing. The agent may change only the pass/fail field, never\n", " the tests.\n", "- An **init script** that starts the app, and a **progress log** plus\n", " version history the agent reads first.\n", "\n", "| Failure | Fix |\n", "|------------------------------------|------------------------------------|\n", "| Declares the whole job done early | Read the feature list and pick one unfinished feature per session |\n", "| Marks a feature done after unit tests only | Verify end to end in a real browser, as a user would |\n", "| Starts new work on a broken app | Run the app and a smoke test before touching anything |\n", "| Wastes time working out how to run the app | Read the init script |\n", "\n", "Self-evaluation is lenient, so the stronger designs split the work into\n", "a **generator** and a separate **evaluator** agent that clicks through\n", "the running app. Out of the box the evaluator found real bugs and then\n", "talked itself into approving them. It was tuned by reading its logs\n", "where its judgement differed from a human’s and revising its prompt,\n", "over several rounds. Before each block of work, generator and evaluator\n", "agree a **sprint contract**: what “done” means and how it will be\n", "checked (one contract had 27 criteria). Any criterion under its\n", "threshold fails the sprint." ], "id": "c621fcd7-28df-4b80-bcfe-1fb85480655b" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The cost trade is explicit. A solo run took 20 minutes and cost \\$9, and\n", "the core feature did not work. The full harness took 6 hours and cost\n", "\\$200, and the app worked. With a later, stronger model, the sprint\n", "structure could be dropped and the evaluator kept for one final pass: 3\n", "hours 50 minutes, about \\$125.\n", "\n", "> **Key Insight**\n", ">\n", "> Every component of a harness, and every grader in a suite, encodes an\n", "> assumption about what the model cannot do on its own. Those\n", "> assumptions expire. When the model changes, remove one component at a\n", "> time and measure which ones are still load-bearing.\n", "\n", "### Context and artifacts are eval targets too\n", "\n", "Long runs fail when context fills with material that no longer matters.\n", "One fix replaces each document the agent has read with a summary of the\n", "part that was relevant, reclaiming the tokens. Another fix keeps\n", "decisions in files rather than in chat history: a brief, a decisions\n", "log, the source material. That gives a simple, gradeable test: **can a\n", "fresh session, reading only the files, continue without repeating\n", "earlier mistakes?** Tests written before the code serve the same role\n", "for a coding agent. They are a specification it can be pointed at and\n", "checked against.\n", "\n", "A durable session log, kept separate from the model loop and the\n", "execution sandbox, makes every run replayable. Any past trace can be\n", "re-graded when a grader improves.\n", "\n", "## How Agentic Infrastructure Fails\n", "\n", "[Module 29](../m29_agentic_infra/lesson.qmd) built the layers around an\n", "agent: a durable runtime, a context builder, memory, tool protocols, a\n", "model gateway, isolation, security policy and multi-worker coordination.\n", "Each layer fails in its own way. Most of those failures share one\n", "property that makes them dangerous: **the final answer still looks\n", "fine.** A duplicated refund, a dropped instruction or a silently\n", "degraded fallback model all produce a fluent reply. A vibe check of\n", "outputs will not find them.\n", "\n", "What finds them is the trace. Every failure below leaves a mark in the\n", "spans of the run, and nearly every one can be caught by a\n", "**deterministic** check over the trace or the final state. That is the\n", "strongest argument for instrumenting the infrastructure (m29) before\n", "building the eval suite (this module). The cheapest grader for an\n", "infrastructure failure is usually a query over a span." ], "id": "28a4110b-b4a2-4883-aaab-b22b5fa70cf7" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Three patterns repeat across the atlas:\n", "\n", "1. **Check the state, not the reply.** The runtime, protocol and\n", " coordination failures are invisible in the reply text and obvious in\n", " the database, the tool result or the task log.\n", "2. **Slice the panel by an infrastructure attribute.** The gateway and\n", " context failures affect only some requests: those served by the\n", " fallback model, or long conversations. An unsliced pass rate dilutes\n", " them. Slicing by the span attribute that names the model or the\n", " context size isolates them, the same way the root-only evaluator\n", " isolated non-product inputs in Principle 2.\n", "3. **Make adversarial cases part of the suite.** Isolation and security\n", " failures do not occur in normal traffic often enough to be sampled.\n", " They need tasks written to provoke them (an injected page, a trial\n", " that leaves an answer on disk), run on every change like any other\n", " regression test.\n", "\n", "## Closing the Loop: The Self-Improving System\n", "\n", "The designs converge on one loop:\n", "\n", "1. Production traces and **expert corrections** are captured.\n", "2. Error analysis turns them into **structured eval cases**.\n", "3. An engineer, or a coding agent, diagnoses the largest failure mode\n", " and proposes a change to the *harness*: prompts, tools, retrieval,\n", " code.\n", "4. The regression suite and a human review **gate** the change.\n", "5. The change ships, and monitoring feeds step 1.\n", "\n", "The part that improves is the harness, not the model’s weights. A tax\n", "preparation system built this way was reported to have processed about\n", "7,000 returns, cut preparation time by about a third, and reached draft\n", "accuracy of up to 97%. In a chip-verification deployment, engineers\n", "embedded with domain experts built evals before writing the system. The\n", "key eval scores moved from about 60% to 80% to above 90% before\n", "production. The rule was that no LLM-based code is “done” until a\n", "verification for it exists. The same team reported that building the\n", "pipeline took six to eight weeks and earning users’ trust took about\n", "four more months.\n", "\n", "The simulator below runs that loop against a vibe-check loop on the same\n", "product with the same random side effects." ], "id": "7a01f80c-ab87-4cf2-a2ca-6e13237e988e" }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "vibe quality 0.704 -> 0.810 regressions 4 blocked 0\n", "eval quality 0.704 -> 0.884 regressions 0 blocked 11" ] } ], "source": [ "from flywheel import simulate\n", "\n", "for policy in (\"vibe\", \"eval\"):\n", " h = simulate(policy)\n", " start, end = h[0], h[-1]\n", " print(f\"{policy:5s} quality {start['quality']:.3f} -> {end['quality']:.3f} \"\n", " f\"regressions {end['shipped_regressions']} blocked {end['blocked']}\")" ], "id": "1439b01b" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Two effects separate the curves. The eval-gated loop fixes the *largest*\n", "mode, because error analysis on 60 traces ranks the modes and five\n", "traces cannot. It also blocks changes whose side effects land on a mode\n", "the suite already covers. The cost shows in the “blocked” count. A gate\n", "that blocks also slows progress, and the eval-gated curve flattens when\n", "most proposals are rejected. A real team would answer that by making\n", "changes smaller, not by removing the gate.\n", "\n", "## Common Pitfalls\n", "\n", "- **Generic metrics instead of product failure modes.** “Helpfulness\n", " 4.1/5” says nothing about what to fix. Start from error analysis.\n", "- **Judging before checking.** Reaching for a model judge when a schema\n", " check or regex would do. Deterministic first.\n", "- **Reporting raw judge agreement.** Report TPR and TNR on a held-out\n", " split.\n", "- **One aggregate score.** It says that something regressed, not what.\n", "- **Trusting healthy proxies.** Counts, schema validity and coverage can\n", " all look fine while recall collapses.\n", "- **Shared state between trials.** Isolate every trial.\n", "- **Never reading transcripts.** Every surprising number deserves a read\n", " before it is reported.\n", "- **Keeping harness components forever.** Re-test each one when the\n", " model changes.\n", "\n", "## Exercises\n", "\n", "### Exercise 1: Rank the failure modes\n", "\n", "Change the weights in `errors._DEMO_MODES` so that `tone` is the largest\n", "mode, regenerate the traces, and check that the Pareto order and the\n", "saturation curve change the way you expect." ], "id": "152477ce-3cab-447a-b202-42b5a421cac9" }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from errors import build_taxonomy, demo_traces, pareto\n", "# Your implementation here" ], "id": "1c98348e" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2: Weights that mis-rank\n", "\n", "Design weights for a four-level taxonomy. Find a pair of predictions\n", "where your weights rank the worse error (by your own judgement) as the\n", "better one." ], "id": "bd72b7e8-64c3-423c-9f4c-21f9eb99f31f" }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "from graders import hierarchical_score\n", "# Your implementation here" ], "id": "d8cdae85" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3: Localize a regression\n", "\n", "Build a `Panel` for a three-call pipeline, inject a drop into one call,\n", "and confirm that `Panel.diff` names it while `aggregate` moves by a\n", "third as much." ], "id": "c2729514-ae45-4a79-84d0-434fffa8fa54" }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "from graders import Panel\n", "# Your implementation here" ], "id": "c80f79dc" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4: When is a judge useless?\n", "\n", "With `SimulatedJudge`, find the value of TPR + TNR below which the 95%\n", "interval of the corrected pass rate is wider than 0.4, with 200 labels\n", "and 1,000 production traces." ], "id": "2dbe8320-055d-4419-ba4c-9037a21259d2" }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "from judge import SimulatedJudge, bootstrap_corrected\n", "# Your implementation here" ], "id": "702e96f0" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5: Recall versus noise\n", "\n", "Extend `degradation_sweep` with a fourth layout and plot recall against\n", "character error rate for each layout separately. Which layout fails\n", "first, and why?" ], "id": "6e0a9161-4881-4bf9-9986-0b493778a434" }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "from extraction import degradation_sweep\n", "# Your implementation here" ], "id": "173b6b26" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6: Catch the hallucinating extractor\n", "\n", "Add a fourth extractor to `demo_pooling` that copies A and adds three\n", "invented facts. Show that pooling ranks it first and that singletons\n", "expose it." ], "id": "b1a52c12-777e-4c2a-8ce9-8a1472093fe0" }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "from extraction import demo_pooling, pooled_recall, singletons\n", "# Your implementation here" ], "id": "d326f3fe" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7: Audit a suite\n", "\n", "Write a five-task suite with one task whose reference fails its graders\n", "and one task that can only pass with leaked state. Use `audit` and\n", "`demo_leakage` as models to find both." ], "id": "3e445490-7add-4e7e-9c5c-3bc6e9777d05" }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "from suite import Task, audit, run_suite\n", "# Your implementation here" ], "id": "fa6d83b2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8: Gate a change\n", "\n", "Generate per-case scores for two runs on 300 cases where one dimension\n", "drops by three points. Use `regression_gate` to find the smallest number\n", "of cases at which the gate reliably blocks it." ], "id": "af1d7088-0745-4d82-8ff1-66599aee394f" }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "from suite import regression_gate\n", "# Your implementation here" ], "id": "68b12681" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9: Break the flywheel\n", "\n", "In `flywheel.simulate`, make the eval-gated policy’s error-analysis\n", "sample as small as the vibe-check sample. How much of the advantage\n", "remains, and which effect is responsible for it?" ], "id": "de58cc37-a3e5-497f-8eec-2358604ecaee" }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "from flywheel import simulate\n", "# Your implementation here" ], "id": "5ef79a83" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "Key takeaways:\n", "\n", "1. **An eval is an instrument for one product.** The product improves\n", " only as fast as the instrument lets you see, explain and verify a\n", " change.\n", "2. **Read before measuring.** Error analysis on real traces produces\n", " the failure modes worth measuring. Keep reading.\n", "3. **Define correctness first, and decompose it.** Use graded scores\n", " where errors differ in cost. Keep one score per dimension and per\n", " LLM call.\n", "4. **Climb the grader ladder only as far as needed.** A model judge is\n", " a classifier. Report its TPR and TNR, and correct the pass rate it\n", " reports.\n", "5. **Ground truth has many sources.** They range from real failures and\n", " historical records to synthetic-by-construction data and label-free\n", " pooling. Healthy proxies can hide a broken system.\n", "6. **Suspect the eval first.** Audit tasks, graders and isolation\n", " before reading a score as a property of the system.\n", "7. **The same pieces build every larger design.** Agents are graded on\n", " outcomes. Multi-agent systems need typed boundaries. Long-running\n", " harnesses carry their eval as an artifact. The self-improving loop\n", " gates every change on the suite it keeps growing.\n", "\n", "### Going Deeper\n", "\n", "**Practice write-ups (cited for the case studies and figures in this\n", "module):**\n", "\n", "- [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/) —\n", " Husain, 2024. The three levels of evaluation, scoped assertions, trace\n", " viewers, judge alignment.\n", "- [Hamel Husain](https://latent.space/wiki/hamel-husain/) — Latent Space\n", " wiki profile (2026); critique-based judge calibration and\n", " judge-as-classifier reporting.\n", "- [Building and Scaling a Production Generative AI Assistant for\n", " Professional\n", " Networking](https://www.zenml.io/llmops-database/building-and-scaling-a-production-generative-ai-assistant-for-professional-networking)\n", " — ZenML LLMOps Database (2024). The 80%/95% plateau, annotation at\n", " scale, tolerant parsing, time-between-tokens.\n", "- [Demystifying evals for AI\n", " agents](https://www.anthropic.com/research/demystifying-evals-for-ai-agents)\n", " — Grace, Hadfield, Olivares, De Jonghe, 2026. Vocabulary, grader\n", " families, pass@k and pass^k, grader-bug audits.\n", "- [How we built our multi-agent research\n", " system](https://www.anthropic.com/research/multi-agent-research-system)\n", " — Hadfield et al., 2025. Effort rules, delegation, tool-description\n", " rewriting, rubric judges.\n", "- [Effective harnesses for long-running\n", " agents](https://www.anthropic.com/research/effective-harnesses-for-long-running-agents)\n", " — Young, 2025. Feature lists, progress files, end-to-end verification.\n", "- [Harness design for long-running application\n", " development](https://www.anthropic.com/research/harness-design-long-running-apps)\n", " — Rajasekaran, 2026. Generator/evaluator split, sprint contracts,\n", " cost, ablation.\n", "- [Scaling Managed Agents: Decoupling the brain from the\n", " hands](https://www.anthropic.com/research/managed-agents) — Martin,\n", " Cemaj, Cohen, 2026. Durable session logs; harness assumptions that\n", " expire.\n", "- [Multi-agent workflows often fail. Here’s how to engineer ones that\n", " don’t.](https://github.blog/ai-and-ml/generative-ai/multi-agent-workflows-often-fail-heres-how-to-engineer-ones-that-dont/)\n", " — Davis, 2026. Typed schemas, action schemas, enforcement at the tool\n", " boundary.\n", "- [Harness engineering: leveraging Codex in an agent-first\n", " world](https://openai.com/index/harness-engineering/) — 2026.\n", "- [Building self-improving tax agents with\n", " Codex](https://openai.com/index/building-self-improving-tax-agents-with-codex/)\n", " — 2026. Figures in this module come from secondary coverage; verify\n", " against the original.\n", "- [What FDE\n", " means](https://openai.com/index/forward-deployed-engineering-means/) —\n", " Jarvis. Eval-driven development with embedded engineers.\n", "- [AI-native company\n", " workflows](https://openai.com/index/ai-native-company-workflows/) —\n", " 2026.\n", "- [How we build evals for document\n", " extraction](https://www.ode.com/engineering/evals-for-document-extraction)\n", " — Serebryany, 2026. Precision/recall for facts, synthetic rendering,\n", " pooling, quote validation.\n", "- [Your evals have a Rotten Tomatoes\n", " problem](https://www.ode.com/engineering/your-evals-have-a-rotten-tomatoes-problem)\n", " — Girellini, 2026. Dimensional scoring and per-call evals.\n", "- [Insisting On Known Knowns: Using Evaluators to Drive\n", " Reliability](https://www.ode.com/engineering/insisting-on-known-knowns-using-evaluators-to-drive-reliability)\n", " — Marker, 2024. The hierarchical scorer and the root-only evaluator.\n", "- [Using historical codebase to build an eval\n", " suite](https://www.ode.com/engineering/using-historical-codebase-to-build-an-eval-suite)\n", " — Kadish, 2024.\n", "- [The Virtues of Showing Your Work: Do LLM Explanations Actually\n", " Help?](https://www.ode.com/engineering/the-virtues-of-showing-your-work-do-llm-explanations-actually-help)\n", " — Pentney, 2026.\n", "- [Fine Tuning Quirk](https://www.ode.com/engineering/fine-tuning-quirk)\n", " — Kadish, 2024.\n", "- [Artifact-driven AI\n", " creation](https://www.ode.com/engineering/artifact-driven-ai-creation)\n", " — Mann, 2026.\n", "- [If I could turn back\n", " time](https://www.ode.com/engineering/if-i-could-turn-back-time) —\n", " Frank, 2026.\n", "- [Do what I mean](https://www.ode.com/engineering/do-what-i-mean) —\n", " Marker, 2025.\n", "\n", "**Method and benchmarks:**\n", "\n", "- S. Yao, N. Shinn, P. Razavi, K. Narasimhan, “τ-bench: A Benchmark for\n", " Tool-Agent-User Interaction in Real-World Domains”, 2024.\n", " [arXiv:2406.12045](https://arxiv.org/abs/2406.12045). pass^k and the\n", " retail result.\n", "\n", "- L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot\n", " Arena”, 2023. [arXiv:2306.05685](https://arxiv.org/abs/2306.05685).\n", " Judge biases and agreement with humans.\n", "\n", "- S. Es, J. James, L. Espinosa-Anke, S. Schockaert, “RAGAS: Automated\n", " Evaluation of Retrieval Augmented Generation”, 2023.\n", " [arXiv:2309.15217](https://arxiv.org/abs/2309.15217). Faithfulness.\n", "\n", "- G. Mialon et al., “GAIA: a benchmark for General AI Assistants”, 2023.\n", " [arXiv:2311.12983](https://arxiv.org/abs/2311.12983).\n", "\n", "- T. Xie et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended\n", " Tasks in Real Computer Environments”, 2024.\n", " [arXiv:2404.07972](https://arxiv.org/abs/2404.07972).\n", "\n", "- S. Zhou et al., “WebArena: A Realistic Web Environment for Building\n", " Autonomous Agents”, 2023.\n", " [arXiv:2307.13854](https://arxiv.org/abs/2307.13854).\n", "\n", "- C. E. Jimenez et al., “SWE-bench: Can Language Models Resolve\n", " Real-World GitHub Issues?”, 2023.\n", " [arXiv:2310.06770](https://arxiv.org/abs/2310.06770).\n", "\n", "- W. J. Rogan and B. Gladen, “Estimating prevalence from the results of\n", " a screening test”, *American Journal of Epidemiology* 107(1), 1978.\n", " The pass-rate correction.\n", "\n", "## What’s Next\n", "\n", "This is the last module of the main sequence. [Module\n", "29](../m29_agentic_infra/lesson.qmd) builds the trace store and runtime\n", "that feed these evals, and [Module 17](../m17_evaluation/lesson.qmd)\n", "holds the statistics (error bars, paired differences, judge bias) used\n", "throughout. [Appendix A: Local Inference\n", "Engines](../appendix_a_local_inference/lesson.qmd) covers the engines\n", "underneath the model gateway, for readers who run models on their own\n", "hardware." ], "id": "460ab22f-38d9-45af-bd04-ca6e0591d88e" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3 (ipykernel)", "language": "python", "path": "/usr/local/share/jupyter/kernels/python3" }, "language_info": { "name": "python", "codemirror_mode": { "name": "ipython", "version": "3" }, "file_extension": ".py", "mimetype": "text/x-python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.18" } } }