Module 30: Production Evals

Introduction

Module 17 measured models on benchmarks: exact match, pass@k, judges, leaderboards, error bars. A product team asks a different question. Its system is a model wrapped in prompts, retrieval, tools and code, serving one kind of user with one kind of request, and the question is: is this system doing what we meant, on the inputs we actually get, and did the change we just made make it better or worse? No public benchmark answers that. The team has to build the instrument itself. That instrument is called an eval.

A common history repeats across production write-ups. A team gets a working prototype quickly. A conversational assistant for job seekers, for example, reached about 80% of its target experience in the first month, then took four more months to get past 95%. In the second phase the fixes become whack-a-mole: a prompt change repairs one complaint and silently breaks another, the prompt grows a paragraph per edge case, and nobody can say whether last week’s release was an improvement. The model did not stop improving. The team ran out of ways to see.

This module builds that sight from first principles. It starts with the smallest unit (reading one trace and writing down what went wrong) and ends with the full loop that lets a system improve from its own production traffic.

What You’ll Learn

After this module, you can:

  • State the mental model: an eval is an instrument built for one product, and the product improves only as fast as the instrument lets you see, explain and verify a change.
  • Run error analysis: read traces, code failures, rank failure modes, and know when to stop reading.
  • Define correctness before tuning, including for labels where errors differ in cost.
  • Replace one score with an instrument panel that says what regressed.
  • Choose a grader from the grader ladder and validate a model judge as a classifier, then correct the pass rate it reports.
  • Build ground truth five ways, from real failures to label-free pooling.
  • Audit an eval for the bugs that make a score mean nothing.
  • Explain how these pieces combine for agents, multi-agent systems and long-running harnesses, and run a simulation of the improvement loop that ties them together.

Prerequisites

Intuition: Three Capabilities, One Loop

Every team that ships an LLM product needs three capabilities:

  1. Measure quality: a repeatable way to say how good the system is.
  2. Debug: a way to look at what the system did on a given input and why.
  3. Change the system: edit prompts, code, retrieval, tools, or fine-tune.

Most teams invest almost entirely in the third. Changing a prompt is easy and feels like progress. Without the first two, though, each change is a guess judged by a glance at a few outputs (a “vibe check”). Vibe checks are useful early and never stop being useful. They stop being sufficient once the easy failures are gone, because the remaining failures are rare, scattered, and easy to reintroduce.

Step through the loop. Each step names what stalls when it is missing.

NoteKey Insight

The rate at which an LLM product improves is set by the slowest arm of the loop. A team that can change the system in minutes but needs a day to know whether the change helped improves at the speed of a day.

Vocabulary

Production eval teams have converged on a small vocabulary. Module 29 built the layer that records it as spans. A task is one input with success criteria. A trial is one attempt at it. A grader scores some aspect of a trial. The transcript (or trace) records everything the system did. The outcome is the final state of the world, not the system’s claim about it. An eval harness runs tasks and grades them. It is not the same thing as the agent harness, the code that turns a model into an agent. An eval of an agent always measures the model and its harness together.

Two kinds of suite behave differently. A capability suite asks “can it do this yet?” and starts with a low pass rate. A regression suite asks “did we break anything?” and should stay near 100%. When a capability suite saturates, its tasks graduate into the regression suite.

Principle 1: Look at the Data First

The failure categories that matter for a product are not in any generic metric list. They are in the product’s traces. The first eval activity is not writing a metric. It is reading.

The procedure has four steps:

  1. Sample traces from real or realistic traffic.
  2. Open-code: for each trace, write one short note on the first thing that went wrong. The first error matters most, because later errors are often consequences of it. Mark the trace pass or fail. Binary verdicts come before any 1–5 scale. A scale invites disagreement about what a 3 means and hides the decision you actually need, which is ship or fix.
  3. Axial-code: group the notes into a small set of named failure modes.
  4. Count and rank. Fix the largest mode first.

Keep reading in batches until new batches stop producing new modes. That point is called saturation. Reading never stops entirely, because traffic changes. But saturation tells you when another batch is unlikely to teach you anything new about the current distribution.

Friction decides whether this happens at all. Teams that read traces routinely almost always build a small purpose-made viewer. It shows the input, the output, the retrieved context and the tool calls on one screen, with a pass/fail button and a notes field. A generic log search tool is rarely enough.

from errors import build_taxonomy, demo_traces, pareto, saturation_curve

traces = demo_traces()          # 60 reviewed traces from a scheduling assistant
for r in pareto(build_taxonomy(traces)):
    print(f"{r.mode:22s}{r.count:3d} {r.share:5.0%}  cum {r.cumulative:5.0%}")
print("new modes per batch:", saturation_curve(traces, 10))
wrong-date-range        9   36%  cum   36%
missed-constraint       6   24%  cum   60%
hallucinated-attendee   4   16%  cum   76%
tool-error-hidden       3   12%  cum   88%
tone                    2    8%  cum   96%
leaked-internal-id      1    4%  cum  100%
new modes per batch: [4, 1, 0, 0, 1, 0]

The top two modes account for 60% of the failures. A team that had been tuning tone, because a stakeholder forwarded one curt reply, was working on 8% of the problem.

TipTry This

In the figure, set the batch slider to 5 and then 20. A small batch makes the saturation curve noisy, and a late new mode can arrive after several zeros. That is why “stop at saturation” means “stop for now”.

Principle 2: Define Correctness Before Tuning

Eyeballing outputs and editing prompts builds intuition, but it does not build anything you can repeat. If you change the prompt before deciding what “right” means, every result looks like progress. Write the grader first.

Exact match is the simplest definition, and it is often too blunt. Consider a classifier that maps free-text product descriptions onto a taxonomy with about 10,000 leaves, eight levels deep. Getting the top-level department wrong (“Electronics” for a pair of boots) is far worse than getting the eighth-level leaf wrong. Exact match scores both as 0.

A hierarchical score charges for the first level that goes wrong, with a weight that shrinks with depth:

\text{score} = 1 - w_{\ell^\ast}, \qquad w = (1.0,\ 0.7,\ 0.5,\ 0.4,\ 0.3,\ 0.2,\ 0.15,\ 0.1),

where \ell^\ast is the first mismatched level. A wrong root scores 0. A wrong eighth level scores 0.9.

from graders import hierarchical_score, root_only

gold = ["Apparel", "Shoes", "Boots", "Ankle boots"]
for pred in (gold,
             ["Apparel", "Shoes", "Boots", "Chelsea boots"],
             ["Apparel", "Shoes", "Sandals", "Slides"],
             ["Electronics", "Phones", "Cases", "Folio"]):
    score, root = hierarchical_score(pred, gold), root_only(pred, gold)
    print(f"{score:.2f}  root={root:.0f}  {' > '.join(pred)}")
1.00  root=1  Apparel > Shoes > Boots > Ankle boots
0.60  root=1  Apparel > Shoes > Boots > Chelsea boots
0.50  root=1  Apparel > Shoes > Sandals > Slides
0.00  root=0  Electronics > Phones > Cases > Folio

The story continues past the scorer, and the continuation is the reason to build more than one evaluator. Most errors were deep and cheap. A second evaluator that scored only the root level isolated the expensive ones. Filtering failures by it showed that many were not product descriptions at all. They were audience-targeting labels that only made sense to the person who wrote them, such as “purchase-prediction segment, shopping, 90 days”. The model was forcing a product category onto them. No prompt fix was right. The fix was architectural: detect this input class and label it “uncategorizable”.

NoteKey Insight

A second evaluator is a lens, not a duplicate. Each grader you add lets you filter failures by a different property, and filtering is how an undiscovered input class becomes visible.

Principle 3: One Score Says That, a Panel Says What

A single aggregate score is like a film’s percentage on a review aggregator. It tells you how many critics liked the film, not whether it is funny or frightening or both. When one eval score drops from 0.87 to 0.81 after a prompt change, the team knows that something regressed. It does not know what, and the usual response is an hour of re-running the pipeline and guessing.

The fix is to score dimensions, each on a simple scale, and to score each LLM call in the pipeline separately (“shrink-wrapping” the call). A pipeline-level score alone is an integration test with no unit tests. A support-ticket summarizer, for example, produces a category, a priority and a draft reply. It gets three graders, not one:

Dimension Grader Scale
Category accuracy exact match against a label 0 / 1
Priority calibration within one step of the label 0 / 0.5 / 1
Reply relevance model judge with a rubric 0 / 1

The panel below runs a four-dimension pipeline twice. The second run degrades one LLM call.

from graders import demo_panel_runs

runs = demo_panel_runs()
a, b = runs["aggregate_before"], runs["aggregate_after"]
print(f"aggregate: {a:.3f} -> {b:.3f}")
for dim, delta in runs["diff"].items():
    before, after = runs["before"][dim], runs["after"][dim]
    print(f"  {dim:22s} {before:.2f} -> {after:.2f} ({delta:+.2f})")
aggregate: 0.925 -> 0.885
  citation grounding     0.88 -> 0.72 (-0.16)
  extraction precision   0.94 -> 0.94 (+0.00)
  policy compliance      0.97 -> 0.97 (+0.00)
  tone                   0.91 -> 0.91 (+0.00)

Two side effects make the panel worth its cost. First, writing a dimension forces the team to define it. A team cannot write a “realism” grader without agreeing on what realism means, so building evals doubles as requirements discovery. Second, the panel separates evals from validation. An eval is an instrument you read to make a decision. A validation check runs on every production response and acts on it (see Runtime validation).

Principle 4: The Grader Ladder

Graders come in three families, ordered by cost:

  1. Deterministic checks: schema validation, exact or tolerant numeric match, regular expressions, set membership, checks on the environment’s final state, checks on which tools were called. They are fast, free and reproducible. Their weakness is brittleness. Use them for everything they can express. A single generic assertion can cover a whole class of failures, for example a regex that fails any reply that shows the user a raw internal ID.
  2. Model-based graders: a rubric, a natural-language assertion (“the reply does not promise a refund”), a pairwise comparison, or a comparison against a reference answer. They handle nuance. They are non-deterministic, they cost money, and they must be validated (next section).
  3. Human review: expert grading, spot checks, A/B tests with real users. Humans are the reference that calibrates everything else. They are also the slowest and most expensive rung.

Run each rung at the cadence its cost allows. Assertions run on every change. Judges and human review run on a schedule. A/B tests run after significant changes to a mature product.

from graders import exact, numeric_close, regex_absent, schema_check

UUID = r"[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}"
schema = {"category": str, "priority": int}
reply = "Booked: 3f2b1c9a-1d2e-4f5a-8b9c-0d1e2f3a4b5c"
checks = [schema_check({"category": "billing", "priority": "high"}, schema),
          regex_absent(reply, UUID, name="no-internal-ids"),
          exact("96.12", "96.124991"),
          numeric_close("96.12", 96.124991, rel=1e-3)]
for r in checks:
    print(f"{r.name:16s} {r.verdict.value:5s} {r.detail}")
schema           fail  wrong type: priority
no-internal-ids  fail  found '3f2b1c9a-1d2e-4f5a-8b9c-0d1e2f3a4b5c'
exact            fail  '96.12' != '96.124991'
numeric_close    pass

A pass rate is a product decision, not automatically 100%. A team decides which failures it can tolerate, at what rate, and writes that threshold down.

Principle 4, continued: A Judge Is a Classifier

A model judge reads an output and returns pass or fail. Before you trust its verdicts you have to know how often it is right, which means treating the judge like any other binary classifier.

The procedure that works in practice:

  1. A domain expert labels a sample of outputs pass/fail and writes a short critique explaining each verdict.
  2. The judge prompt is written from those critiques. A few labelled examples go into the prompt as demonstrations.
  3. The remaining labels are split into a dev set, used to iterate on the prompt, and a held-out test set, used once to report the judge’s accuracy.
  4. Report the true-positive rate (of outputs the expert passed, the share the judge passed) and the true-negative rate (of outputs the expert failed, the share the judge failed) separately.

Step 4 is where most teams go wrong. Most production outputs pass, so a judge that passes everything has high raw agreement with the expert and catches nothing.

from judge import confusion, cohen_kappa, raw_agreement, tpr_tnr

human = [True] * 90 + [False] * 10   # 90% of outputs are good
lazy = [True] * 100                  # a judge that always says PASS
c = confusion(lazy, human)
tpr, tnr = tpr_tnr(c)
print(f"raw agreement {raw_agreement(lazy, human):.0%}")
print(f"TPR {tpr:.2f}  TNR {tnr:.2f}  kappa {cohen_kappa(c):.2f}")
raw agreement 90%
TPR 1.00  TNR 0.00  kappa 0.00

Further practices: use one judge per dimension rather than one judge for everything; let the judge answer “unknown” rather than forcing a guess; prefer binary verdicts. Iterating on a judge prompt with a single expert over three rounds has been reported to reach more than 90% agreement on a balanced set. Without a separate evaluator, a system grading its own work is lenient. A separate judge tuned toward skepticism is easier to fix than a generator asked to criticize itself.

A judge also has known biases. It tends to prefer the first of two answers it is shown (position), the longer answer (verbosity) and answers written in its own style (self-enhancement), and it can fail on reasoning it cannot do itself. Strong judges agree with human raters more than 80% of the time, about as often as humans agree with each other. That figure is a ceiling for a well-validated judge, not a default for an unvalidated one. Running a pairwise comparison in both orders, and counting a win only when both verdicts agree, controls the position effect at twice the cost.

The Math: Correcting the Pass Rate a Judge Reports

Suppose the judge has measured rates TPR and TNR, and on unlabelled production traffic it passes a share q of outputs. The true pass rate p is unknown. Each output is either truly good (probability p), in which case the judge passes it with probability TPR, or truly bad, in which case the judge passes it with probability 1 - \text{TNR}:

q = \text{TPR}\cdot p + (1 - \text{TNR})(1 - p).

Solve for p:

\hat p = \frac{q + \text{TNR} - 1}{\text{TPR} + \text{TNR} - 1}.

The denominator \text{TPR} + \text{TNR} - 1 measures how much information the judge carries. When it is 0, the verdict is independent of the truth and no correction is possible. When it is small, small errors in TPR, TNR or q become large errors in \hat p. The error bar comes from resampling both the labelled split (uncertainty in TPR and TNR) and the production verdicts (uncertainty in q).

from judge import demo_judge

d = demo_judge()
print(f"TPR {d['tpr']:.2f}  TNR {d['tnr']:.2f}  (on 200 labels)")
print(f"judge reports   {d['q']:.1%}")
print(f"corrected       {d['point']:.1%}  95% CI {d['lo']:.1%}-{d['hi']:.1%}")
print(f"true rate       {d['true_production_rate']:.1%}")
TPR 0.95  TNR 0.64  (on 200 labels)
judge reports   77.9%
corrected       70.8%  95% CI 60.9%-78.9%
true rate       68.7%
TipTry This

Set TNR to 0.55 and TPR to 0.5. The interval explodes, because the judge barely separates good from bad. Then raise the number of labels. More labels narrow the interval only up to a point. A judge near chance cannot be rescued by measuring it more precisely.

Where Ground Truth Comes From

A grader compares an output with something. Labels are the expensive part of every eval. Five sources, ordered from most to least direct:

  1. Real failures. 20–50 tasks drawn from actual problems are enough to start. Early changes tend to have large effects. One research assistant’s success rate went from 30% to 80% on about 20 queries after one prompt change, so a small set detects them.
  2. Historical records. A closed ticket paired with the merged code change that resolved it is a solved example with no annotation needed. One team built several hundred labelled cases this way for a system that suggests API endpoints for integration requests. Each case asked two questions: are the needed endpoints in the top five suggestions, and is the most-needed one ranked first?
  3. Synthetic by construction. Generate the facts first, then render them into documents. The label set is exhaustive because it existed before the document did. Render the same facts into many layouts, then add degradation: character noise at the levels of a clean scan, a light fax and a heavy fax, or a column header that appears only on page one. Synthetic user inputs work the same way: enumerate features × scenarios (a search that returns zero, one or many results) and generate inputs for each cell.
  4. Partial expert labels. Experts label only the facts that matter. This measures recall on what matters. It cannot measure precision, because unlabelled facts might be correct.
  5. No labels. Run several systems, pool their outputs and treat the union as pseudo-truth. This gives a relative ranking only. Facts that only one system produced are singletons and need review. A system that produces many singletons without winning on recall is probably hallucinating.

Healthy proxies, broken system

For extraction from scanned documents, recall is the share of the page’s facts the system captured, and precision is the share of extracted facts the page supports. Both need exhaustive labels. One team reported an extractor that, on a badly degraded page, returned about forty well-typed, quoted facts per page. Its true recall was 9%. The values were built from misread characters. Fact counts, schema conformance and quote coverage all looked healthy. Only hand labels showed the problem.

The sweep below reproduces the shape with a facts-first synthetic generator and a layout-parsing extractor.

from extraction import degradation_sweep

print("  CER  facts/page  schema  precision  recall")
for r in degradation_sweep():
    cer, n, ok = r["cer"], r["facts_per_page"], r["schema_rate"]
    p, rec = r["precision"], r["recall"]
    print(f"{cer:5.1%}  {n:9.1f}  {ok:6.0%}  {p:9.0%}  {rec:6.0%}")
  CER  facts/page  schema  precision  recall
 0.0%       12.0    100%       100%    100%
 0.7%       11.8    100%        94%     92%
 4.5%       10.8    100%        55%     49%
13.3%        9.3    100%        18%     14%
25.0%        8.5     90%         5%      3%

At a heavy-fax error rate the extractor still returns three quarters as many facts as on a clean page, and every one of them is schema-valid. Recall is 14%.

from extraction import demo_pooling

p = demo_pooling()
print("    pooled  singletons | true recall  true precision")
for k in "ABC":
    print(f"{k}:  {p['pooled_recall'][k]:6.0%}  {p['singletons'][k]:10d} | "
          f"{p['true_recall'][k]:11.0%}  {p['true_precision'][k]:14.0%}")
    pooled  singletons | true recall  true precision
A:     67%           3 |         83%            100%
B:     47%           0 |         58%            100%
C:     73%           5 |         50%             55%

Pooling ranks C first because pooling rewards volume. C’s singleton count is the warning. Its extra facts are inventions, and its true recall is the lowest of the three.

WarningA high score can hide the failure that matters

A 95% recall says nothing about which 5% is missing. If the missing facts are the allergy and the dose, the extractor is unusable. Partial expert labels on the critical facts exist to catch exactly this case.

Runtime Validation Is Not an Eval

Some checks run on every production response and act on the result: they reject, retry, repair or escalate. They borrow eval machinery but play a different role.

  • Verbatim quotes. Each extracted fact must carry a quote that occurs on the page and contains the value. Facts whose quote matches nothing are discarded. This is an upper bound on hallucination, not a measurement of it, because OCR noise and line breaks also break matches.
  • Tolerant parsing. One assistant found about 10% of its structured responses malformed. Re-prompting would have added latency and load. A parser that recognized and repaired the common mistakes, plus hints in the prompt about those mistakes, brought the rate to about 0.01%.
  • Typed boundaries. In a multi-agent workflow, every message between agents and every tool call is validated against a schema, and an agent may return only one of a fixed set of actions (request more information, assign, close as duplicate, no action). A violation is a contract failure: retry, repair or escalate before bad state propagates.
from extraction import Fact, quote_validate

page = "DOSE: 5 mg\nROUTE: oral"
kept, rejected = quote_validate([Fact("dose", "5 mg", "DOSE: 5 mg"),
                                 Fact("route", "iv", "ROUTE: iv")], page)
print("kept", [f.value for f in kept], "| rejected", [f.value for f in rejected])
kept ['5 mg'] | rejected ['iv']

Principle 5: Suspect the Eval First

A surprising score is a bug in the task, the grader or the harness until the transcripts show otherwise. Published audits of agent benchmarks have found all of the following:

  • A grader required the string 96.124991… and rejected 96.12. After this and similar grader fixes, plus a less constrained harness, one model’s score on the benchmark went from 42% to 95%. The model did not change.
  • A task penalized a system for following the task’s stated instruction.
  • A test assumed a file path that the task description never stated.
  • Trials shared state, and an agent read a previous trial’s history to solve the next one.
  • An agent found a loophole in a booking policy that served the user better, and the eval as written marked it failed.

A 0% pass rate over many trials for a capable system usually means a broken task, not an incapable system.

from suite import Task, audit, demo_leakage

tasks = [
    Task("ok", "2+2", reference="4", graders=[lambda o: o == "4"]),
    Task("no-ref", "summarize", reference=None, graders=[lambda o: len(o) < 200]),
    # the spec never names the output path the grader expects
    Task("impossible", "path?", reference="out.txt", graders=[lambda o: o == "data/out.txt"]),
]
for f in audit(tasks):
    print(f"{f.task_id:10s} {f.problem}")
leak = demo_leakage()
print(f"trial pass rate: isolated {leak['isolated_trial_rate']:.0%}, "
      f"shared state {leak['shared_trial_rate']:.0%}")
no-ref     no reference solution
impossible reference solution fails its own graders
trial pass rate: isolated 33%, shared state 68%

The audit checklist:

  1. Every task has a reference solution, and the reference passes its own graders.
  2. Two domain experts reading the task would reach the same verdict.
  3. Trials start from a clean environment.
  4. Someone reads transcripts before the score is reported.
  5. Saturated suites are retired to regression duty and replaced.
WarningExplanations are not evidence

Asking a model to explain its answer is tempting to read as a window into its decision. Controlled tests on classification tasks found that adding an explanation before or after the label changed accuracy by zero to three points and sometimes lowered it. When a hint changed the answer, the model produced a matching justification for the new answer. Treat an explanation as a plausible story. If you want to know whether reasoning helps your task, measure it on your task.

The same caution applies to unexplained gains. One team fine-tuned a moderation classifier on generated rationales and saw a large improvement. They then found that a bug had left the post text out of the training prompts. A gain you cannot explain is a reason to check the eval, not to ship.

Feedback Speed

A team that waits a day for its eval signal improves at the speed of a day. Production teams typically run three loops at different speeds:

  • Engineer checks: minutes, coarse, on a handful of examples.
  • Annotator review: about a day. One team scaled this to a group of linguists reviewing up to 500 conversations per day, scored on overall quality, hallucination, policy violations, coherence and style. That signal became its main indicator of production readiness.
  • User feedback: three or more days per change.

Automated graders and validated judges exist to move signal from the slow loops into the fast one. Latency also belongs on the panel. One team bounded time to first token but not the time between tokens. With a 200-token reasoning step, a 10 ms increase per token adds 2 seconds per response, and a public launch triggered timeout alerts.

Worked Examples: The Principles on Real Systems

The principles above come from teams that shipped LLM products and wrote about what broke. This part walks through five of those systems. Each one is anonymized and, where the source gave no numbers, simulated so the figure can be stepped through. Each follows the same arc: the system, what went wrong, which principle applied and how, and what changed. The figures label any number the source did not report as illustrative.

Worked Example 1: A Job-Search Assistant That Plateaued

The system. A conversational assistant answers questions about jobs, companies and posts. A router sends each question to one of several specialized agents. A retrieval step calls internal search services. A generation step writes the answer with a larger model.

What went wrong. The first month reached about 80% of the target experience. The next four months were spent getting past 95%. Three separate problems were hiding inside one overall number:

  1. Quality plateaued. Overall quality is the product of the stage pass rates, because a request succeeds only when every stage succeeds. Once the router and retrieval were good, nearly all remaining failures were in generation, mostly hallucination. That was the hardest stage to measure and fix.
  2. About 10% of structured outputs were malformed. Each agent emitted parameters for the next step in a structured format. Re-prompting on a failure would have added latency and load.
  3. Responses timed out at launch. The team had bounded time to first token but not the time between tokens. A reasoning step multiplies every per-token delay.

Applying the principles.

  • Decompose (Principle 3): score each stage separately, so the plateau shows up as one weak stage rather than a flat overall line.
  • Deterministic first (Principle 4): a parser that recognizes and repairs the common format mistakes, plus prompt hints about those mistakes, took malformed outputs from about 10% to about 0.01% without a second model call.
  • Feedback speed: a group of linguists reviewed up to 500 conversations a day on quality, hallucination, policy, coherence and style. That became the main readiness signal. Model-based hallucination detection was the first automated grader to work.
  • Latency on the panel: time between tokens was added as a tracked dimension next to time to first token.
TipTry This

Move from month 1 to month 5. The router gains 2.5 points and generation gains 9, while overall quality gains 15, because it is the product of the three stages. Generation is the weakest stage in every month. Then set the time between tokens to 30 ms with 200 reasoning tokens. The extra 10 ms per token adds 3.5 s per response, and the response (about 11 s) crosses the 10 s timeout.

Worked Example 2: A CRM Assistant Stuck in Whack-a-Mole

The system. An assistant inside a real-estate CRM finds listings, creates contacts, schedules showings and drafts emails in natural language.

What went wrong. Prompt changes made fast early progress and then stalled. Each fix for a reported problem broke something nobody was watching, and the prompt grew a paragraph per edge case. The team checked each change on the two flows it demoed every week, so the breakages were invisible until users hit them.

Applying the principles.

  • Look at the data (Principle 1): reading traces showed that failures clustered by scenario (no results, many results, an ambiguous name, a missing field) more than by feature.
  • Define correctness (Principle 2): one scoped assertion per feature × scenario cell. For example, “when a search returns zero listings, the reply says so and offers to widen the search”. Synthetic inputs were generated for every cell. Generic assertions, such as a regex that fails any reply showing a raw internal ID, ran on every cell.
  • Cadence (Principle 4): the assertion suite ran on every change. Trace review and an aligned judge ran on a schedule. A/B tests were saved for large changes.

Step through five prompt versions. A star marks the two cells the weekly demo covered.

Version 2 is the lesson. The demo passes, the targeted fix works, and the overall pass rate falls, because two cells nobody demoed broke. With the matrix in place, versions 4 and 5 fix exactly the red cells and cannot silently break green ones.

Worked Example 3: Aligning a Judge With One Expert

The system. An assistant turns a plain-language question into a query in a product’s query language. Whether a query is right depends on whether it answers the question asked, which only a domain expert can judge reliably. No rule captures that.

What went wrong. The first judge prompt was a generic rubric. It agreed with the expert on most outputs, and it passed almost every query that ran, including queries that answered a different question. Raw agreement looked fine because most outputs were good.

Applying the principles.

  • A judge is a classifier (Principle 4): the expert labelled outputs pass or fail and wrote a one-paragraph critique for each. The labels were split into train (examples for the prompt), dev (for iterating) and a held-out test split. TPR and TNR were reported separately.
  • Each round folded the expert’s critiques into the judge prompt as explicit rules and examples. After three rounds, agreement with the expert was reported above 90%.

The first round is the trap: raw agreement in the mid-70s, while the judge catches fewer than a third of the bad queries. Only TNR shows the problem. The rounds that follow raise TNR from 30% to 90% while TPR stays between 90% and 100%, because the critiques describe why a query fails.

Worked Example 4: Ground Truth From History

The system. Engineers who build integrations receive feature requests as tickets. A recommender pre-fills each ticket with the API endpoints it will probably need, plus documentation links.

What went wrong. Nothing yet, and that was the problem. The team could not tell whether the recommendations were good without hundreds of labelled tickets. Labelling them by hand would have cost a great deal of engineer time.

Applying the principles.

  • Where ground truth comes from: every closed ticket that added a new capability and links to a merged code change is a solved example. The change shows which endpoints were actually used. A model extracted the endpoints from each code change, and the suite was built from those.
  • Define correctness (Principle 2): two questions per case. Are the needed endpoints in the top five suggestions? Is the most-needed endpoint ranked first?

The funnel’s stage counts are illustrative; the source reported only that the filters left “a few hundred” cases. The filters matter more than the count. Excluding fixes keeps only tickets where the change is the whole answer. Requiring a merged change guarantees that the “answer” was accepted by a reviewer.

Worked Example 5: A Long-Running Coding Agent That Declared Victory

The system. An agent builds a web application over many sessions. Each session starts with an empty context window.

What went wrong. Without structure, the agent tried to build too much at once and marked features done after a unit test or a single request. It never checked them end to end. A later session saw a mostly finished project and declared the whole job complete. The claimed progress and the true progress drifted apart, and nothing in the loop measured the gap.

Applying the principles. The fix turns the eval into an artifact the agent reads every session. That artifact is a feature list of over 200 items, each with verification steps, all initially failing:

  • Define correctness first (Principle 2): every feature has its verification steps before any code is written.
  • Grade the outcome (Designs): a feature is marked passing only after an end-to-end check in a real browser.
  • Suspect the state (Principle 5): each session starts by running the app and a smoke test, which reopens anything a later change broke.

Session counts and break rates here are simulated. The pattern, a widening claimed-versus-true gap and an early stop, is the reported one.

Designs: How the Pieces Fit

Everything so far applies to a single LLM call. Agents, multi-agent systems and long-running harnesses use the same principles, combined differently.

Agents: grade the outcome

An agent changes the world, so grade the final state: the reservation exists in the database, the test suite passes, the file is on disk. The agent’s claim that it finished is not evidence. Check the path as a supplement (it should not have deleted the database on the way), give partial credit, and add checkpoints for long workflows. Run several trials per task and report both pass@k and pass^k (next subsection): a 75% per-trial rate is only 42% reliable across three consecutive runs.

When the agent retrieves before it answers, one metric separates “answered” from “supported”:

\text{faithfulness} = \frac{\#\ \text{claims in the response supported by the retrieved context}}{\#\ \text{claims in the response}}.

A correct answer with low faithfulness came from the model’s memory, not from the sources it cited, and it will fail when the sources and the memory disagree.

No single method catches everything. Production teams layer automated suites, production monitoring, A/B tests, user feedback, transcript review and periodic human studies. Each layer catches failures the others miss.

Reliability: pass@k versus pass^k

An agent’s output varies from run to run, so one trial per task measures luck. Two summaries of k trials are easy to confuse and mean opposite things. pass@k is the probability that at least one of k attempts succeeds, and it rises with k. pass^k is the probability that all k attempts succeed, and it falls. With p the per-trial success rate and independent trials,

\text{pass@}k = 1 - (1 - p)^k, \qquad \text{pass}^k = p^k .

pass@k answers “can it ever do this?”, the right question when a person or a verifier picks the best of k attempts. pass^k answers “will it do this every time?”, the right question for an agent that runs unattended. Both equal p at k = 1 and then move apart. At p = 0.9, ten consecutive successes still happen only 0.9^{10} \approx 35\% of the time. A published benchmark of tool-using agents in simulated customer-service domains found that the best function-calling agents of the time solved fewer than half of the tasks, and that pass^8 was below 25% in its retail domain.

Independence is the figure’s assumption. Real failures are correlated: a task that fails for a structural reason fails every time, which makes pass@k lower than the formula says and pass^k higher.

Reading public agent benchmarks

Public agent benchmarks are useful as a reference point and misleading as a product eval. Their reported results make two lessons visible. First, read the gap to human, not the ranking. At release, people solved 92% of one multi-step web-and-tool question set against 15% for a leading model with plugins; on real desktop tasks the figures were 72% against 12%; on realistic web tasks, 78% against 14%. Second, the harness and environment are part of the score. An issue-fixing benchmark graded by each repository’s own tests measures the agent harness, the test suite and the model together, and a grader bug can move the score as much as a model change (Principle 5). A product suite built from your own traces answers the question a leaderboard cannot: does it work on your traffic?

Multi-agent systems: failures are coordination failures

A research assistant built from an orchestrator and parallel worker agents illustrates the failure-and-fix pattern at system scale:

Failure Fix
The orchestrator spawned 50 workers for a simple question Explicit effort rules: one worker and 3–10 tool calls for a fact; 2–4 workers for a comparison; more than 10 only for broad research
Vague delegation (“research the shortage”) produced duplicated work Every delegation states an objective, an output format, tool guidance and boundaries
Agents went down wrong paths because tool descriptions were poor An agent exercised each tool and rewrote its description; later tasks finished 40% faster
Users reported “it can’t find obvious information” Undiagnosable until full tracing was added
Agents preferred low-quality sources Caught by human testers, not by the automated judge

The judge for this system worked best as one call returning a 0–1 score and a pass/fail verdict over a rubric: factual accuracy, citation accuracy, completeness, source quality, tool efficiency.

A multi-agent workflow behaves more like a distributed system than a conversation. Treat it like one: typed messages at every boundary, one valid action per step, logged intermediate state, retries and partial failure designed in from the start.

Long-running agents: the harness is an eval machine

An agent that works across many sessions forgets everything between them. The harnesses that work turn the eval into an artifact the agent reads at the start of every session:

  • A feature list in a machine-readable file: over 200 features for one web-app build, each with step-by-step verification, all initially marked failing. The agent may change only the pass/fail field, never the tests.
  • An init script that starts the app, and a progress log plus version history the agent reads first.
Failure Fix
Declares the whole job done early Read the feature list and pick one unfinished feature per session
Marks a feature done after unit tests only Verify end to end in a real browser, as a user would
Starts new work on a broken app Run the app and a smoke test before touching anything
Wastes time working out how to run the app Read the init script

Self-evaluation is lenient, so the stronger designs split the work into a generator and a separate evaluator agent that clicks through the running app. Out of the box the evaluator found real bugs and then talked itself into approving them. It was tuned by reading its logs where its judgement differed from a human’s and revising its prompt, over several rounds. Before each block of work, generator and evaluator agree a sprint contract: what “done” means and how it will be checked (one contract had 27 criteria). Any criterion under its threshold fails the sprint.

The cost trade is explicit. A solo run took 20 minutes and cost $9, and the core feature did not work. The full harness took 6 hours and cost $200, and the app worked. With a later, stronger model, the sprint structure could be dropped and the evaluator kept for one final pass: 3 hours 50 minutes, about $125.

NoteKey Insight

Every component of a harness, and every grader in a suite, encodes an assumption about what the model cannot do on its own. Those assumptions expire. When the model changes, remove one component at a time and measure which ones are still load-bearing.

Context and artifacts are eval targets too

Long runs fail when context fills with material that no longer matters. One fix replaces each document the agent has read with a summary of the part that was relevant, reclaiming the tokens. Another fix keeps decisions in files rather than in chat history: a brief, a decisions log, the source material. That gives a simple, gradeable test: can a fresh session, reading only the files, continue without repeating earlier mistakes? Tests written before the code serve the same role for a coding agent. They are a specification it can be pointed at and checked against.

A durable session log, kept separate from the model loop and the execution sandbox, makes every run replayable. Any past trace can be re-graded when a grader improves.

How Agentic Infrastructure Fails

Module 29 built the layers around an agent: a durable runtime, a context builder, memory, tool protocols, a model gateway, isolation, security policy and multi-worker coordination. Each layer fails in its own way. Most of those failures share one property that makes them dangerous: the final answer still looks fine. A duplicated refund, a dropped instruction or a silently degraded fallback model all produce a fluent reply. A vibe check of outputs will not find them.

What finds them is the trace. Every failure below leaves a mark in the spans of the run, and nearly every one can be caught by a deterministic check over the trace or the final state. That is the strongest argument for instrumenting the infrastructure (m29) before building the eval suite (this module). The cheapest grader for an infrastructure failure is usually a query over a span.

Three patterns repeat across the atlas:

  1. Check the state, not the reply. The runtime, protocol and coordination failures are invisible in the reply text and obvious in the database, the tool result or the task log.
  2. Slice the panel by an infrastructure attribute. The gateway and context failures affect only some requests: those served by the fallback model, or long conversations. An unsliced pass rate dilutes them. Slicing by the span attribute that names the model or the context size isolates them, the same way the root-only evaluator isolated non-product inputs in Principle 2.
  3. Make adversarial cases part of the suite. Isolation and security failures do not occur in normal traffic often enough to be sampled. They need tasks written to provoke them (an injected page, a trial that leaves an answer on disk), run on every change like any other regression test.

Closing the Loop: The Self-Improving System

The designs converge on one loop:

  1. Production traces and expert corrections are captured.
  2. Error analysis turns them into structured eval cases.
  3. An engineer, or a coding agent, diagnoses the largest failure mode and proposes a change to the harness: prompts, tools, retrieval, code.
  4. The regression suite and a human review gate the change.
  5. The change ships, and monitoring feeds step 1.

The part that improves is the harness, not the model’s weights. A tax preparation system built this way was reported to have processed about 7,000 returns, cut preparation time by about a third, and reached draft accuracy of up to 97%. In a chip-verification deployment, engineers embedded with domain experts built evals before writing the system. The key eval scores moved from about 60% to 80% to above 90% before production. The rule was that no LLM-based code is “done” until a verification for it exists. The same team reported that building the pipeline took six to eight weeks and earning users’ trust took about four more months.

The simulator below runs that loop against a vibe-check loop on the same product with the same random side effects.

from flywheel import simulate

for policy in ("vibe", "eval"):
    h = simulate(policy)
    start, end = h[0], h[-1]
    print(f"{policy:5s} quality {start['quality']:.3f} -> {end['quality']:.3f}  "
          f"regressions {end['shipped_regressions']}  blocked {end['blocked']}")
vibe  quality 0.704 -> 0.810  regressions 4  blocked 0
eval  quality 0.704 -> 0.884  regressions 0  blocked 11

Two effects separate the curves. The eval-gated loop fixes the largest mode, because error analysis on 60 traces ranks the modes and five traces cannot. It also blocks changes whose side effects land on a mode the suite already covers. The cost shows in the “blocked” count. A gate that blocks also slows progress, and the eval-gated curve flattens when most proposals are rejected. A real team would answer that by making changes smaller, not by removing the gate.

Common Pitfalls

  • Generic metrics instead of product failure modes. “Helpfulness 4.1/5” says nothing about what to fix. Start from error analysis.
  • Judging before checking. Reaching for a model judge when a schema check or regex would do. Deterministic first.
  • Reporting raw judge agreement. Report TPR and TNR on a held-out split.
  • One aggregate score. It says that something regressed, not what.
  • Trusting healthy proxies. Counts, schema validity and coverage can all look fine while recall collapses.
  • Shared state between trials. Isolate every trial.
  • Never reading transcripts. Every surprising number deserves a read before it is reported.
  • Keeping harness components forever. Re-test each one when the model changes.

Exercises

Exercise 1: Rank the failure modes

Change the weights in errors._DEMO_MODES so that tone is the largest mode, regenerate the traces, and check that the Pareto order and the saturation curve change the way you expect.

from errors import build_taxonomy, demo_traces, pareto
# Your implementation here

Exercise 2: Weights that mis-rank

Design weights for a four-level taxonomy. Find a pair of predictions where your weights rank the worse error (by your own judgement) as the better one.

from graders import hierarchical_score
# Your implementation here

Exercise 3: Localize a regression

Build a Panel for a three-call pipeline, inject a drop into one call, and confirm that Panel.diff names it while aggregate moves by a third as much.

from graders import Panel
# Your implementation here

Exercise 4: When is a judge useless?

With SimulatedJudge, find the value of TPR + TNR below which the 95% interval of the corrected pass rate is wider than 0.4, with 200 labels and 1,000 production traces.

from judge import SimulatedJudge, bootstrap_corrected
# Your implementation here

Exercise 5: Recall versus noise

Extend degradation_sweep with a fourth layout and plot recall against character error rate for each layout separately. Which layout fails first, and why?

from extraction import degradation_sweep
# Your implementation here

Exercise 6: Catch the hallucinating extractor

Add a fourth extractor to demo_pooling that copies A and adds three invented facts. Show that pooling ranks it first and that singletons expose it.

from extraction import demo_pooling, pooled_recall, singletons
# Your implementation here

Exercise 7: Audit a suite

Write a five-task suite with one task whose reference fails its graders and one task that can only pass with leaked state. Use audit and demo_leakage as models to find both.

from suite import Task, audit, run_suite
# Your implementation here

Exercise 8: Gate a change

Generate per-case scores for two runs on 300 cases where one dimension drops by three points. Use regression_gate to find the smallest number of cases at which the gate reliably blocks it.

from suite import regression_gate
# Your implementation here

Exercise 9: Break the flywheel

In flywheel.simulate, make the eval-gated policy’s error-analysis sample as small as the vibe-check sample. How much of the advantage remains, and which effect is responsible for it?

from flywheel import simulate
# Your implementation here

Summary

Key takeaways:

  1. An eval is an instrument for one product. The product improves only as fast as the instrument lets you see, explain and verify a change.
  2. Read before measuring. Error analysis on real traces produces the failure modes worth measuring. Keep reading.
  3. Define correctness first, and decompose it. Use graded scores where errors differ in cost. Keep one score per dimension and per LLM call.
  4. Climb the grader ladder only as far as needed. A model judge is a classifier. Report its TPR and TNR, and correct the pass rate it reports.
  5. Ground truth has many sources. They range from real failures and historical records to synthetic-by-construction data and label-free pooling. Healthy proxies can hide a broken system.
  6. Suspect the eval first. Audit tasks, graders and isolation before reading a score as a property of the system.
  7. The same pieces build every larger design. Agents are graded on outcomes. Multi-agent systems need typed boundaries. Long-running harnesses carry their eval as an artifact. The self-improving loop gates every change on the suite it keeps growing.

Going Deeper

Practice write-ups (cited for the case studies and figures in this module):

Method and benchmarks:

  • S. Yao, N. Shinn, P. Razavi, K. Narasimhan, “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, 2024. arXiv:2406.12045. pass^k and the retail result.

  • L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, 2023. arXiv:2306.05685. Judge biases and agreement with humans.

  • S. Es, J. James, L. Espinosa-Anke, S. Schockaert, “RAGAS: Automated Evaluation of Retrieval Augmented Generation”, 2023. arXiv:2309.15217. Faithfulness.

  • G. Mialon et al., “GAIA: a benchmark for General AI Assistants”, 2023. arXiv:2311.12983.

  • T. Xie et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments”, 2024. arXiv:2404.07972.

  • S. Zhou et al., “WebArena: A Realistic Web Environment for Building Autonomous Agents”, 2023. arXiv:2307.13854.

  • C. E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, 2023. arXiv:2310.06770.

  • W. J. Rogan and B. Gladen, “Estimating prevalence from the results of a screening test”, American Journal of Epidemiology 107(1), 1978. The pass-rate correction.

What’s Next

This is the last module of the main sequence. Module 29 builds the trace store and runtime that feed these evals, and Module 17 holds the statistics (error bars, paired differences, judge bias) used throughout. Appendix A: Local Inference Engines covers the engines underneath the model gateway, for readers who run models on their own hardware.