Module 17: Evaluation

Introduction

Every module so far built capability: tokenize (m03), attend (m05), train (m07), align (m12), reason (m13). This one builds the discipline that keeps all of it honest — evaluation. The moment you claim a model is “better,” you owe an answer to measured how? — and the wrong metric can make a worse model look better.

Evaluation is genuinely hard, for three reasons this module takes head-on:

  • Different tasks need different metrics. “What is 6×7?” has one right answer; “write a function that sorts a list” has infinitely many correct programs; “which essay is better?” has no ground truth at all. One number does not fit them.
  • The obvious formula is often biased. For code, the standard metric pass@k has an unbiased estimator that is not the tempting 1−(1−c/n)^k — the same kind of “the naive version is subtly wrong” story as m13’s Condorcet vote.
  • Scores lie when the test leaks. If a benchmark appeared in the training data (contamination), every number on it is inflated, and you would never know without checking.

Why it matters for LLMs: leaderboards, ablations, and every “we improved X” in the book rest on evaluation. Build the metrics from scratch and you can read any model card critically — and never be fooled by a good-looking number again.

What You’ll Learn

After this module, you can:

  • Explain why evaluation is hard — task-metric mismatch, Goodhart’s law, and contamination — and pick the right metric family for a task.
  • Build exact-match accuracy with proper answer normalization, and see how a missing normalization step silently costs correct answers their points.
  • Derive and implement pass@k — the unbiased functional-correctness estimator for code — and see exactly how the naive estimator misleads.
  • Use an LLM-as-judge for open-ended answers, and measure its position bias by swapping the order and watching the winner flip.
  • Detect contamination with n-gram overlap between a test example and training text.

Prerequisites

This module requires familiarity with:

  • Module 07: Training — cross-entropy loss and perplexity, the intrinsic metric this module recaps and moves beyond.
  • Module 08: Generation — the generate loop that produces the samples pass@k and the judge score.
  • Module 13: Reasoning — verifiers and answer extraction; evaluation formalizes “did it check out?” into a benchmark number.

Intuition: What Are We Even Measuring?

A metric is a bet about what “correct” means. Match the bet to the task and the number is meaningful; mismatch it and the number is noise. Four families cover most of what you’ll ever report — step through them:

NoteKey Insight

The metric is part of the claim. “GPT-X scores 90%” is meaningless until you know which 90% — exact-match on MMLU, pass@1 on HumanEval, or win-rate judged by another model. Report the metric, or the number says nothing.

Exact Match & Normalized Accuracy

The simplest metric, for tasks with one right answer: does the prediction equal the gold answer? The catch is normalization. Models add trailing periods, articles, and stray capitalization; compare raw strings and a correct answer scores zero. normalize_answer lowercases, strips punctuation and articles, and collapses whitespace before comparing — the same idea as m13’s extract_answer, now as a scoring step:

from evaluation import normalize_answer, exact_match, accuracy

print(normalize_answer("The Answer is  42."))          # -> 'answer is 42'
print(exact_match("42.", "42"))                        # True after normalizing
print(exact_match("forty-two", "42"))                  # still False — different tokens

preds = ["42.", "Paris", "the dog"]
golds = ["42",  "paris", "a dog"]
print("accuracy:", accuracy(preds, golds))             # 3/3 — normalization saves all three
answer is 42
True
False
accuracy: 1.0
WarningNormalization is not neutral

Normalization is a choice that changes the score. Strip too little and “42.” is wrong; strip too much and “not 42” normalizes toward “42”. Every benchmark ships an exact normalization spec for this reason — report yours.

The Math: pass@k

Exact match works when there is one right string. Code breaks that: infinitely many programs sort a list, and none of them is a target string to match. So you don’t grade the text — you run the unit tests. That gives functional correctness, and the standard metric is pass@k: sample many completions, and ask how likely it is that at least one of k of them passes.

Concretely: draw n samples for a problem, run the tests, and count the c that pass. If you had reported only k of those n (drawn without replacement), the chance all k fail is \(\binom{n-c}{k} / \binom{n}{k}\), so

\[ \text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}. \]

The tempting shortcut \(1 - (1 - c/n)^k\) is biased — it samples with replacement, treating each of the k draws as independent. The combinatorial form is unbiased. Drive c, n, and k and watch the two curves diverge:

TipTry This
  1. Rare successes. Set c small (say 2 of 200). pass@1 is tiny, but pass@100 is large — sampling many times rescues a model that is usually wrong but occasionally right. This is why reasoning systems (m13) sample and aggregate.
  2. Watch the bias. With c/n ≈ 0.25, the biased curve sits below the correct one at mid-k — report the biased number and you understate the model. At k=1 the two agree exactly (both equal c/n).

Code: Functional Correctness

evaluation.py implements the stable product form (no giant binomials) and averages over problems — exactly what “HumanEval pass@k” means:

from evaluation import pass_at_k, estimate_pass_at_k

# One problem: 5 of 20 samples passed the tests.
print("pass@1: ", round(pass_at_k(20, 5, 1), 4))     # = c/n = 0.25
print("pass@10:", round(pass_at_k(20, 5, 10), 4))    # many tries -> much higher

# A benchmark: average pass@k over all its problems.
n_per = [20, 20, 20]          # samples per problem
c_per = [5, 0, 12]            # how many passed each
print("benchmark pass@1:", round(estimate_pass_at_k(n_per, c_per, 1), 4))
pass@1:  0.25
pass@10: 0.9837
benchmark pass@1: 0.2833

demonstrate_pass_at_k sweeps k for a model whose samples pass independently with some probability, reporting the unbiased estimate beside the biased one:

from evaluation import demonstrate_pass_at_k

curve = demonstrate_pass_at_k(n=20, per_sample_p=0.25)
============================================================
pass@k: unbiased vs. biased estimator
============================================================
  n=20 samples, 5/20 pass  (per-sample p=0.25)

     k    unbiased      biased       gap
     1      0.2500      0.2500   +0.0000
     2      0.4474      0.4375   -0.0099
     5      0.8063      0.7627   -0.0436
    10      0.9837      0.9437   -0.0401
    20      1.0000      0.9968   -0.0032

  The naive (biased) estimator samples WITH replacement and
  systematically misestimates; the combinatorial form is unbiased.
# Bridge the sweep to the plot below.
pak_points = [
    {"k": k, "unbiased": v["unbiased"], "biased": v["biased"]}
    for k, v in curve.items()
]
ojs_define(pak_points = pak_points)

LLM-as-Judge

For open-ended answers — essays, chat, summaries — there is no gold string and no test to run. The scalable modern answer is LLM-as-judge: show a strong model two answers and ask which is better, then report a win rate. It is cheap and correlates with human preference — but it has biases, and the sharpest is position bias: the judge can favor whichever answer is shown first, irrespective of quality.

You catch it by running every comparison both ways — original order and swapped. A judge that grades the answers names the same answer both times (so its A/B label flips when the order flips); a judge that grades the position keeps the same label. position_bias measures how often the judge picked the same slot both ways:

from evaluation import pairwise_win_rate, position_bias, judge_agreement

# Answer-consistent judge: label flips with the order (A->B) => no position bias.
print("fair position bias:", position_bias(["A", "A", "B"], ["B", "B", "A"]))
# Slot-biased judge: always picks whatever is shown first.
print("biased position bias:", position_bias(["A", "A", "A"], ["A", "A", "A"]))

# Validity check: does the judge agree with human labels?
print("agreement:", round(judge_agreement(["A", "B", "A", "B"], ["A", "B", "B", "B"]), 2))
fair position bias: 0.0
biased position bias: 1.0
agreement: 0.75

demonstrate_judge simulates a judge with a tunable bias and measures the fallout — how a biased judge inflates the win rate of whichever answer sits in the first slot:

from evaluation import demonstrate_judge

results = [
    {"bias": b, **demonstrate_judge(num_pairs=400, bias_strength=b, seed=0, verbose=False)}
    for b in [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]
]
for r in results:
    print(f"bias={r['bias']:.1f}  position_bias={r['position_bias']:.2f}  "
          f"slot-A win rate={r['slotA_win_rate']:.2f}")
bias=0.0  position_bias=0.00  slot-A win rate=0.51
bias=0.2  position_bias=0.20  slot-A win rate=0.61
bias=0.4  position_bias=0.39  slot-A win rate=0.70
bias=0.6  position_bias=0.59  slot-A win rate=0.80
bias=0.8  position_bias=0.81  slot-A win rate=0.90
bias=1.0  position_bias=1.00  slot-A win rate=1.00
ojs_define(judge_points = results)
NoteKey Insight

An unvalidated judge is a rumor. Always (1) swap the order and require the verdict to hold, and (2) check agreement with human labels on a sample. A win rate from a judge that flips with position is measuring layout, not quality.

Contamination

Every metric above assumes the test set is unseen. If a benchmark leaked into the training data — and web-scraped corpora are full of leaked benchmarks — the model can recite the answer, and the score measures memorization, not skill. A cheap first check is n-gram overlap: what fraction of a test example’s n-grams also appear in the training text?

from evaluation import ngram_overlap

test_q = "the mitochondria is the powerhouse of the cell"
clean_train = "cells contain many organelles with distinct roles"
leaked_train = "biology fact: the mitochondria is the powerhouse of the cell"

print("overlap vs clean corpus: ", round(ngram_overlap(test_q, clean_train, n=5), 2))
print("overlap vs leaked corpus:", round(ngram_overlap(test_q, leaked_train, n=5), 2))
overlap vs clean corpus:  0.0
overlap vs leaked corpus: 1.0

A near-1 overlap against a training shard means the example is contaminated and its score should be discarded. Real pipelines run this at scale (e.g. 13-gram or 50-char matches) across the whole corpus before trusting a benchmark.

Common Pitfalls

When evaluating models, watch out for:

  1. Reporting a metric without its normalization / prompt. Exact-match accuracy depends entirely on the normalization spec and the prompt format. Two papers’ “MMLU 70%” can be incomparable. State exactly how you scored.
  2. Using the biased pass@k. 1−(1−c/n)^k is not pass@k; it drifts from the unbiased combinatorial estimator for k>1. Use the combinatorial form (or its stable product) that HumanEval defines.
  3. Trusting an LLM judge unchecked. Position, verbosity, and self-preference biases are real. Swap the order, validate against humans, and never let a model grade its own outputs unaudited.
  4. Ignoring contamination. A sky-high score on a public benchmark is a red flag, not a triumph — check n-gram overlap against training before believing it.
  5. Goodhart’s law. “When a measure becomes a target, it ceases to be a good measure.” Optimizing directly for a benchmark (or a reward model, m12) produces models that ace the metric and fail the task. Hold out a fresh test set.
  6. Single-number reductionism. One aggregate hides per-category collapses (great on easy items, zero on hard ones). Break scores down before declaring victory.

Exercises

Exercise 1: The pass@k break-even

from evaluation import pass_at_k

# A model passes each sample with probability 0.1 (so c ≈ 0.1·n). How many samples k
# does pass@k need to first exceed 0.5? Try n = 100, c = 10, and loop k. Then explain
# why sampling is cheaper than making the model 5× better per-sample.

# Your implementation here:

Exercise 2: Normalization matters

from evaluation import accuracy, normalize_answer

# Given preds = ["The answer is 42.", "42", "forty two"] and golds = ["42","42","42"],
# compute accuracy. Then write a stricter normalizer (numbers only) and a looser one
# (word2number) and show how the SAME predictions score differently. Which is "right"?

# Your implementation here:

Exercise 3: Debias the judge

from evaluation import demonstrate_judge, position_bias

# demonstrate_judge exposes a position-biased judge. A standard fix is to average the
# two orderings into one verdict (a win only counts if it survives the swap). Simulate
# this "consistency-required" scoring and show it removes the slot-A inflation even at
# bias_strength=0.6.

# Your implementation here:

Summary

Key takeaways:

  1. The metric is part of the claim. Intrinsic (perplexity), closed-form (exact match), functional (pass@k), and judged (win rate) each fit a different task. A number without its metric says nothing.
  2. Normalize before you match. Trailing punctuation, articles, and case sink correct answers; every benchmark ships a normalization spec for this reason.
  3. pass@k is unbiased only in the combinatorial form. \(1 - \binom{n-c}{k}/\binom{n}{k}\) — not \(1-(1-c/n)^k\) — estimates the chance that one of k code samples passes. More samples raise pass@k, which is why reasoning systems sample and aggregate.
  4. LLM-as-judge is scalable but biased. Position bias flips the winner when you swap the order; always run both orders and validate against humans.
  5. Contamination invalidates everything. If the test leaked into training, the score measures memorization. Check n-gram overlap before you believe a benchmark.
  6. Goodhart looms. Optimizing the metric destroys the metric — hold out a fresh test set and read scores skeptically.

What’s Next

You can now measure a model as well as build one — the last missing piece of the loop. From here the book turns to putting models to work: tool use and agents (models that act, not just answer) and interpretability (opening the box to see why a model scores the way it does). Evaluation is the thread through all of it: every new capability is only as real as the metric that confirms it.

Going Deeper

Core Papers:

Practical Resources: