Module 17: Evaluation

Introduction

Every module so far built capability: tokenize (m03), attend (m05), train (m07), align (m12), reason (m13). This one builds the discipline that keeps all of it honest — evaluation. The moment you claim a model is “better,” you owe an answer to measured how? — and the wrong metric can make a worse model look better.

Evaluation is genuinely hard, for three reasons this module takes head-on:

  • Different tasks need different metrics. “What is 6×7?” has one right answer; “write a function that sorts a list” has infinitely many correct programs; “which essay is better?” has no ground truth at all. One number does not fit them.
  • The obvious formula is often biased. For code, the standard metric pass@k has an unbiased estimator that is not the tempting 1−(1−c/n)^k — the same kind of “the naive version is subtly wrong” story as m13’s Condorcet vote.
  • Scores lie when the test leaks. If a benchmark appeared in the training data (contamination), every number on it is inflated, and you would never know without checking.

Why it matters for LLMs: leaderboards, ablations, and every “we improved X” in the book rest on evaluation. Build the metrics from scratch and you can read any model card critically — and never be fooled by a good-looking number again.

What You’ll Learn

After this module, you can:

  • Explain why evaluation is hard — task-metric mismatch, Goodhart’s law, and contamination — and pick the right metric family for a task.
  • Build exact-match accuracy with proper answer normalization, and see how a missing normalization step silently costs correct answers their points.
  • Derive and implement pass@k — the unbiased functional-correctness estimator for code — and see exactly how the naive estimator misleads.
  • Use an LLM-as-judge for open-ended answers, and measure its position bias by swapping the order and watching the winner flip.
  • Measure whether a model’s confidence is honest with the Expected Calibration Error and a reliability diagram, and fix an overconfident model with temperature scaling — one scalar that rescales confidence without touching accuracy.
  • Detect contamination with n-gram overlap between a test example and training text.
  • Explain why some “emergent abilities” are a metric mirage — build the nonlinear metric (exact_match = pᴸ) that turns a smooth skill curve into an apparent cliff, and the linear metric that dissolves it.

Prerequisites

This module requires familiarity with:

  • Module 07: Training — cross-entropy loss and perplexity, the intrinsic metric this module recaps and moves beyond.
  • Module 08: Generation — the generate loop that produces the samples pass@k and the judge score.
  • Module 13: Reasoning — verifiers and answer extraction; evaluation formalizes “did it check out?” into a benchmark number.

Intuition: What Are We Even Measuring?

A metric is a bet about what “correct” means. Match the bet to the task and the number is meaningful; mismatch it and the number is noise. Four families cover most of what you’ll ever report — step through them:

NoteKey Insight

The metric is part of the claim. “GPT-X scores 90%” is meaningless until you know which 90% — exact-match on MMLU, pass@1 on HumanEval, or win-rate judged by another model. Report the metric, or the number says nothing.

Exact Match & Normalized Accuracy

The simplest metric, for tasks with one right answer: does the prediction equal the gold answer? The catch is normalization. Models add trailing periods, articles, and stray capitalization; compare raw strings and a correct answer scores zero. normalize_answer lowercases, strips punctuation and articles, and collapses whitespace before comparing — the same idea as m13’s extract_answer, now as a scoring step:

from evaluation import normalize_answer, exact_match, accuracy

print(normalize_answer("The Answer is  42."))          # -> 'answer is 42'
print(exact_match("42.", "42"))                        # True after normalizing
print(exact_match("forty-two", "42"))                  # still False — different tokens

preds = ["42.", "Paris", "the dog"]
golds = ["42",  "paris", "a dog"]
print("accuracy:", accuracy(preds, golds))             # 3/3 — normalization saves all three
answer is 42
True
False
accuracy: 1.0
WarningNormalization is not neutral

Normalization is a choice that changes the score. Strip too little and “42.” is wrong; strip too much and “not 42” normalizes toward “42”. Every benchmark ships an exact normalization spec for this reason — report yours.

The Math: pass@k

Exact match works when there is one right string. Code breaks that: infinitely many programs sort a list, and none of them is a target string to match. So you don’t grade the text — you run the unit tests. That gives functional correctness, and the standard metric is pass@k: sample many completions, and ask how likely it is that at least one of k of them passes.

Concretely: draw n samples for a problem, run the tests, and count the c that pass. If you had reported only k of those n (drawn without replacement), the chance all k fail is \binom{n-c}{k} / \binom{n}{k}, so

\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.

The tempting shortcut 1 - (1 - c/n)^k is biased — it samples with replacement, treating each of the k draws as independent. The combinatorial form is unbiased. Drive c, n, and k and watch the two curves diverge:

TipTry This
  1. Rare successes. Set c small (say 2 of 200). pass@1 is tiny, but pass@100 is large — sampling many times rescues a model that is usually wrong but occasionally right. This is why reasoning systems (m13) sample and aggregate.
  2. Watch the bias. With c/n ≈ 0.25, the biased curve sits below the correct one at mid-k — report the biased number and you understate the model. At k=1 the two agree exactly (both equal c/n).

Code: Functional Correctness

evaluation.py implements the stable product form (no giant binomials) and averages over problems — exactly what “HumanEval pass@k” means:

from evaluation import pass_at_k, estimate_pass_at_k

# One problem: 5 of 20 samples passed the tests.
print("pass@1: ", round(pass_at_k(20, 5, 1), 4))     # = c/n = 0.25
print("pass@10:", round(pass_at_k(20, 5, 10), 4))    # many tries -> much higher

# A benchmark: average pass@k over all its problems.
n_per = [20, 20, 20]          # samples per problem
c_per = [5, 0, 12]            # how many passed each
print("benchmark pass@1:", round(estimate_pass_at_k(n_per, c_per, 1), 4))
pass@1:  0.25
pass@10: 0.9837
benchmark pass@1: 0.2833

demonstrate_pass_at_k sweeps k for a model whose samples pass independently with some probability, reporting the unbiased estimate beside the biased one:

from evaluation import demonstrate_pass_at_k

curve = demonstrate_pass_at_k(n=20, per_sample_p=0.25)
============================================================
pass@k: unbiased vs. biased estimator
============================================================
  n=20 samples, 5/20 pass  (per-sample p=0.25)

     k    unbiased      biased       gap
     1      0.2500      0.2500   +0.0000
     2      0.4474      0.4375   -0.0099
     5      0.8063      0.7627   -0.0436
    10      0.9837      0.9437   -0.0401
    20      1.0000      0.9968   -0.0032

  The naive (biased) estimator samples WITH replacement and
  systematically misestimates; the combinatorial form is unbiased.
# Bridge the sweep to the plot below.
pak_points = [
    {"k": k, "unbiased": v["unbiased"], "biased": v["biased"]}
    for k, v in curve.items()
]
ojs_define(pak_points = pak_points)

LLM-as-Judge

For open-ended answers — essays, chat, summaries — there is no gold string and no test to run. The scalable modern answer is LLM-as-judge: show a strong model two answers and ask which is better, then report a win rate. It is cheap and correlates with human preference — but it has biases, and the sharpest is position bias: the judge can favor whichever answer is shown first, irrespective of quality.

You catch it by running every comparison both ways — original order and swapped. A judge that grades the answers names the same answer both times (so its A/B label flips when the order flips); a judge that grades the position keeps the same label. position_bias measures how often the judge picked the same slot both ways:

from evaluation import pairwise_win_rate, position_bias, judge_agreement

# Answer-consistent judge: label flips with the order (A->B) => no position bias.
print("fair position bias:", position_bias(["A", "A", "B"], ["B", "B", "A"]))
# Slot-biased judge: always picks whatever is shown first.
print("biased position bias:", position_bias(["A", "A", "A"], ["A", "A", "A"]))

# Validity check: does the judge agree with human labels?
print("agreement:", round(judge_agreement(["A", "B", "A", "B"], ["A", "B", "B", "B"]), 2))
fair position bias: 0.0
biased position bias: 1.0
agreement: 0.75

demonstrate_judge simulates a judge with a tunable bias and measures the fallout — how a biased judge inflates the win rate of whichever answer sits in the first slot:

from evaluation import demonstrate_judge

results = [
    {"bias": b, **demonstrate_judge(num_pairs=400, bias_strength=b, seed=0, verbose=False)}
    for b in [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]
]
for r in results:
    print(f"bias={r['bias']:.1f}  position_bias={r['position_bias']:.2f}  "
          f"slot-A win rate={r['slotA_win_rate']:.2f}")
bias=0.0  position_bias=0.00  slot-A win rate=0.51
bias=0.2  position_bias=0.20  slot-A win rate=0.61
bias=0.4  position_bias=0.39  slot-A win rate=0.70
bias=0.6  position_bias=0.59  slot-A win rate=0.80
bias=0.8  position_bias=0.81  slot-A win rate=0.90
bias=1.0  position_bias=1.00  slot-A win rate=1.00
ojs_define(judge_points = results)
NoteKey Insight

An unvalidated judge is a rumor. Always (1) swap the order and require the verdict to hold, and (2) check agreement with human labels on a sample. A win rate from a judge that flips with position is measuring layout, not quality.

Ranking Models by Battles: Bradley-Terry & Elo

A single judgment is one bit: A beat B on this prompt. A modern leaderboard — LMArena (the LMSYS Chatbot Arena) — collects millions of those bits, across dozens of anonymous models, from human voters (and, on automated benchmarks, from the LLM judge of the previous section). Then it has to answer a much harder question than any single win rate: given only who beat whom, what is each model’s one number, and can you trust the ordering it induces?

The trick is to invent a hidden strength for every model and demand it explain the battles. Give model i a latent strength \beta_i on a log scale, and predict every battle with a logistic curve — the Bradley-Terry model (Bradley & Terry, 1952):

P(i \text{ beats } j) \;=\; \sigma(\beta_i - \beta_j) \;=\; \frac{1}{1 + e^{-(\beta_i - \beta_j)}}.

Equal strength is a coin flip; a one-unit edge is a 0.73 win rate. Finding the \beta that best explains the observed wins is nothing more exotic than logistic regression with one \pm 1 indicator per model. And — the punchline of this section — the famous chess Elo rating is this exact model wearing different units.

NoteKey Insight

An arena does not measure models against a fixed answer key — there isn’t one for open-ended chat. It measures them against each other, then solves for the single set of strengths that best reproduces every head-to-head. The leaderboard is a fit, not a tally.

The Math: One Logistic, Two Names

Only the difference \beta_i - \beta_j ever appears, so adding the same constant to every strength changes no probability. The model is identifiable only up to a global shift — so we pin it down by centering the strengths at 0 (or anchoring one model), exactly as an arena anchors its scale.

To get from strengths to the numbers you see on a leaderboard, rescale and shift:

R \;=\; 1000 + \beta \cdot \frac{400}{\ln 10}.

Under this change of variables the Bradley-Terry probability becomes the Elo expected-score formula, letter for letter:

\sigma(\beta_A - \beta_B) \;=\; \frac{1}{1 + 10^{-(R_A - R_B)/400}} \;=\; E_A .

The chess convention only chooses the axis: base 10 instead of e, a spread of 400 points for a 10\!:\!1 win ratio, and a starting rating of 1000. Same model, friendlier numbers.

Code: Fit a Leaderboard from Battles

bradley_terry_mle fits the strengths by gradient descent on the logistic loss — the same batch maximum-likelihood an arena runs — then beta_to_elo prints them on the familiar scale. We stage a synthetic arena with known strengths so we can check the fit recovers the truth (see arena.py):

from arena import (
    bt_win_prob, elo_expected, beta_to_elo,
    bradley_terry_mle, elo_ratings, rank_players, kendall_tau,
    simulate_battles, bootstrap_ci,
)

# Five models with known latent strengths (top to bottom ≈ 250 Elo points).
true_beta = {"sonnet": 0.75, "opus": 0.55, "haiku": 0.10, "gpt": -0.40, "small": -1.00}
battles = simulate_battles(true_beta, num_battles=3000, seed=0)   # (winner, loser) pairs

# Fit strengths from the battles alone, then read them as Elo.
fit_beta = bradley_terry_mle(battles)
mle_elo = {m: beta_to_elo(b) for m, b in fit_beta.items()}
true_elo = {m: beta_to_elo(b) for m, b in true_beta.items()}

print(f"{'model':<8}{'true Elo':>10}{'fitted Elo':>12}")
for m in rank_players(true_beta):
    print(f"{m:<8}{true_elo[m]:>10.0f}{mle_elo[m]:>12.0f}")
print("\nranking recovered exactly:", rank_players(fit_beta) == rank_players(true_beta))
model     true Elo  fitted Elo
sonnet        1130        1130
opus          1096        1085
haiku         1017        1024
gpt            931         937
small          826         824

ranking recovered exactly: True

From nothing but who beat whom, the fit lands within a handful of Elo of the true strengths and recovers the exact ordering. The probability the fit assigns to any matchup is just bt_win_prob, and elo_expected on the fitted ratings agrees to the last digit — the two formulas are one model:

p_bt = bt_win_prob(fit_beta["sonnet"], fit_beta["gpt"])
p_elo = elo_expected(mle_elo["sonnet"], mle_elo["gpt"])
print(f"P(sonnet beats gpt): Bradley-Terry={p_bt:.4f}  Elo formula={p_elo:.4f}")
P(sonnet beats gpt): Bradley-Terry=0.7524  Elo formula=0.7524
NoteKey Insight

The leaderboard number is a fitted log-strength in disguise. “1400 vs 1000” is not a score anyone earned point by point — it is the pair of latent strengths whose logistic difference best predicts a decade of battles, printed in base-10/400 units.

Online Elo Is Gradient Descent on This Loss

Chess never fit a giant logistic regression — it updated one rating at a time:

R_A \;\leftarrow\; R_A + K\,(S_A - E_A),

where S_A \in \{0, 1\} is the outcome and E_A the expected score. That “score minus expectation” is not a heuristic — it is exactly the negative gradient of the Bradley-Terry loss. For one battle where A wins, the loss is \mathcal{L} = -\ln \sigma(\beta_A - \beta_B), and

\frac{\partial \mathcal{L}}{\partial \beta_A} = -\bigl(1 - \sigma(\beta_A-\beta_B)\bigr) = -(S_A - E_A).

So one Elo update is one SGD step on the Bradley-Terry NLL, with learning rate \eta = K \cdot \ln 10 / 400 once you account for the rating scale. We can check the identity to numerical precision with autograd:

import torch
from math import log

bA, bB, K = 0.3, -0.2, 24.0
eta = K * log(10.0) / 400.0                      # Elo K, in β-space

beta = torch.tensor([bA, bB], requires_grad=True)
nll = -torch.nn.functional.logsigmoid(beta[0] - beta[1])   # A won
(grad,) = torch.autograd.grad(nll, beta)
S_A, E_A = 1.0, bt_win_prob(bA, bB)
print(f"∂L/∂βA = {grad[0]:+.5f}   −(S−E) = {-(S_A - E_A):+.5f}")   # identical

# One SGD step in β, mapped to Elo, vs one online-Elo update with factor K:
sgd_elo_A = beta_to_elo(bA - eta * float(grad[0]))
r = {"A": beta_to_elo(bA), "B": beta_to_elo(bB)}
e_a = elo_expected(r["A"], r["B"]); r["A"] += K * (1.0 - e_a)
print(f"SGD→Elo = {sgd_elo_A:.3f}   online Elo = {r['A']:.3f}")
∂L/∂βA = -0.37754   −(S−E) = -0.37754
SGD→Elo = 1061.176   online Elo = 1061.176

The blessing of online Elo — O(1) per battle, no refit — comes with a curse: because each step depends on the current ratings, the final numbers depend on the order the battles arrive in. Shuffle history and a model’s rating moves. That order sensitivity is precisely why an arena publishes the order-free batch MLE for its leaderboard, and keeps online Elo for live, provisional updates.

Interactive: The Live Leaderboard

Watch online Elo find the ratings one battle at a time. Each faint line is a model’s provisional rating as battles stream in; the dashed rules are the order-free MLE it is converging toward. Turn the K-factor up and the ratings snap to their neighborhood faster — but jitter forever, never settling; turn it down and they crawl in, smooth but slow. There is no K that is both fast and calm: that tension is the whole reason the public number is the batch fit, not the last online value.

TipTry This
  1. Chase the curse. Set K to 32 and watch the top two models trade places every few hundred battles even though their MLE lines never cross — the ranking flickers purely from update noise. Drop to 8 and the flicker stops, but early on the lines lag far behind their targets.
  2. Read the fit as the truth. The dashed MLE rules barely move regardless of K — they are computed from all battles at once. That stability is what a leaderboard sells.
  3. Widen the field. In simulate_battles, spread true_beta further apart and the lines separate cleanly; bunch them within ~0.1 and even 3000 battles leave the middle ranks tangled — a hint that some gaps are simply not resolvable yet.

Trust the Gap? Bootstrap the Ratings

A leaderboard that prints “1287 vs 1281” invites a false conclusion: that the first model is better. With finite battles, every rating is an estimate with error, and the honest tool is the bootstrap — resample the battle list with replacement, refit, and repeat, to see how much each rating would have wandered on a different draw of the same size. bootstrap_ci returns a (\text{low}, \text{median}, \text{high}) band per model:

ci = bootstrap_ci(battles, rounds=100, seed=0)
print(f"{'model':<8}{'low':>8}{'median':>9}{'high':>8}{'±width':>9}")
for m in rank_players({k: v[1] for k, v in ci.items()}):
    lo, med, hi = ci[m]
    print(f"{m:<8}{lo:>8.0f}{med:>9.0f}{hi:>8.0f}{(hi - lo) / 2:>9.0f}")
model        low   median    high   ±width
sonnet      1113     1130    1149       18
opus        1067     1085    1099       16
haiku       1008     1024    1037       14
gpt          922      937     953       16
small        805      827     841       18

Two models whose bands overlap are, on this much evidence, a statistical tie — no matter which one sits a few points higher in the table. This is why real leaderboards show intervals and often a rank range, not a bare ordering.

The ✕ marks each model’s true Elo, and every one lands inside its band — the bootstrap is honest. At 3000 battles these five strengths are spread far enough to separate cleanly; shrink the gaps in true_beta (or the battle count) and neighboring bars start to overlap, and any pair whose bands overlap is a statistical tie no matter who sits higher. More battles shrink every bar — the only way to earn a finer ranking.

WarningA rating gap is not a ranking until the bands separate

The single most common misreading of a model leaderboard is treating adjacent rows as a settled order. Before you claim model A beats model B, check that A’s bootstrap interval clears B’s — and remember the whole tower rests on the judge that produced the battles. A biased or unvalidated judge (previous section) yields a beautifully-fit, confidently wrong leaderboard.

Calibration: Does 90% Mean 90%?

Every metric so far grades what the model answered. None of them audits how sure it said it was. That second number matters: a model that is 80% accurate but always reports “99% confident” is miscalibrated — confidently wrong — and a downstream system that trusts its confidence (to abstain, to route to a human, to gate a tool call) will be badly misled. Calibration asks the honesty question directly: of all the times the model said 90%, was it right 90% of the time?

Accuracy cannot see this. Two models can share the same accuracy while one reports honest probabilities and the other reports noise. So confidence gets its own metric — and, it turns out, its own one-parameter fix.

The uncomfortable modern fact: large networks are systematically overconfident. Guo et al. (2017) showed that as networks got deeper and wider their accuracy rose but their calibration degraded — confidence outran correctness. The same happens to LLMs after alignment: a pretrained model can be well-calibrated on multiple-choice questions, and RLHF then pushes its confidences up past its accuracy (the GPT-4 report; Kadavath et al., 2022).

The Math: Reliability and ECE

Take each prediction’s confidence — the probability mass on the class it actually predicted, \text{conf} = \max_i p_i — and sort all predictions into M equal-width bins by that confidence. In bin B_m, compare two numbers: the average confidence \text{conf}(B_m) the model claimed, and the actual accuracy \text{acc}(B_m) it achieved. Perfect calibration is \text{acc}(B_m) = \text{conf}(B_m) in every bin — a straight diagonal on a plot of accuracy against confidence, the reliability diagram. Bars that fall below the diagonal are overconfident.

The Expected Calibration Error collapses that diagram into one number — the average gap, weighted by how many predictions land in each bin:

\text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n}\, \bigl|\, \text{acc}(B_m) - \text{conf}(B_m) \,\bigr|.

ECE is 0 for a perfectly calibrated model and 0.5 for one that is always 100% confident but only right half the time. A companion, the Brier score \frac{1}{n}\sum_i (\text{conf}_i - \text{correct}_i)^2, needs no binning and is a proper scoring rule (Brier, 1950): it is minimized only by reporting true probabilities, so it rewards being accurate and honest at once.

Code: Measuring the Gap

calibration.py implements these as model-free functions over (confidence, correct) arrays — exactly what confidence_and_correctness extracts from logits and labels:

import torch
from calibration import (confidence_and_correctness, bin_predictions,
                         expected_calibration_error, brier_score)

# A tiny 3-class model on 6 examples: sharp logits, but half the argmaxes are wrong.
logits = torch.tensor([
    [4.0, 0.0, 0.0], [4.0, 0.0, 0.0], [4.0, 0.0, 0.0],   # very confident...
    [4.0, 0.0, 0.0], [0.0, 4.0, 0.0], [0.0, 0.0, 4.0],
])
labels = torch.tensor([0, 0, 0, 1, 1, 2])                 # examples 4 & 5 are right, 3 wrong-ish

conf, correct = confidence_and_correctness(logits, labels)
print("confidences:", [round(c, 3) for c in conf.tolist()])
print("correct:    ", correct.tolist())
print("accuracy:   ", round(correct.float().mean().item(), 3))
print("ECE:        ", round(expected_calibration_error(conf, correct, n_bins=10), 3))
print("Brier:      ", round(brier_score(conf, correct), 3))
confidences: [0.965, 0.965, 0.965, 0.965, 0.965, 0.965]
correct:     [True, True, True, False, True, True]
accuracy:    0.833
ECE:         0.131
Brier:       0.156

The model is ~0.98 confident on every example but only two-thirds right — a large ECE, even though nothing is wrong with its accuracy reporting. bin_predictions returns the full reliability table (per-bin count, confidence, accuracy, gap) that the diagram below draws.

Temperature Scaling: the One-Knob Fix

The fix is astonishingly cheap. Divide the logits by a single learned scalar T before the softmax — \text{softmax}(z / T) — and fit T to minimize negative log-likelihood on a held-out set. Because dividing by a positive constant never changes which logit is largest, temperature scaling leaves every prediction — and therefore the accuracy — exactly unchanged. It only rescales the confidences: T > 1 cools an overconfident model; T < 1 sharpens an underconfident one.

from calibration import fit_temperature, demonstrate_calibration

demonstrate_calibration(num_examples=2000, num_classes=5, overconfidence=3.0, seed=0)
============================================================
Calibration: an overconfident model, then temperature scaling
============================================================
  2000 predictions, 5 classes, overconfidence x3.0
  accuracy (unchanged by scaling): 0.460
  fitted temperature T*         : 3.012

    metric    before (T=1)    after (T*)
       ECE          0.2903        0.0200
       MCE          0.3837        0.1407
     BRIER          0.3265        0.2325

  Temperature scaling never touched accuracy - it only cooled the
  confidences until they matched how often the model is actually right.
{'accuracy': 0.4595,
 'fitted_temperature': 3.012009978294363,
 'n_bins': 10,
 'before': {'ece': 0.29028341287374493,
  'mce': 0.38368964261663324,
  'brier': 0.32648228809074326,
  'bins': [{'lo': 0.0,
    'hi': 0.1,
    'count': 0,
    'confidence': 0.0,
    'accuracy': 0.0,
    'gap': 0.0},
   {'lo': 0.1,
    'hi': 0.2,
    'count': 0,
    'confidence': 0.0,
    'accuracy': 0.0,
    'gap': 0.0},
   {'lo': 0.2,
    'hi': 0.30000000000000004,
    'count': 1,
    'confidence': 0.2980251610279083,
    'accuracy': 0.0,
    'gap': -0.2980251610279083},
   {'lo': 0.30000000000000004,
    'hi': 0.4,
    'count': 68,
    'confidence': 0.36610588694320007,
    'accuracy': 0.27941176470588236,
    'gap': -0.0866941222373177},
   {'lo': 0.4,
    'hi': 0.5,
    'count': 193,
    'confidence': 0.45087650047682726,
    'accuracy': 0.33678756476683935,
    'gap': -0.1140889357099879},
   {'lo': 0.5,
    'hi': 0.6,
    'count': 276,
    'confidence': 0.5511944691340128,
    'accuracy': 0.358695652173913,
    'gap': -0.1924988169600998},
   {'lo': 0.6,
    'hi': 0.7,
    'count': 238,
    'confidence': 0.6447008965896959,
    'accuracy': 0.35714285714285715,
    'gap': -0.2875580394468387},
   {'lo': 0.7,
    'hi': 0.8,
    'count': 267,
    'confidence': 0.7528509814641002,
    'accuracy': 0.4044943820224719,
    'gap': -0.34835659944162833},
   {'lo': 0.8,
    'hi': 0.9,
    'count': 359,
    'confidence': 0.8516562164327892,
    'accuracy': 0.467966573816156,
    'gap': -0.38368964261663324},
   {'lo': 0.9,
    'hi': 1.0,
    'count': 598,
    'confidence': 0.9615888880646747,
    'accuracy': 0.6270903010033445,
    'gap': -0.3344985870613302}]},
 'after': {'ece': 0.02000890847295522,
  'mce': 0.14066285557217062,
  'brier': 0.2325359127870524,
  'bins': [{'lo': 0.0,
    'hi': 0.1,
    'count': 0,
    'confidence': 0.0,
    'accuracy': 0.0,
    'gap': 0.0},
   {'lo': 0.1,
    'hi': 0.2,
    'count': 0,
    'confidence': 0.0,
    'accuracy': 0.0,
    'gap': 0.0},
   {'lo': 0.2,
    'hi': 0.30000000000000004,
    'count': 148,
    'confidence': 0.2792638396290508,
    'accuracy': 0.30405405405405406,
    'gap': 0.024790214425003232},
   {'lo': 0.30000000000000004,
    'hi': 0.4,
    'count': 580,
    'confidence': 0.35219184468532433,
    'accuracy': 0.3413793103448276,
    'gap': -0.010812534340496727},
   {'lo': 0.4,
    'hi': 0.5,
    'count': 620,
    'confidence': 0.4469357417475793,
    'accuracy': 0.44193548387096776,
    'gap': -0.0050002578766115136},
   {'lo': 0.5,
    'hi': 0.6,
    'count': 323,
    'confidence': 0.5440515440314916,
    'accuracy': 0.5386996904024768,
    'gap': -0.005351853629014802},
   {'lo': 0.6,
    'hi': 0.7,
    'count': 186,
    'confidence': 0.6431230668739606,
    'accuracy': 0.6827956989247311,
    'gap': 0.03967263205077054},
   {'lo': 0.7,
    'hi': 0.8,
    'count': 99,
    'confidence': 0.7467234616327767,
    'accuracy': 0.6060606060606061,
    'gap': -0.14066285557217062},
   {'lo': 0.8,
    'hi': 0.9,
    'count': 41,
    'confidence': 0.8365425816396388,
    'accuracy': 0.926829268292683,
    'gap': 0.09028668665304418},
   {'lo': 0.9,
    'hi': 1.0,
    'count': 3,
    'confidence': 0.919233242670695,
    'accuracy': 1.0,
    'gap': 0.08076675732930505}]}}

The demo builds an honest model, makes it overconfident by scaling its logits ×3, then fits T. The fit recovers T^\* \approx 3 — it finds the exact factor the model was overconfident by — and ECE collapses from ~0.29 to ~0.02 while accuracy does not move a hair. Temperature scaling didn’t make the model smarter; it made its confidence honest.

import torch
from calibration import fit_temperature

# Bridge a fixed overconfident set to the widget: honest logits, sampled labels, ×3.
_g = torch.Generator().manual_seed(0)
_true = torch.randn(1500, 5, generator=_g)
_labels = torch.multinomial(torch.softmax(_true, dim=-1), 1, generator=_g).squeeze(-1)
_logits = _true * 3.0
_Tstar = fit_temperature(_logits, _labels)
ojs_define(calibLogits = _logits.tolist(),
           calibLabels = _labels.tolist(),
           calibTstar  = float(_Tstar))

Interactive Exploration: The Reliability Dial

Drag the temperature. At T = 1 the overconfident model’s bars sag far below the diagonal — every bin claims more confidence than it earns. Cool it and the bars climb onto the line; the ECE readout bottoms out right at T^\*; push past it and the model turns underconfident and the bars overshoot. The accuracy never changes — only the honesty of the numbers does.

The gap segment on each bar is the miscalibration the reader can see: red where the bar is overconfident (accuracy below the claimed confidence), shrinking to nothing as T approaches T^\*. The curve below traces ECE across every temperature, so the minimum — and how sharply the fit lands on it — is visible at a glance:

NoteKey Insight

Calibration is a second axis of quality, orthogonal to accuracy. A model can be accurate and dishonest about it. Measure the gap with ECE (or Brier), and close it with temperature scaling — one scalar that rescales confidence while leaving every prediction, and therefore the accuracy, exactly where it was. Where the ECE curve bottoms out is T^\*, and for an overconfident model T^\* > 1.

TipTry This
  1. Find the overconfidence factor. Cool the dial until the bars snap onto the diagonal. The temperature you land on is the fitted T^\* \approx 3 — the exact factor this model was overconfident by.
  2. Over-cool it. Push T past T^\* toward 6. The bars now rise above the diagonal: the model has become underconfident, and ECE climbs back up. There is a single sweet spot, not “colder is better.”
  3. Watch accuracy. Read the “accuracy = … (never moves)” line as you drag. No temperature ever changes it — proof that calibration and accuracy are independent.

Contamination

Every metric above assumes the test set is unseen. If a benchmark leaked into the training data — and web-scraped corpora are full of leaked benchmarks — the model can recite the answer, and the score measures memorization, not skill. A cheap first check is n-gram overlap: what fraction of a test example’s n-grams also appear in the training text?

from evaluation import ngram_overlap

test_q = "the mitochondria is the powerhouse of the cell"
clean_train = "cells contain many organelles with distinct roles"
leaked_train = "biology fact: the mitochondria is the powerhouse of the cell"

print("overlap vs clean corpus: ", round(ngram_overlap(test_q, clean_train, n=5), 2))
print("overlap vs leaked corpus:", round(ngram_overlap(test_q, leaked_train, n=5), 2))
overlap vs clean corpus:  0.0
overlap vs leaked corpus: 1.0

A near-1 overlap against a training shard means the example is contaminated and its score should be discarded. Real pipelines run this at scale (e.g. 13-gram or 50-char matches) across the whole corpus before trusting a benchmark.

Intuition: Emergence, or a Metric Mirage?

Contamination was one way a metric misleads. Here is a subtler, more famous one — where the metric doesn’t just misrank models, it invents a phenomenon.

Plot a hard task’s benchmark score against training compute and you often see a cliff: near-zero for every small model, then — past some scale — a sudden jump to competence. The capability looks absent below a threshold and present above it. This is an emergent ability (Wei et al., 2022): a skill “not present in smaller models but present in larger models,” seemingly unlocked all at once, and — the unsettling part — unpredictable from the smaller models’ scores.

But there are two very different explanations for a cliff, and the plot alone can’t tell them apart:

  • The model is discontinuous. Something qualitatively new switches on at scale.
  • The ruler is discontinuous. The model improves smoothly — exactly as the scaling laws of m07 predict — and the metric turns that smooth gain into an apparent jump.

Schaeffer, Miranda & Koyejo (2023) made the deflationary case: for many reported emergent abilities, it’s the ruler. Their argument is not hand-waving — it’s one line of algebra you can watch manufacture and then dissolve a cliff. That’s this section.

The Math: One Smooth Skill, Two Rulers

Model the underlying skill as per-token accuracy p(C) — the chance the model gets a single token right, as a function of training compute C. Scaling laws say this rises smoothly: loss falls as a power law, so p climbs gently with no jump. We’ll use a smooth logistic in \log_{10} C for p(C); the exact shape doesn’t matter, only that it is continuous.

Now grade a task whose answer is L tokens long, with two different metrics.

A nonlinear ruler — exact match. Many benchmarks demand the whole answer be right: exact string match, or multiple-choice where one wrong token fails the item. If tokens are right independently with probability p, the chance all L are right is

\text{exact\_match}(p, L) = p^{L}.

For a long answer this is microscopic until p is very close to 1, then it rockets upward — a manufactured cliff. At p = 0.9 and L = 30, exact match is 0.9^{30} \approx 4\%; nudge p to 0.99 and it leaps to 0.99^{30} \approx 74\%. A small, smooth gain in the skill becomes a huge, sudden gain in the score.

A linear ruler — token edit similarity. Credit partial progress: score the fraction of tokens correct (equivalently, 1 minus the normalized edit distance). Its expectation is simply

\text{edit\_similarity}(p) = p ,

which is the smooth skill curve — no cliff, ever. Same for expected edit distance L(1-p), the unnormalized twin.

That is the entire trick. The same p(C) produces an emergent-looking jump under p^{L} and a boring smooth ramp under p. Drive L and watch the cliff appear:

The blue curve is a textbook “emergent ability”; the orange curve is the same model. Only the ruler changed. Slide L to 1 and the two coincide exactly — with a single-token answer there is no all-or-nothing penalty, so exact match is the linear metric and no mirage is possible. Every extra required token bends the nonlinear ruler further.

NoteKey Insight

A sharp “emergence” plot has two readings — a discontinuous model or a discontinuous metric — and they look identical on the axes people usually show. Because \text{exact\_match} = p^{L} is nonlinear in a smoothly-improving skill, a continuous capability gain routinely renders as a cliff. The jump is real on the plot and absent in the model.

Code: Manufacture (and Dissolve) an Emergence

emergence.py builds exactly the two rulers above and puts them on one smooth skill curve. demonstrate_mirage sweeps compute once, reads the same p(C) at every point, and scores it both ways:

import importlib.util, sys
from pathlib import Path

spec = importlib.util.spec_from_file_location("emergence", Path("emergence.py").resolve())
emergence = importlib.util.module_from_spec(spec)
sys.modules["emergence"] = emergence
spec.loader.exec_module(emergence)

demo = emergence.demonstrate_mirage(length=30)

# Same underlying skill, two rulers — report a few points along the sweep.
print(f"{'log10 C':>8} | {'skill p':>8} | {'exact pᴸ':>9} | {'edit p':>7}")
for c, p, em in list(zip(demo["computes"], demo["skill"], demo["exact_match"]))[::30]:
    import math
    print(f"{math.log10(c):>8.1f} | {p:>8.3f} | {em:>9.3f} | {p:>7.3f}")

print(f"\nexact-match transition width: {demo['exact_width']:.2f} decades of compute")
print(f"linear-metric transition width: {demo['linear_width']:.2f} decades of compute")
print("→ the nonlinear ruler looks", round(demo['linear_width']/demo['exact_width'], 2),
      "× sharper on the *identical* model")
 log10 C |  skill p |  exact pᴸ |  edit p
    12.0 |    0.000 |     0.000 |   0.000
    15.0 |    0.000 |     0.000 |   0.000
    18.0 |    0.001 |     0.000 |   0.001
    21.0 |    0.500 |     0.000 |   0.500
    24.0 |    0.999 |     0.970 |   0.999
    27.0 |    1.000 |     1.000 |   1.000
    30.0 |    1.000 |     1.000 |   1.000

exact-match transition width: 1.36 decades of compute
linear-metric transition width: 1.91 decades of compute
→ the nonlinear ruler looks 1.41 × sharper on the *identical* model

The two score columns are computed from the same skill column — exact is just skill ** 30. The transition_width helper measures how many decades of compute each curve takes to climb from 10% to 90% of its range: the nonlinear ruler crosses in a narrow band (a “phase transition”), the linear one in a gentle ramp.

How sharp is the cliff? It’s a property of L, not the model.

Push L up and the same smooth skill looks ever more abruptly emergent. Below, sharpness_sweep measures the exact-match transition width for a range of answer lengths — bridged into the plot. Watch the width shrink as L grows:

sweep = emergence.sharpness_sweep([1, 2, 4, 8, 16, 32, 64])
ojs_define(
    sweepL=[r["length"] for r in sweep],
    sweepWidth=[r["width"] for r in sweep],
)

At L = 1 the width equals the linear metric’s own smooth ramp (no mirage); each doubling of the required length narrows the apparent transition. “The capability emerged at 10^{23} FLOPs” is, on this reading, a statement about the answer length the metric demands, not about the model.

The sampling artifact

There is a second, compounding trick: model families are sampled at only a handful of scales (a few dots on a log axis). Connect a nonlinear curve’s sparse samples with straight lines and even a smooth-underneath ramp reads as a literal step. Drive the number of sampled models:

With just 3–5 sampled models the connected sparse line looks like a flat floor and a cliff; sample densely and the same curve reveals a continuous ramp. Sparse sampling and a nonlinear metric reinforce each other — which is exactly the regime most emergence plots live in.

WarningThis is not “emergence is fake”

The argument is narrower and more useful than that. It says: a discontinuous metric manufactures the appearance of a sharp jump from a smooth curve, so a cliff on a nonlinear metric is not evidence of a discontinuous capability. It does not prove that no ability is ever genuinely sharp. The discipline it demands: before claiming emergence, re-score with a continuous metric (token edit distance, Brier score, per-token log-likelihood) and sample scale densely. If the jump survives that, you may have found something real; if it dissolves, you found a property of your ruler.

Common Pitfalls

When evaluating models, watch out for:

  1. Reporting a metric without its normalization / prompt. Exact-match accuracy depends entirely on the normalization spec and the prompt format. Two papers’ “MMLU 70%” can be incomparable. State exactly how you scored.
  2. Using the biased pass@k. 1−(1−c/n)^k is not pass@k; it drifts from the unbiased combinatorial estimator for k>1. Use the combinatorial form (or its stable product) that HumanEval defines.
  3. Trusting an LLM judge unchecked. Position, verbosity, and self-preference biases are real. Swap the order, validate against humans, and never let a model grade its own outputs unaudited.
  4. Reading a leaderboard gap as a ranking. Elo/Bradley-Terry ratings are estimates with error. Adjacent rows separated by a few points are usually a statistical tie — check that the bootstrap intervals clear each other, and prefer online Elo’s order-free MLE over its last provisional value.
  5. Ignoring contamination. A sky-high score on a public benchmark is a red flag, not a triumph — check n-gram overlap against training before believing it.
  6. Goodhart’s law. “When a measure becomes a target, it ceases to be a good measure.” Optimizing directly for a benchmark (or a reward model, m12) produces models that ace the metric and fail the task. Hold out a fresh test set.
  7. Single-number reductionism. One aggregate hides per-category collapses (great on easy items, zero on hard ones). Break scores down before declaring victory.
  8. Reading ECE without its bin count. ECE depends on the number of bins — too few hides miscalibration, too many leaves bins empty and noisy. Report n_bins, and don’t compare ECE values computed with different binning.
  9. Assuming an aligned model is calibrated. A pretrained model can be well-calibrated on multiple choice; RLHF routinely makes it overconfident. Kadavath et al. (2022) found this is largely a reversible temperature distortion — a single T ≈ 2.5 restores much of the calibration — so check ECE and temperature-scale before trusting confidences.

Exercises

Exercise 1: The pass@k break-even

from evaluation import pass_at_k

# A model passes each sample with probability 0.1 (so c ≈ 0.1·n). How many samples k
# does pass@k need to first exceed 0.5? Try n = 100, c = 10, and loop k. Then explain
# why sampling is cheaper than making the model 5× better per-sample.

# Your implementation here:

Exercise 2: Normalization matters

from evaluation import accuracy, normalize_answer

# Given preds = ["The answer is 42.", "42", "forty two"] and golds = ["42","42","42"],
# compute accuracy. Then write a stricter normalizer (numbers only) and a looser one
# (word2number) and show how the SAME predictions score differently. Which is "right"?

# Your implementation here:

Exercise 3: Debias the judge

from evaluation import demonstrate_judge, position_bias

# demonstrate_judge exposes a position-biased judge. A standard fix is to average the
# two orderings into one verdict (a win only counts if it survives the swap). Simulate
# this "consistency-required" scoring and show it removes the slot-A inflation even at
# bias_strength=0.6.

# Your implementation here:

Exercise 4: Find the emergence threshold

from emergence import smooth_capability, exact_match_rate

# For a fixed smooth skill p(compute), the answer length L controls where exact-match
# "emerges". Sweep compute (1e14..1e28) and, for L in {5, 20, 80}, find the compute at
# which exact_match first crosses 50%. Show the "threshold" marches rightward as L grows
# — evidence the threshold is a property of the metric, not a new capability.

# Your implementation here:

Exercise 5: Dissolve a cliff with a continuous metric

from emergence import demonstrate_mirage

# demonstrate_mirage returns exact_width and linear_width (transition widths in decades).
# Confirm the nonlinear ruler looks sharper (smaller width) for length in {10, 40, 100},
# and print the sharpness ratio linear_width / exact_width. Then argue in one sentence
# what a *genuine* emergent ability would have to look like under the linear metric.

# Your implementation here:

Exercise 6: Order sensitivity of online Elo

from arena import simulate_battles, elo_ratings, bradley_terry_mle, kendall_tau, beta_to_elo
import random

# Online Elo depends on battle order; the batch MLE does not. Simulate one battle set,
# then shuffle it 20 different ways. For each shuffle, run elo_ratings and record the top
# model. Show the online winner sometimes changes, while bradley_terry_mle (recomputed on
# any shuffle) always returns the same ratings. Quantify with kendall_tau vs the MLE order.

# Your implementation here:

Exercise 7: Temperature scaling never changes accuracy

from calibration import confidence_and_correctness, expected_calibration_error, fit_temperature
import torch

# Build an overconfident model (honest logits x4, sampled labels). For T in
# {1, T*, 4}, print accuracy and ECE. Confirm accuracy is IDENTICAL for all three
# temperatures while ECE is minimized at the fitted T* — the whole point of the
# method. (Hint: confidence_and_correctness(logits, labels, T) gives both arrays.)

# Your implementation here:

Summary

Key takeaways:

  1. The metric is part of the claim. Intrinsic (perplexity), closed-form (exact match), functional (pass@k), and judged (win rate) each fit a different task. A number without its metric says nothing.
  2. Normalize before you match. Trailing punctuation, articles, and case sink correct answers; every benchmark ships a normalization spec for this reason.
  3. pass@k is unbiased only in the combinatorial form. 1 - \binom{n-c}{k}/\binom{n}{k} — not 1-(1-c/n)^k — estimates the chance that one of k code samples passes. More samples raise pass@k, which is why reasoning systems sample and aggregate.
  4. LLM-as-judge is scalable but biased. Position bias flips the winner when you swap the order; always run both orders and validate against humans.
  5. A leaderboard is a fit, not a tally. Bradley-Terry gives each model a latent strength with P(i beats j) = σ(β_i − β_j); fitting it is logistic regression over battles. Elo is the same model in base-10/400 units (R = 1000 + β·400/ln10), and online Elo’s K(S − E) update is one SGD step on that loss. Read gaps through bootstrap intervals — overlapping bands are a tie.
  6. Calibration is a second axis, orthogonal to accuracy. ECE — the bin-weighted gap between confidence and accuracy on a reliability diagram — measures whether “90%” means 90%. Modern nets, and LLMs after RLHF, tend to be overconfident.
  7. Temperature scaling is the one-knob fix. Dividing logits by a scalar T before the softmax rescales confidence without moving any argmax, so it lowers ECE while leaving accuracy untouched; fitting T recovers exactly how overconfident the model was.
  8. Contamination invalidates everything. If the test leaked into training, the score measures memorization. Check n-gram overlap before you believe a benchmark.
  9. Goodhart looms. Optimizing the metric destroys the metric — hold out a fresh test set and read scores skeptically.
  10. A cliff can be the ruler, not the model. Because exact_match = pᴸ is nonlinear in a smoothly-improving skill, a continuous capability gain routinely renders as a sharp “emergent” jump. The transition sharpens with the required answer length L, not with any change in the model.
  11. Adjudicate emergence with a continuous metric. Re-score with token edit distance, Brier score, or per-token log-likelihood, and sample scale densely. If the jump survives, it may be real; if it dissolves, it was a property of the metric.

What’s Next

You can now measure a model as well as build one — the last missing piece of the loop. From here the book turns to putting models to work: tool use and agents (models that act, not just answer) and interpretability (opening the box to see why a model scores the way it does). Evaluation is the thread through all of it: every new capability is only as real as the metric that confirms it.

Going Deeper

Core Papers:

Practical Resources: