{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Module 17: Evaluation" ], "id": "b3c56bcf-b1bd-4266-bc20-b0a704794e72" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [], "id": "af62101a-45cf-4497-ac52-019a88ffd743" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "695d2d59-ffc5-4f16-91ed-be9de178f6a2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "Every module so far *built* capability: tokenize (m03), attend (m05),\n", "train (m07), align (m12), reason (m13). This one builds the discipline\n", "that keeps all of it honest — **evaluation**. The moment you claim a\n", "model is “better,” you owe an answer to *measured how?* — and the wrong\n", "metric can make a worse model look better.\n", "\n", "Evaluation is genuinely hard, for three reasons this module takes\n", "head-on:\n", "\n", "- **Different tasks need different metrics.** “What is 6×7?” has one\n", " right answer; “write a function that sorts a list” has infinitely many\n", " correct programs; “which essay is better?” has no ground truth at all.\n", " One number does not fit them.\n", "- **The obvious formula is often biased.** For code, the standard metric\n", " **pass@k** has an unbiased estimator that is *not* the tempting\n", " `1−(1−c/n)^k` — the same kind of “the naive version is subtly wrong”\n", " story as m13’s Condorcet vote.\n", "- **Scores lie when the test leaks.** If a benchmark appeared in the\n", " training data (**contamination**), every number on it is inflated, and\n", " you would never know without checking.\n", "\n", "Why it matters for LLMs: leaderboards, ablations, and every “we improved\n", "X” in the book rest on evaluation. Build the metrics from scratch and\n", "you can read any model card critically — and never be fooled by a\n", "good-looking number again.\n", "\n", "### What You’ll Learn\n", "\n", "After this module, you can:\n", "\n", "- Explain why evaluation is hard — task-metric mismatch, Goodhart’s law,\n", " and contamination — and pick the right metric family for a task.\n", "- Build **exact-match accuracy** with proper answer **normalization**,\n", " and see how a missing normalization step silently costs correct\n", " answers their points.\n", "- Derive and implement **pass@k** — the *unbiased*\n", " functional-correctness estimator for code — and see exactly how the\n", " naive estimator misleads.\n", "- Use an **LLM-as-judge** for open-ended answers, and measure its\n", " **position bias** by swapping the order and watching the winner flip.\n", "- Detect **contamination** with n-gram overlap between a test example\n", " and training text.\n", "\n", "### Prerequisites\n", "\n", "This module requires familiarity with:\n", "\n", "- [Module 07: Training](../m07_training/lesson.qmd) — cross-entropy loss\n", " and **perplexity**, the intrinsic metric this module recaps and moves\n", " beyond.\n", "- [Module 08: Generation](../m08_generation/lesson.qmd) — the `generate`\n", " loop that produces the samples pass@k and the judge score.\n", "- [Module 13: Reasoning](../m13_reasoning/lesson.qmd) — verifiers and\n", " answer extraction; evaluation formalizes “did it check out?” into a\n", " benchmark number.\n", "\n", "## Intuition: What Are We Even Measuring?\n", "\n", "A metric is a bet about what “correct” means. Match the bet to the task\n", "and the number is meaningful; mismatch it and the number is noise. Four\n", "families cover most of what you’ll ever report — step through them:" ], "id": "a21ddb7e-33e1-4387-b5ca-a7b4197b73d5" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "49902811-9663-4b28-8723-2684875b6ce2" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "a06b596a-54ea-4210-8a21-e858fe5f38dc" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "05f18dad-3569-4516-95a3-e38500d1e156" }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Key Insight**\n", ">\n", "> The metric is part of the claim. “GPT-X scores 90%” is meaningless\n", "> until you know *which* 90% — exact-match on MMLU, pass@1 on HumanEval,\n", "> or win-rate judged by another model. Report the metric, or the number\n", "> says nothing.\n", "\n", "## Exact Match & Normalized Accuracy\n", "\n", "The simplest metric, for tasks with one right answer: does the\n", "prediction equal the gold answer? The catch is **normalization**. Models\n", "add trailing periods, articles, and stray capitalization; compare raw\n", "strings and a *correct* answer scores zero. `normalize_answer`\n", "lowercases, strips punctuation and articles, and collapses whitespace\n", "before comparing — the same idea as m13’s `extract_answer`, now as a\n", "scoring step:" ], "id": "399fd1a7-2afd-4023-8c44-3763fb4afbdb" }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "answer is 42\n", "True\n", "False\n", "accuracy: 1.0" ] } ], "source": [ "from evaluation import normalize_answer, exact_match, accuracy\n", "\n", "print(normalize_answer(\"The Answer is 42.\")) # -> 'answer is 42'\n", "print(exact_match(\"42.\", \"42\")) # True after normalizing\n", "print(exact_match(\"forty-two\", \"42\")) # still False — different tokens\n", "\n", "preds = [\"42.\", \"Paris\", \"the dog\"]\n", "golds = [\"42\", \"paris\", \"a dog\"]\n", "print(\"accuracy:\", accuracy(preds, golds)) # 3/3 — normalization saves all three" ], "id": "12284efe" }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Normalization is not neutral**\n", ">\n", "> Normalization is a *choice* that changes the score. Strip too little\n", "> and “42.” is wrong; strip too much and “not 42” normalizes toward\n", "> “42”. Every benchmark ships an exact normalization spec for this\n", "> reason — report yours.\n", "\n", "## The Math: pass@k\n", "\n", "Exact match works when there is one right string. **Code** breaks that:\n", "infinitely many programs sort a list, and none of them is a target\n", "string to match. So you don’t grade the text — you **run the unit\n", "tests**. That gives functional correctness, and the standard metric is\n", "**pass@k**: sample many completions, and ask how likely it is that at\n", "least one of `k` of them passes.\n", "\n", "Concretely: draw `n` samples for a problem, run the tests, and count the\n", "`c` that pass. If you had reported only `k` of those `n` (drawn without\n", "replacement), the chance **all `k` fail** is\n", "$\\binom{n-c}{k} / \\binom{n}{k}$, so\n", "\n", "$$\\text{pass@}k = 1 - \\frac{\\binom{n-c}{k}}{\\binom{n}{k}}.$$\n", "\n", "The tempting shortcut $1 - (1 - c/n)^k$ is **biased** — it samples\n", "*with* replacement, treating each of the `k` draws as independent. The\n", "combinatorial form is unbiased. Drive `c`, `n`, and `k` and watch the\n", "two curves diverge:" ], "id": "627e9a31-4147-4dd4-a293-57ca434dad4f" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} }, { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "7516c16e-10ea-4f18-9527-de224f13ee58" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} }, { "output_type": "display_data", "metadata": {}, "data": {} }, { "output_type": "display_data", "metadata": {}, "data": {} }, { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "0f532286-785e-4338-b9b6-ed5cec08564c" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "836a505f-5982-47be-b64c-53d723f7e587" }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Try This**\n", ">\n", "> 1. **Rare successes.** Set `c` small (say 2 of 200). pass@1 is tiny,\n", "> but pass@100 is large — sampling many times rescues a model that\n", "> is usually wrong but occasionally right. This is why reasoning\n", "> systems (m13) sample and aggregate.\n", "> 2. **Watch the bias.** With `c/n ≈ 0.25`, the biased curve sits\n", "> *below* the correct one at mid-`k` — report the biased number and\n", "> you understate the model. At `k=1` the two agree exactly (both\n", "> equal `c/n`).\n", "\n", "## Code: Functional Correctness\n", "\n", "`evaluation.py` implements the stable product form (no giant binomials)\n", "and averages over problems — exactly what “HumanEval pass@k” means:" ], "id": "02da1f7c-d4a4-47b9-b005-b824db020a2b" }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "pass@1: 0.25\n", "pass@10: 0.9837\n", "benchmark pass@1: 0.2833" ] } ], "source": [ "from evaluation import pass_at_k, estimate_pass_at_k\n", "\n", "# One problem: 5 of 20 samples passed the tests.\n", "print(\"pass@1: \", round(pass_at_k(20, 5, 1), 4)) # = c/n = 0.25\n", "print(\"pass@10:\", round(pass_at_k(20, 5, 10), 4)) # many tries -> much higher\n", "\n", "# A benchmark: average pass@k over all its problems.\n", "n_per = [20, 20, 20] # samples per problem\n", "c_per = [5, 0, 12] # how many passed each\n", "print(\"benchmark pass@1:\", round(estimate_pass_at_k(n_per, c_per, 1), 4))" ], "id": "6405a943" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`demonstrate_pass_at_k` sweeps `k` for a model whose samples pass\n", "independently with some probability, reporting the unbiased estimate\n", "beside the biased one:" ], "id": "d75a484f-e898-4518-a3f1-78a620bf3d47" }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "============================================================\n", "pass@k: unbiased vs. biased estimator\n", "============================================================\n", " n=20 samples, 5/20 pass (per-sample p=0.25)\n", "\n", " k unbiased biased gap\n", " 1 0.2500 0.2500 +0.0000\n", " 2 0.4474 0.4375 -0.0099\n", " 5 0.8063 0.7627 -0.0436\n", " 10 0.9837 0.9437 -0.0401\n", " 20 1.0000 0.9968 -0.0032\n", "\n", " The naive (biased) estimator samples WITH replacement and\n", " systematically misestimates; the combinatorial form is unbiased." ] } ], "source": [ "from evaluation import demonstrate_pass_at_k\n", "\n", "curve = demonstrate_pass_at_k(n=20, per_sample_p=0.25)" ], "id": "08dc6f50" }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Bridge the sweep to the plot below.\n", "pak_points = [\n", " {\"k\": k, \"unbiased\": v[\"unbiased\"], \"biased\": v[\"biased\"]}\n", " for k, v in curve.items()\n", "]\n", "ojs_define(pak_points = pak_points)" ], "id": "644a93e1" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "cb313c79-0c00-476c-afdd-7e659d1d3329" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## LLM-as-Judge\n", "\n", "For open-ended answers — essays, chat, summaries — there is no gold\n", "string and no test to run. The scalable modern answer is\n", "**LLM-as-judge**: show a strong model two answers and ask which is\n", "better, then report a **win rate**. It is cheap and correlates with\n", "human preference — but it has biases, and the sharpest is **position\n", "bias**: the judge can favor whichever answer is shown *first*,\n", "irrespective of quality.\n", "\n", "You catch it by running every comparison **both ways** — original order\n", "and swapped. A judge that grades the *answers* names the same answer\n", "both times (so its A/B label *flips* when the order flips); a judge that\n", "grades the *position* keeps the same label. `position_bias` measures how\n", "often the judge picked the same slot both ways:" ], "id": "e4f95f14-5a76-455f-9f78-cfafb38ea660" }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "fair position bias: 0.0\n", "biased position bias: 1.0\n", "agreement: 0.75" ] } ], "source": [ "from evaluation import pairwise_win_rate, position_bias, judge_agreement\n", "\n", "# Answer-consistent judge: label flips with the order (A->B) => no position bias.\n", "print(\"fair position bias:\", position_bias([\"A\", \"A\", \"B\"], [\"B\", \"B\", \"A\"]))\n", "# Slot-biased judge: always picks whatever is shown first.\n", "print(\"biased position bias:\", position_bias([\"A\", \"A\", \"A\"], [\"A\", \"A\", \"A\"]))\n", "\n", "# Validity check: does the judge agree with human labels?\n", "print(\"agreement:\", round(judge_agreement([\"A\", \"B\", \"A\", \"B\"], [\"A\", \"B\", \"B\", \"B\"]), 2))" ], "id": "29bfb548" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`demonstrate_judge` simulates a judge with a tunable bias and measures\n", "the fallout — how a biased judge inflates the win rate of whichever\n", "answer sits in the first slot:" ], "id": "bf64b7a3-da49-4863-b542-ade93f530237" }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "bias=0.0 position_bias=0.00 slot-A win rate=0.51\n", "bias=0.2 position_bias=0.20 slot-A win rate=0.61\n", "bias=0.4 position_bias=0.39 slot-A win rate=0.70\n", "bias=0.6 position_bias=0.59 slot-A win rate=0.80\n", "bias=0.8 position_bias=0.81 slot-A win rate=0.90\n", "bias=1.0 position_bias=1.00 slot-A win rate=1.00" ] } ], "source": [ "from evaluation import demonstrate_judge\n", "\n", "results = [\n", " {\"bias\": b, **demonstrate_judge(num_pairs=400, bias_strength=b, seed=0, verbose=False)}\n", " for b in [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]\n", "]\n", "for r in results:\n", " print(f\"bias={r['bias']:.1f} position_bias={r['position_bias']:.2f} \"\n", " f\"slot-A win rate={r['slotA_win_rate']:.2f}\")" ], "id": "2bfc1f87" }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "ojs_define(judge_points = results)" ], "id": "d0f3301b" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "0dac2d6f-fef0-4c14-b8fa-07f8089125a6" }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Key Insight**\n", ">\n", "> An unvalidated judge is a rumor. Always (1) swap the order and require\n", "> the verdict to hold, and (2) check agreement with human labels on a\n", "> sample. A win rate from a judge that flips with position is measuring\n", "> layout, not quality.\n", "\n", "## Contamination\n", "\n", "Every metric above assumes the test set is *unseen*. If a benchmark\n", "leaked into the training data — and web-scraped corpora are full of\n", "leaked benchmarks — the model can recite the answer, and the score\n", "measures memorization, not skill. A cheap first check is **n-gram\n", "overlap**: what fraction of a test example’s n-grams also appear in the\n", "training text?" ], "id": "1d13655f-1537-48c2-972a-b11b9a20a4c9" }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "overlap vs clean corpus: 0.0\n", "overlap vs leaked corpus: 1.0" ] } ], "source": [ "from evaluation import ngram_overlap\n", "\n", "test_q = \"the mitochondria is the powerhouse of the cell\"\n", "clean_train = \"cells contain many organelles with distinct roles\"\n", "leaked_train = \"biology fact: the mitochondria is the powerhouse of the cell\"\n", "\n", "print(\"overlap vs clean corpus: \", round(ngram_overlap(test_q, clean_train, n=5), 2))\n", "print(\"overlap vs leaked corpus:\", round(ngram_overlap(test_q, leaked_train, n=5), 2))" ], "id": "06d24fc6" }, { "cell_type": "markdown", "metadata": {}, "source": [ "A near-1 overlap against a training shard means the example is\n", "contaminated and its score should be discarded. Real pipelines run this\n", "at scale (e.g. 13-gram or 50-char matches) across the whole corpus\n", "before trusting a benchmark.\n", "\n", "## Common Pitfalls\n", "\n", "When evaluating models, watch out for:\n", "\n", "1. **Reporting a metric without its normalization / prompt.**\n", " Exact-match accuracy depends entirely on the normalization spec and\n", " the prompt format. Two papers’ “MMLU 70%” can be incomparable. State\n", " exactly how you scored.\n", "2. **Using the biased pass@k.** `1−(1−c/n)^k` is not pass@k; it drifts\n", " from the unbiased combinatorial estimator for `k>1`. Use the\n", " combinatorial form (or its stable product) that HumanEval defines.\n", "3. **Trusting an LLM judge unchecked.** Position, verbosity, and\n", " self-preference biases are real. Swap the order, validate against\n", " humans, and never let a model grade its own outputs unaudited.\n", "4. **Ignoring contamination.** A sky-high score on a public benchmark\n", " is a *red flag*, not a triumph — check n-gram overlap against\n", " training before believing it.\n", "5. **Goodhart’s law.** “When a measure becomes a target, it ceases to\n", " be a good measure.” Optimizing directly for a benchmark (or a reward\n", " model, m12) produces models that ace the metric and fail the task.\n", " Hold out a fresh test set.\n", "6. **Single-number reductionism.** One aggregate hides per-category\n", " collapses (great on easy items, zero on hard ones). Break scores\n", " down before declaring victory.\n", "\n", "## Exercises\n", "\n", "### Exercise 1: The pass@k break-even" ], "id": "1a7ed3b9-2669-45fc-adf4-9995e33b4bfd" }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "from evaluation import pass_at_k\n", "\n", "# A model passes each sample with probability 0.1 (so c ≈ 0.1·n). How many samples k\n", "# does pass@k need to first exceed 0.5? Try n = 100, c = 10, and loop k. Then explain\n", "# why sampling is cheaper than making the model 5× better per-sample.\n", "\n", "# Your implementation here:" ], "id": "70bd3439" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2: Normalization matters" ], "id": "176fdbd9-b302-4fad-bded-9dbb4b23e6df" }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from evaluation import accuracy, normalize_answer\n", "\n", "# Given preds = [\"The answer is 42.\", \"42\", \"forty two\"] and golds = [\"42\",\"42\",\"42\"],\n", "# compute accuracy. Then write a stricter normalizer (numbers only) and a looser one\n", "# (word2number) and show how the SAME predictions score differently. Which is \"right\"?\n", "\n", "# Your implementation here:" ], "id": "ff17e935" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3: Debias the judge" ], "id": "e1b6e947-168b-458d-ab25-dfd2700d79db" }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "from evaluation import demonstrate_judge, position_bias\n", "\n", "# demonstrate_judge exposes a position-biased judge. A standard fix is to average the\n", "# two orderings into one verdict (a win only counts if it survives the swap). Simulate\n", "# this \"consistency-required\" scoring and show it removes the slot-A inflation even at\n", "# bias_strength=0.6.\n", "\n", "# Your implementation here:" ], "id": "c58ef36d" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "Key takeaways:\n", "\n", "1. **The metric is part of the claim.** Intrinsic (perplexity),\n", " closed-form (exact match), functional (pass@k), and judged (win\n", " rate) each fit a different task. A number without its metric says\n", " nothing.\n", "2. **Normalize before you match.** Trailing punctuation, articles, and\n", " case sink correct answers; every benchmark ships a normalization\n", " spec for this reason.\n", "3. **pass@k is unbiased only in the combinatorial form.**\n", " $1 - \\binom{n-c}{k}/\\binom{n}{k}$ — *not* $1-(1-c/n)^k$ — estimates\n", " the chance that one of `k` code samples passes. More samples raise\n", " pass@k, which is why reasoning systems sample and aggregate.\n", "4. **LLM-as-judge is scalable but biased.** Position bias flips the\n", " winner when you swap the order; always run both orders and validate\n", " against humans.\n", "5. **Contamination invalidates everything.** If the test leaked into\n", " training, the score measures memorization. Check n-gram overlap\n", " before you believe a benchmark.\n", "6. **Goodhart looms.** Optimizing the metric destroys the metric — hold\n", " out a fresh test set and read scores skeptically.\n", "\n", "## What’s Next\n", "\n", "You can now *measure* a model as well as build one — the last missing\n", "piece of the loop. From here the book turns to putting models to work:\n", "**tool use and agents** (models that act, not just answer) and\n", "**interpretability** (opening the box to see *why* a model scores the\n", "way it does). Evaluation is the thread through all of it: every new\n", "capability is only as real as the metric that confirms it.\n", "\n", "### Going Deeper\n", "\n", "**Core Papers:**\n", "\n", "- [Evaluating Large Language Models Trained on\n", " Code](https://arxiv.org/abs/2107.03374) — Chen et al. (2021),\n", " HumanEval and the **pass@k** unbiased estimator built here.\n", "- [Measuring Massive Multitask Language\n", " Understanding](https://arxiv.org/abs/2009.03300) — Hendrycks et\n", " al. (2020), MMLU — the multiple-choice, exact-match benchmark.\n", "- [Training Verifiers to Solve Math Word\n", " Problems](https://arxiv.org/abs/2110.14168) — Cobbe et al. (2021),\n", " GSM8K and exact-match answer grading (ties to m13).\n", "- [Judging LLM-as-a-Judge with MT-Bench and Chatbot\n", " Arena](https://arxiv.org/abs/2306.05685) — Zheng et al. (2023),\n", " LLM-as-judge and its position bias.\n", "\n", "**Practical Resources:**\n", "\n", "- [Holistic Evaluation of Language Models\n", " (HELM)](https://arxiv.org/abs/2211.09110) — Liang et al. (2022),\n", " evaluating across many metrics and scenarios at once.\n", "- [Goodhart’s Law](https://en.wikipedia.org/wiki/Goodhart%27s_law) — why\n", " a metric that becomes a target stops being a good metric." ], "id": "ea94de77-b21c-463a-a9fa-2c09af46cb05" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3 (ipykernel)", "language": "python", "path": "/opt/buildhome/.asdf/installs/python/3.11.14/share/jupyter/kernels/python3" }, "language_info": { "name": "python", "codemirror_mode": { "name": "ipython", "version": "3" }, "file_extension": ".py", "mimetype": "text/x-python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.14" } } }