{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Module 24: Pretraining Data" ], "id": "cfc11dfd-464d-47b3-a4cb-2b9c348b8feb" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [], "id": "73317c47-ec28-49a6-b0db-8414849cb1e2" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "e3bf688e-e37e-41f0-9584-99894e817554" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "Every module so far has been about the **model** — how it tokenizes\n", "([m03](../m03_tokenization/lesson.qmd)), attends\n", "([m05](../m05_attention/lesson.qmd)), trains\n", "([m07](../m07_training/lesson.qmd)), and scales\n", "([m23](../m23_distributed/lesson.qmd)). But the corpus has always\n", "arrived pre-cleaned, as if by magic. In practice **the data is the\n", "lever**: given a fixed architecture and compute budget, the single\n", "biggest thing you control is *what goes in*. **Pretraining data\n", "curation** is the pipeline that turns trillions of raw, filthy web\n", "tokens into a training set worth learning from.\n", "\n", "That pipeline is a funnel with three stages, and this module builds each\n", "from scratch:\n", "\n", "- **Quality filtering** — cheap, deterministic heuristics that throw out\n", " documents that are too short, too repetitive, symbol-spammy, or\n", " missing the function words real language always has.\n", "- **Deduplication** — the web is full of near-copies, and duplicated\n", " data makes models memorize instead of generalize. Exact dedup is a\n", " hash set; **near**-dedup needs **MinHash**, a sketch whose collision\n", " rate *equals* set overlap.\n", "- **Data mixing** — a corpus is many domains of unequal size. How much\n", " to sample from each is a temperature dial you will drive yourself.\n", "\n", "Why it matters for LLMs:\n", "\n", "- **Garbage in, garbage out — measurably.** Deduplicating training data\n", " alone lowers perplexity and cuts memorization (Lee et al. 2021);\n", " quality filtering is why Gopher, Llama, and every frontier model spend\n", " more engineering on data than on the model.\n", "- **It’s cheap and parallel.** Every function here is a pure map over\n", " documents — no gradients, no GPUs — so it runs at corpus scale on\n", " CPUs.\n", "- **The estimator is exact.** MinHash isn’t a heuristic: the fraction of\n", " matching signature positions is an *unbiased* estimate of the Jaccard\n", " similarity, with a variance you control by making the signature\n", " longer.\n", "\n", "### What You’ll Learn\n", "\n", "After this module, you can:\n", "\n", "- Implement the **Gopher/C4 quality heuristics** as pure functions and\n", " read a per-document pass/fail report.\n", "- Explain why **duplicated data hurts**, and build **exact** and\n", " **near**-dedup.\n", "- Derive the **MinHash** property $P[\\text{min-hash equal}] = J(A,B)$\n", " and use it to estimate document overlap; scale it with **LSH**.\n", "- Compute **temperature-weighted mixing** $p_i \\propto n_i^\\alpha$ and\n", " the **effective epochs** each domain is seen.\n", "\n", "### Prerequisites\n", "\n", "This module requires familiarity with:\n", "\n", "- [Module 03: Tokenization](../m03_tokenization/lesson.qmd) — what a\n", " training corpus *is*, and how text becomes tokens once it’s clean.\n", "- [Module 07: Training](../m07_training/lesson.qmd) — where this data\n", " goes, and why the token budget (m07’s scaling laws) makes *quality per\n", " token* matter.\n", "\n", "## Intuition: The Data Funnel\n", "\n", "Start with **Common Crawl**: petabytes of raw HTML, most of it junk —\n", "SEO spam, navigation bars, boilerplate, machine translation, and endless\n", "near-duplicates. You cannot train on it directly. Each stage of the\n", "funnel is a filter that a document must survive:\n", "\n", " raw web ─▶ language ID ─▶ quality filters ─▶ dedup ─▶ domain mixing ─▶ training set\n", " (trillions) (drop junk) (drop copies) (reweight) (what m07 sees)\n", "\n", "The stages are cheap and *composable*: each is a map from documents to a\n", "keep/drop decision (or a weight). The art is in the thresholds — filter\n", "too little and you train on spam; filter too much and you strip exactly\n", "the rare, high-quality text you wanted. By the end of this lesson you\n", "will drive that whole funnel over a toy corpus and watch it shrink and\n", "clean itself.\n", "\n", "> **Key Insight**\n", ">\n", "> Data curation is not preprocessing you do once and forget — it *is*\n", "> the model’s inductive bias about what language looks like. A filter\n", "> that removes all code removes the model’s ability to write code. Every\n", "> threshold is a curriculum choice.\n", "\n", "## The Math & Code: Quality Filtering\n", "\n", "The first funnel stage is a set of **heuristics** from the Gopher paper\n", "(Rae et al. 2021) and C4 (Raffel et al. 2020). None of them look at\n", "meaning; they exploit cheap statistical signatures of junk. The real\n", "thresholds Gopher uses:\n", "\n", "| Heuristic | Keep the document if… | Catches |\n", "|-------------------|--------------------------------------|----------------|\n", "| **word count** | between 50 and 100,000 words | stubs, dumps |\n", "| **mean word length** | between 3 and 10 characters | menus, base64/URL soup |\n", "| **symbol-to-word ratio** | $\\le 0.1$ (`#`, ellipses) | hashtag walls, snippets |\n", "| **stop words** | $\\ge 2$ of *{the, be, to, of, and, that, have, with}* | keyword spam, tables |\n", "| **duplicate lines** | $< 30\\%$ of lines are repeats | boilerplate footers |\n", "\n", "Each is a pure function in `pretraining_data.py`. They are exact and\n", "checkable:" ], "id": "04ef89e6-f8fb-447e-9239-f74f97ab8260" }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "text: # # deals # # cheap # # buy now # # click # # # subscribe # # #\n", "mean word length: 2.2\n", "symbol/word ratio: 0.7\n", "stop words: 0" ] } ], "source": [ "import sys\n", "sys.path.insert(0, \".\")\n", "from pretraining_data import (\n", " mean_word_length, symbol_to_word_ratio, stop_word_count,\n", " fraction_duplicate_lines, quality_report, CORPUS, DEMO_FILTERS,\n", ")\n", "\n", "spam = CORPUS[1][\"text\"] # a hashtag wall\n", "print(\"text:\", spam)\n", "print(\"mean word length:\", round(mean_word_length(spam), 2))\n", "print(\"symbol/word ratio:\", round(symbol_to_word_ratio(spam), 2))\n", "print(\"stop words:\", stop_word_count(spam))" ], "id": "e2a19052" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`quality_report` runs every heuristic and returns which ones a document\n", "passes — so you see *why* it was rejected, not just that it was:" ], "id": "796d91f0-2f18-4551-9787-71e5cd939c25" }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " word_count PASS\n", " mean_word_length FAIL\n", " symbol_ratio FAIL\n", " stop_words FAIL\n", " duplicate_lines PASS" ] } ], "source": [ "for name, ok in quality_report(spam, DEMO_FILTERS).items():\n", " print(f\" {name:16} {'PASS' if ok else 'FAIL'}\")" ], "id": "4428a531" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The hashtag wall fails three ways at once: its “words” are mostly single\n", "`#` characters (mean length too low), its symbol ratio is $0.7$ (seven\n", "times the limit), and it has no stop words. That triple failure is the\n", "fingerprint of non-language.\n", "\n", "> **These are thresholds, not truths**\n", ">\n", "> The 50-word minimum, the $0.1$ symbol ratio — none are laws. They were\n", "> tuned on English web text and *will* mis-fire: a valid haiku is too\n", "> short, a code file is symbol-heavy, a non-English document has\n", "> different stop words. Production pipelines pair these heuristics with\n", "> a learned quality classifier and per-language thresholds. Filter\n", "> conservatively; you can’t un-drop a document.\n", "\n", "### Walk a document through the filters\n", "\n", "Step through the five heuristics applied to the symbol-spam document.\n", "Each row shows the measured value against the rule, and whether it\n", "passes (green) or the document dies there (red)." ], "id": "27b285c7-939d-422d-af04-e8de3252554d" }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [], "id": "3ce2af79" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "c2cbd61b-eba3-420b-b2d3-4d05a775c3bb" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "b9beb9eb-9800-4343-a4f3-01bc8d7efbcc" }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Try This**\n", ">\n", "> 1. **Step to “symbol / word ratio”.** The measured $0.7$ dwarfs the\n", "> $0.1$ rule — one heuristic alone would have caught this.\n", "> 2. **Note “word count” passes.** A junk document can clear *some*\n", "> filters; you need the whole battery. This is why quality filtering\n", "> is a conjunction, not a vote.\n", "\n", "## The Math & Code: Deduplication\n", "\n", "Near-duplicate documents are everywhere on the web — reposts, templated\n", "pages, scraped mirrors. Training on them is actively harmful: the model\n", "spends capacity **memorizing** repeated spans instead of learning\n", "general structure, and duplicated text inflates benchmark contamination.\n", "Lee et al. (2021) showed that deduplicating the training set lowers\n", "perplexity and cuts verbatim memorization.\n", "\n", "**Exact** dedup is easy — normalize whitespace and case, hash, keep the\n", "first of each:" ], "id": "7f7dea3c-187f-44ab-b675-ac2de22c464a" }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "normalized: ['the transformer', 'the transformer', 'a different doc']\n", "kept indices: [0, 2]" ] } ], "source": [ "from pretraining_data import exact_dedup, normalize_text\n", "\n", "docs = [\"The Transformer\", \"the transformer\", \"a different doc\"]\n", "print(\"normalized:\", [normalize_text(d) for d in docs])\n", "print(\"kept indices:\", exact_dedup(docs)) # the second is a normalized copy" ], "id": "d39724bf" }, { "cell_type": "markdown", "metadata": {}, "source": [ "But exact dedup misses the *near*-copy — the same article with one word\n", "changed hashes to something completely different. For that we need a\n", "similarity that is cheap to estimate over millions of documents.\n", "\n", "### Shingles and Jaccard\n", "\n", "Represent a document as its set of **shingles** — overlapping word\n", "$k$-grams. Two documents are near-duplicates when their shingle *sets*\n", "overlap heavily, measured by the **Jaccard similarity**:\n", "\n", "$$J(A, B) = \\frac{|A \\cap B|}{|A \\cup B|}$$" ], "id": "d3f90869-2c00-4be1-aa1d-a3effd8281fd" }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "shingles A: ['cat sat on', 'on the mat', 'sat on the', 'the cat sat']\n", "Jaccard(A, B): 0.6" ] } ], "source": [ "from pretraining_data import shingles, jaccard\n", "\n", "a = shingles(\"the cat sat on the mat\", k=3)\n", "b = shingles(\"the cat sat on the rug\", k=3)\n", "print(\"shingles A:\", sorted(a))\n", "print(\"Jaccard(A, B):\", jaccard(a, b))" ], "id": "ddc02bb6" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Exact Jaccard is correct but expensive: comparing every pair of $N$\n", "documents is $O(N^2)$ set operations, hopeless at web scale. **MinHash**\n", "makes each comparison $O(1)$ over a tiny fixed-size sketch.\n", "\n", "### MinHash: the estimator that *is* the similarity\n", "\n", "Pick a random hash function $h$ and, for a set $A$, keep only the\n", "**minimum** hash value over its shingles, $\\min_{x \\in A} h(x)$. The\n", "magic property (Broder, 1997):\n", "\n", "$$P\\big[\\, \\min_{x \\in A} h(x) = \\min_{x \\in B} h(x) \\,\\big] \\;=\\; J(A, B)$$\n", "\n", "The two minima coincide exactly when the overall-minimum element lies in\n", "the intersection — which happens with probability\n", "$|A\\cap B| / |A \\cup B|$. So repeat with $m$ independent hash functions\n", "to get an $m$-value **signature**, and the **fraction of matching\n", "positions is an unbiased estimate of the Jaccard similarity** — with\n", "variance shrinking as $1/m$. A short signature (say 128 ints) replaces\n", "the whole document." ], "id": "b7c86baf-8173-44ff-87fa-20dca48e1d69" }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "signature length: 128\n", "estimated Jaccard: 0.6328125\n", "true Jaccard: 0.6" ] } ], "source": [ "from pretraining_data import minhash_signature, estimated_jaccard\n", "\n", "sig_a = minhash_signature(a, num_perm=128, seed=0)\n", "sig_b = minhash_signature(b, num_perm=128, seed=0)\n", "print(\"signature length:\", len(sig_a))\n", "print(\"estimated Jaccard:\", estimated_jaccard(sig_a, sig_b))\n", "print(\"true Jaccard: \", jaccard(a, b))" ], "id": "0a4c4b82" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Watch the estimate converge\n", "\n", "Two documents with a genuine partial overlap (the clean web doc and its\n", "near-duplicate from the toy corpus, true $J \\approx 0.51$). Slide the\n", "signature length: with few hash functions the estimate is noisy; as $m$\n", "grows it locks onto the true value — the $1/m$ variance made visible." ], "id": "6d98d768-135f-4d7b-b4d4-d03eead3eb78" }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [], "id": "b2d1c48f" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "98c4351f-5225-40d9-8bf5-f99625e79f05" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "b0715333-63cd-49f5-a860-efc9f09de4ec" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Scaling with LSH\n", "\n", "MinHash makes each *comparison* cheap, but you still can’t compare all\n", "$N^2$ pairs. **Locality-Sensitive Hashing** finds the candidate\n", "near-duplicates without looking at most pairs: split the length-$m$\n", "signature into $b$ **bands** of $r$ rows ($m = b \\cdot r$), and hash\n", "each band. Two documents become a **candidate** if they collide in *any*\n", "band. A pair with similarity $s$ survives with probability\n", "\n", "$$P(\\text{candidate}) = 1 - (1 - s^{\\,r})^{b}$$\n", "\n", "an **S-curve**: near-0 below a tunable threshold, near-1 above it.\n", "`near_dedup` ties it together — LSH proposes candidates, MinHash scores\n", "them, and the later document of any near-duplicate pair is dropped:" ], "id": "532ba3bb-1489-4247-9178-2e6d5938a532" }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "kept indices: [0, 1, 2, 5, 6]\n", "dropped: [3, 4]" ] } ], "source": [ "from pretraining_data import near_dedup\n", "\n", "corpus_texts = [d[\"text\"] for d in CORPUS]\n", "kept = near_dedup(corpus_texts, threshold=0.4, num_perm=128, bands=32, rows=4)\n", "print(\"kept indices:\", kept)\n", "print(\"dropped:\", [i for i in range(len(corpus_texts)) if i not in kept])" ], "id": "1d8ee67f" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The exact duplicate (index 3) *and* the near-duplicate (index 4) are\n", "both gone — the near-copy that exact-dedup could never catch.\n", "\n", "> **Dedup against the eval set, or you’ll fool yourself**\n", ">\n", "> The most dangerous duplicates are the ones shared between your\n", "> **training** data and your **evaluation** benchmarks. If a test\n", "> question leaks into pretraining, the model “solves” it by memorization\n", "> and your scores are fiction ([m17](../m17_evaluation/lesson.qmd) calls\n", "> this contamination). Always dedup the training corpus *against* the\n", "> eval sets, not just within itself.\n", "\n", "## The Math & Code: Data Mixing\n", "\n", "After filtering and dedup you have a clean corpus — but it is\n", "**imbalanced**. Web text dwarfs curated sources (books, code,\n", "Wikipedia). Train in natural proportion and the model barely sees the\n", "high-quality tail; train uniformly and you repeat tiny domains until it\n", "memorizes them. The standard lever is **temperature-weighted sampling**:\n", "give domain $i$ (with $n_i$ tokens) a sampling weight\n", "\n", "$$p_i \\;=\\; \\frac{n_i^{\\,\\alpha}}{\\sum_j n_j^{\\,\\alpha}}$$\n", "\n", "$\\alpha = 1$ samples in natural proportion; $\\alpha = 0$ is uniform over\n", "domains (maximal upsampling of small ones); $0 < \\alpha < 1$\n", "interpolates." ], "id": "539c9d47-b5e4-442e-8e97-0fc1414bb33a" }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "alpha=1.0 (natural): [0.8, 0.15, 0.04, 0.01]\n", "alpha=0.5 (tempered): [0.565, 0.245, 0.126, 0.063]\n", "alpha=0.0 (uniform): [0.25, 0.25, 0.25, 0.25]" ] } ], "source": [ "from pretraining_data import mixing_weights, effective_epochs\n", "\n", "sizes = [800, 150, 40, 10] # web, books, code, wiki (billions of tokens)\n", "print(\"alpha=1.0 (natural): \", [round(w, 3) for w in mixing_weights(sizes, 1.0)])\n", "print(\"alpha=0.5 (tempered):\", [round(w, 3) for w in mixing_weights(sizes, 0.5)])\n", "print(\"alpha=0.0 (uniform): \", [round(w, 3) for w in mixing_weights(sizes, 0.0)])" ], "id": "d1d2d185" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The catch: upsampling a small domain means **repeating** it.\n", "`effective_epochs` tells you how many times each domain is seen for a\n", "given token budget — and repetition past a few epochs is where\n", "memorization creeps back in." ], "id": "4303d6f6-8ed8-4dc8-a504-01ff463f1ffe" }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " web seen 0.71×\n", " books seen 1.63×\n", " code seen 3.16×\n", " wiki seen 6.32×" ] } ], "source": [ "eps = effective_epochs(sizes, target_tokens=1000, alpha=0.5)\n", "for n, e in zip([\"web\", \"books\", \"code\", \"wiki\"], eps):\n", " print(f\" {n:6} seen {e:.2f}×\")" ], "id": "13e8d2bf" }, { "cell_type": "markdown", "metadata": {}, "source": [ "At $\\alpha = 0.5$ the 10B-token wiki domain is repeated $6.3\\times$ to\n", "lift its share — a real risk you are trading against diversity.\n", "\n", "### Drive the mixing temperature\n", "\n", "Slide $\\alpha$ from proportional (right) to uniform (left) and watch the\n", "domain shares reshape. The number under each bar is its **effective\n", "epochs** — when it climbs past a few, you’re re-showing the same tokens." ], "id": "8588724a-c50c-4767-9db0-f176e05f693f" }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [], "id": "d39f12e5" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "5c0071d4-eaf2-4d64-ba5c-9ab527758912" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "4aeab9a1-d33b-4b8e-8a95-2ac53aea2b95" }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Try This**\n", ">\n", "> 1. **Drag α to 0.** Every domain gets an equal 25% — but wiki is now\n", "> repeated many times over (its epoch count turns red). Uniform\n", "> mixing trades diversity for memorization risk.\n", "> 2. **Drag α to 1.** Web swamps everything at 80%; the curated domains\n", "> barely register. The truth is in between — most recipes use\n", "> $\\alpha \\approx 0.3$–$0.7$.\n", "\n", "## Interactive Exploration\n", "\n", "Now run the **whole funnel** over the toy corpus. Toggle each stage on\n", "or off and watch the corpus shrink and clean. Each document is a tile,\n", "colored by the stage that drops it (or green if it survives to\n", "training). This is `demonstrate_pipeline` in `pretraining_data.py`,\n", "bridged live." ], "id": "6249ac50-a8b6-4d51-82f3-5ec431d4827b" }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [], "id": "9f511a0b" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "2967ca0f-9fe3-4d28-915b-be03a892564c" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": {} } ], "source": [], "id": "3ba7f43a-53cd-4321-b6fe-5a4872f4f4d1" }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Try This**\n", ">\n", "> 1. **Turn off every stage.** All 7 documents pass through — including\n", "> the hashtag wall and both copies. This is training on raw web.\n", "> 2. **Enable only “quality”.** The spam, the stub, and the boilerplate\n", "> vanish, but both copies of the clean doc survive — filtering can’t\n", "> see duplication.\n", "> 3. **Enable “exact-dedup” then “near-dedup”.** The identical copy\n", "> dies first; only near-dedup catches the reworded near-copy. Two\n", "> clean documents remain.\n", "\n", "## Common Pitfalls\n", "\n", "> **Over-filtering strips the tail you wanted**\n", ">\n", "> Aggressive heuristics disproportionately remove exactly the rare,\n", "> high-value text (technical prose, code, non-English) that makes a\n", "> model capable. The symbol filter that kills spam also kills LaTeX and\n", "> source code. Measure what a filter removes before trusting it — sample\n", "> the rejects and read them.\n", "\n", "> **MinHash estimates, it does not certify**\n", ">\n", "> The signature gives an *estimate* of Jaccard with $1/m$ variance. At a\n", "> threshold, some true duplicates slip through and some distinct docs\n", "> get dropped. Pick $m$ and the LSH band/row split for the\n", "> precision/recall you can tolerate; there is no setting that is exact\n", "> and cheap at once.\n", "\n", "> **Upsampling is repetition in disguise**\n", ">\n", "> A high mixing weight on a small domain means many epochs over the same\n", "> tokens. Past a handful of repeats the model memorizes rather than\n", "> generalizes — the same failure duplication caused. `effective_epochs`\n", "> is the number to watch, not the sampling weight.\n", "\n", "> **Curation choices are silent and compounding**\n", ">\n", "> Every threshold is a curriculum decision the model can never override.\n", "> Drop all code and it can’t program; over-weight forums and it learns\n", "> their tone. Unlike a loss curve, a bad data mix leaves no error\n", "> message — only a weaker model.\n", "\n", "## Exercises\n", "\n", "### Exercise 1: A custom quality filter\n", "\n", "Add a heuristic that rejects documents whose fraction of numeric tokens\n", "exceeds a threshold (catches tables and data dumps). Wire it into a\n", "report." ], "id": "5d5643f5-dfd7-4fb8-8ed0-d0a6d99d1100" }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "clean doc: 0.0\n", "digits: 1.0" ] } ], "source": [ "import re\n", "from pretraining_data import CORPUS\n", "\n", "def numeric_fraction(text: str) -> float:\n", " # Your implementation here: fraction of whitespace tokens that are numbers.\n", " words = text.split()\n", " if not words:\n", " return 0.0\n", " nums = sum(1 for w in words if re.fullmatch(r\"\\d[\\d,.]*\", w))\n", " return nums / len(words)\n", "\n", "print(\"clean doc:\", round(numeric_fraction(CORPUS[0][\"text\"]), 3))\n", "print(\"digits: \", round(numeric_fraction(\"2024 2025 42 3.14 100 7 99 5\"), 3))" ], "id": "50d1a535" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2: Tune the LSH band/row split\n", "\n", "For a fixed signature length $m = b \\cdot r$, the split $(b, r)$ sets\n", "the S-curve’s threshold. Plot the collision probability at a target\n", "similarity for a few splits and pick the one whose curve rises near your\n", "dedup threshold." ], "id": "7436e656-48ab-4a62-8d70-6983261b23d0" }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "b=64 r=2: P(0.8)=1.000 P(0.4)=1.000\n", "b=32 r=4: P(0.8)=1.000 P(0.4)=0.564\n", "b=16 r=8: P(0.8)=0.947 P(0.4)=0.010" ] } ], "source": [ "from pretraining_data import lsh_collision_probability\n", "\n", "m = 128\n", "for bands, rows in [(64, 2), (32, 4), (16, 8)]:\n", " assert bands * rows == m\n", " at_08 = lsh_collision_probability(0.8, bands, rows)\n", " at_04 = lsh_collision_probability(0.4, bands, rows)\n", " print(f\"b={bands:2} r={rows}: P(0.8)={at_08:.3f} P(0.4)={at_04:.3f}\")\n", "# More rows per band → sharper, higher threshold. Which split best separates\n", "# 0.8 (duplicate) from 0.4 (distinct)?" ], "id": "7c1b9a89" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3: Back out α from a target epoch count\n", "\n", "Given domain sizes and a token budget, find the temperature $\\alpha$\n", "that shows your smallest domain a target number of epochs — a common way\n", "to *set* the mix." ], "id": "f273f015-3002-418f-ac00-e6869c27bb97" }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "alpha ≈ 0.82 → wiki seen 2.01×" ] } ], "source": [ "from pretraining_data import effective_epochs\n", "\n", "sizes = [800, 150, 40, 10]\n", "target_tokens = 1000\n", "\n", "def epochs_of_smallest(alpha):\n", " return effective_epochs(sizes, target_tokens, alpha)[-1]\n", "\n", "# Your implementation here: scan alpha for the value giving ~2 epochs on wiki.\n", "best = min([a / 100 for a in range(0, 101)],\n", " key=lambda a: abs(epochs_of_smallest(a) - 2.0))\n", "print(f\"alpha ≈ {best:.2f} → wiki seen {epochs_of_smallest(best):.2f}×\")" ], "id": "2171d9dd" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "Key takeaways:\n", "\n", "1. **Data curation is the lever.** Given fixed compute and\n", " architecture, filtering, deduplication, and mixing determine model\n", " quality more than any single modeling trick — and they run cheaply\n", " on CPUs as pure maps over documents.\n", "2. **Quality filters are cheap heuristics, applied as a conjunction.**\n", " Word count, mean word length, symbol ratio, stop words, and line\n", " repetition each catch a fingerprint of junk; a document must clear\n", " all of them. The thresholds are tuned guesses, not truths.\n", "3. **MinHash turns similarity into a sketch.** Because\n", " $P[\\text{min-hash equal}] =\n", " J(A,B)$, the fraction of matching signature positions is an\n", " *unbiased* Jaccard estimate with $1/m$ variance, and **LSH**\n", " ($1-(1-s^r)^b$) finds candidates without the $O(N^2)$ scan.\n", " Near-dedup catches the copies exact hashing misses.\n", "4. **Mixing is a temperature dial with a repetition cost.**\n", " $p_i \\propto n_i^\\alpha$ interpolates from natural ($\\alpha{=}1$) to\n", " uniform ($\\alpha{=}0$); upsampling a small domain repeats it, so\n", " watch **effective epochs**, not just the weight.\n", "5. **Every threshold is a silent curriculum choice.** Curation shapes\n", " what the model can ever learn, leaves no error message when wrong,\n", " and must be measured — including *against* the eval sets to avoid\n", " contamination.\n", "\n", "## What’s Next\n", "\n", "You can now build the model (m01–m06), feed it a **curated corpus**\n", "(this module), train it (m07), scale that training (m23), align it\n", "(m12), and serve it (m08–m09). The data frontier from here: a learned\n", "**quality classifier** (a small model scoring documents), **synthetic\n", "data** and self-distillation, **data curriculum** (easy-to-hard\n", "ordering), and **decontamination** at scale against every benchmark.\n", "\n", "### Going Deeper\n", "\n", "**Core Papers:**\n", "\n", "- [Scaling Language Models: Methods, Analysis & Insights from Training\n", " Gopher](https://arxiv.org/abs/2112.11446) — Rae et al. (2021): the\n", " MassiveText quality-filter heuristics (word count, mean word length,\n", " symbol ratio, stop words, repetition) built here.\n", "- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n", " Transformer (C4)](https://arxiv.org/abs/1910.10683) — Raffel et\n", " al. (2020): the C4 cleaning pipeline for Common Crawl.\n", "- [Deduplicating Training Data Makes Language Models\n", " Better](https://arxiv.org/abs/2107.06499) — Lee et al. (2021): why\n", " exact + near dedup lowers perplexity and cuts memorization.\n", "- [On the Resemblance and Containment of\n", " Documents](https://ieeexplore.ieee.org/document/666900) — Broder\n", " (1997): MinHash and the $P[\\text{min equal}] = J$ property.\n", "- [The Pile: An 800GB Dataset of Diverse Text for Language\n", " Modeling](https://arxiv.org/abs/2101.00027) — Gao et al. (2020):\n", " domain mixing and diversity in a curated corpus.\n", "\n", "**Practical Resources:**\n", "\n", "- [Mining of Massive Datasets, Ch. 3 (Finding Similar\n", " Items)](http://www.mmds.org/) — Leskovec, Rajaraman & Ullman:\n", " shingling, MinHash, and LSH in depth.\n", "- [The RefinedWeb Dataset for Falcon\n", " LLM](https://arxiv.org/abs/2306.01116) — Penedo et al. (2023): that\n", " well-filtered web data alone can match curated corpora." ], "id": "9d54eb24-c5b6-4831-9673-5b1575f6b455" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3 (ipykernel)", "language": "python", "path": "/opt/buildhome/.asdf/installs/python/3.11.14/share/jupyter/kernels/python3" }, "language_info": { "name": "python", "codemirror_mode": { "name": "ipython", "version": "3" }, "file_extension": ".py", "mimetype": "text/x-python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.14" } } }