Module 24: Pretraining Data

Introduction

Every module so far has been about the model — how it tokenizes (m03), attends (m05), trains (m07), and scales (m23). But the corpus has always arrived pre-cleaned, as if by magic. In practice the data is the lever: given a fixed architecture and compute budget, the single biggest thing you control is what goes in. Pretraining data curation is the pipeline that turns trillions of raw, filthy web tokens into a training set worth learning from.

That pipeline is a funnel with three stages, and this module builds each from scratch:

  • Quality filtering — cheap, deterministic heuristics that throw out documents that are too short, too repetitive, symbol-spammy, or missing the function words real language always has.
  • Deduplication — the web is full of near-copies, and duplicated data makes models memorize instead of generalize. Exact dedup is a hash set; near-dedup needs MinHash, a sketch whose collision rate equals set overlap.
  • Data mixing — a corpus is many domains of unequal size. How much to sample from each is a temperature dial you will drive yourself.

Why it matters for LLMs:

  • Garbage in, garbage out — measurably. Deduplicating training data alone lowers perplexity and cuts memorization (Lee et al. 2021); quality filtering is why Gopher, Llama, and every frontier model spend more engineering on data than on the model.
  • It’s cheap and parallel. Every function here is a pure map over documents — no gradients, no GPUs — so it runs at corpus scale on CPUs.
  • The estimator is exact. MinHash isn’t a heuristic: the fraction of matching signature positions is an unbiased estimate of the Jaccard similarity, with a variance you control by making the signature longer.

What You’ll Learn

After this module, you can:

  • Implement the Gopher/C4 quality heuristics as pure functions and read a per-document pass/fail report.
  • Explain why duplicated data hurts, and build exact and near-dedup.
  • Derive the MinHash property \(P[\text{min-hash equal}] = J(A,B)\) and use it to estimate document overlap; scale it with LSH.
  • Compute temperature-weighted mixing \(p_i \propto n_i^\alpha\) and the effective epochs each domain is seen.

Prerequisites

This module requires familiarity with:

  • Module 03: Tokenization — what a training corpus is, and how text becomes tokens once it’s clean.
  • Module 07: Training — where this data goes, and why the token budget (m07’s scaling laws) makes quality per token matter.

Intuition: The Data Funnel

Start with Common Crawl: petabytes of raw HTML, most of it junk — SEO spam, navigation bars, boilerplate, machine translation, and endless near-duplicates. You cannot train on it directly. Each stage of the funnel is a filter that a document must survive:

raw web  ─▶  language ID  ─▶  quality filters  ─▶  dedup  ─▶  domain mixing  ─▶  training set
 (trillions)                   (drop junk)      (drop copies)   (reweight)      (what m07 sees)

The stages are cheap and composable: each is a map from documents to a keep/drop decision (or a weight). The art is in the thresholds — filter too little and you train on spam; filter too much and you strip exactly the rare, high-quality text you wanted. By the end of this lesson you will drive that whole funnel over a toy corpus and watch it shrink and clean itself.

NoteKey Insight

Data curation is not preprocessing you do once and forget — it is the model’s inductive bias about what language looks like. A filter that removes all code removes the model’s ability to write code. Every threshold is a curriculum choice.

The Math & Code: Quality Filtering

The first funnel stage is a set of heuristics from the Gopher paper (Rae et al. 2021) and C4 (Raffel et al. 2020). None of them look at meaning; they exploit cheap statistical signatures of junk. The real thresholds Gopher uses:

Heuristic Keep the document if… Catches
word count between 50 and 100,000 words stubs, dumps
mean word length between 3 and 10 characters menus, base64/URL soup
symbol-to-word ratio \(\le 0.1\) (#, ellipses) hashtag walls, snippets
stop words \(\ge 2\) of {the, be, to, of, and, that, have, with} keyword spam, tables
duplicate lines \(< 30\%\) of lines are repeats boilerplate footers

Each is a pure function in pretraining_data.py. They are exact and checkable:

import sys
sys.path.insert(0, ".")
from pretraining_data import (
    mean_word_length, symbol_to_word_ratio, stop_word_count,
    fraction_duplicate_lines, quality_report, CORPUS, DEMO_FILTERS,
)

spam = CORPUS[1]["text"]     # a hashtag wall
print("text:", spam)
print("mean word length:", round(mean_word_length(spam), 2))
print("symbol/word ratio:", round(symbol_to_word_ratio(spam), 2))
print("stop words:", stop_word_count(spam))
text: # # deals # # cheap # # buy now # # click # # # subscribe # # #
mean word length: 2.2
symbol/word ratio: 0.7
stop words: 0

quality_report runs every heuristic and returns which ones a document passes — so you see why it was rejected, not just that it was:

for name, ok in quality_report(spam, DEMO_FILTERS).items():
    print(f"  {name:16} {'PASS' if ok else 'FAIL'}")
  word_count       PASS
  mean_word_length FAIL
  symbol_ratio     FAIL
  stop_words       FAIL
  duplicate_lines  PASS

The hashtag wall fails three ways at once: its “words” are mostly single # characters (mean length too low), its symbol ratio is \(0.7\) (seven times the limit), and it has no stop words. That triple failure is the fingerprint of non-language.

WarningThese are thresholds, not truths

The 50-word minimum, the \(0.1\) symbol ratio — none are laws. They were tuned on English web text and will mis-fire: a valid haiku is too short, a code file is symbol-heavy, a non-English document has different stop words. Production pipelines pair these heuristics with a learned quality classifier and per-language thresholds. Filter conservatively; you can’t un-drop a document.

Walk a document through the filters

Step through the five heuristics applied to the symbol-spam document. Each row shows the measured value against the rule, and whether it passes (green) or the document dies there (red).

TipTry This
  1. Step to “symbol / word ratio”. The measured \(0.7\) dwarfs the \(0.1\) rule — one heuristic alone would have caught this.
  2. Note “word count” passes. A junk document can clear some filters; you need the whole battery. This is why quality filtering is a conjunction, not a vote.

The Math & Code: Deduplication

Near-duplicate documents are everywhere on the web — reposts, templated pages, scraped mirrors. Training on them is actively harmful: the model spends capacity memorizing repeated spans instead of learning general structure, and duplicated text inflates benchmark contamination. Lee et al. (2021) showed that deduplicating the training set lowers perplexity and cuts verbatim memorization.

Exact dedup is easy — normalize whitespace and case, hash, keep the first of each:

from pretraining_data import exact_dedup, normalize_text

docs = ["The Transformer", "the   transformer", "a different doc"]
print("normalized:", [normalize_text(d) for d in docs])
print("kept indices:", exact_dedup(docs))   # the second is a normalized copy
normalized: ['the transformer', 'the transformer', 'a different doc']
kept indices: [0, 2]

But exact dedup misses the near-copy — the same article with one word changed hashes to something completely different. For that we need a similarity that is cheap to estimate over millions of documents.

Shingles and Jaccard

Represent a document as its set of shingles — overlapping word \(k\)-grams. Two documents are near-duplicates when their shingle sets overlap heavily, measured by the Jaccard similarity:

\[ J(A, B) = \frac{|A \cap B|}{|A \cup B|} \]

from pretraining_data import shingles, jaccard

a = shingles("the cat sat on the mat", k=3)
b = shingles("the cat sat on the rug", k=3)
print("shingles A:", sorted(a))
print("Jaccard(A, B):", jaccard(a, b))
shingles A: ['cat sat on', 'on the mat', 'sat on the', 'the cat sat']
Jaccard(A, B): 0.6

Exact Jaccard is correct but expensive: comparing every pair of \(N\) documents is \(O(N^2)\) set operations, hopeless at web scale. MinHash makes each comparison \(O(1)\) over a tiny fixed-size sketch.

MinHash: the estimator that is the similarity

Pick a random hash function \(h\) and, for a set \(A\), keep only the minimum hash value over its shingles, \(\min_{x \in A} h(x)\). The magic property (Broder, 1997):

\[ P\big[\, \min_{x \in A} h(x) = \min_{x \in B} h(x) \,\big] \;=\; J(A, B) \]

The two minima coincide exactly when the overall-minimum element lies in the intersection — which happens with probability \(|A\cap B| / |A \cup B|\). So repeat with \(m\) independent hash functions to get an \(m\)-value signature, and the fraction of matching positions is an unbiased estimate of the Jaccard similarity — with variance shrinking as \(1/m\). A short signature (say 128 ints) replaces the whole document.

from pretraining_data import minhash_signature, estimated_jaccard

sig_a = minhash_signature(a, num_perm=128, seed=0)
sig_b = minhash_signature(b, num_perm=128, seed=0)
print("signature length:", len(sig_a))
print("estimated Jaccard:", estimated_jaccard(sig_a, sig_b))
print("true Jaccard:     ", jaccard(a, b))
signature length: 128
estimated Jaccard: 0.6328125
true Jaccard:      0.6

Watch the estimate converge

Two documents with a genuine partial overlap (the clean web doc and its near-duplicate from the toy corpus, true \(J \approx 0.51\)). Slide the signature length: with few hash functions the estimate is noisy; as \(m\) grows it locks onto the true value — the \(1/m\) variance made visible.

Scaling with LSH

MinHash makes each comparison cheap, but you still can’t compare all \(N^2\) pairs. Locality-Sensitive Hashing finds the candidate near-duplicates without looking at most pairs: split the length-\(m\) signature into \(b\) bands of \(r\) rows (\(m = b \cdot r\)), and hash each band. Two documents become a candidate if they collide in any band. A pair with similarity \(s\) survives with probability

\[ P(\text{candidate}) = 1 - (1 - s^{\,r})^{b} \]

an S-curve: near-0 below a tunable threshold, near-1 above it. near_dedup ties it together — LSH proposes candidates, MinHash scores them, and the later document of any near-duplicate pair is dropped:

from pretraining_data import near_dedup

corpus_texts = [d["text"] for d in CORPUS]
kept = near_dedup(corpus_texts, threshold=0.4, num_perm=128, bands=32, rows=4)
print("kept indices:", kept)
print("dropped:", [i for i in range(len(corpus_texts)) if i not in kept])
kept indices: [0, 1, 2, 5, 6]
dropped: [3, 4]

The exact duplicate (index 3) and the near-duplicate (index 4) are both gone — the near-copy that exact-dedup could never catch.

WarningDedup against the eval set, or you’ll fool yourself

The most dangerous duplicates are the ones shared between your training data and your evaluation benchmarks. If a test question leaks into pretraining, the model “solves” it by memorization and your scores are fiction (m17 calls this contamination). Always dedup the training corpus against the eval sets, not just within itself.

The Math & Code: Data Mixing

After filtering and dedup you have a clean corpus — but it is imbalanced. Web text dwarfs curated sources (books, code, Wikipedia). Train in natural proportion and the model barely sees the high-quality tail; train uniformly and you repeat tiny domains until it memorizes them. The standard lever is temperature-weighted sampling: give domain \(i\) (with \(n_i\) tokens) a sampling weight

\[ p_i \;=\; \frac{n_i^{\,\alpha}}{\sum_j n_j^{\,\alpha}} \]

\(\alpha = 1\) samples in natural proportion; \(\alpha = 0\) is uniform over domains (maximal upsampling of small ones); \(0 < \alpha < 1\) interpolates.

from pretraining_data import mixing_weights, effective_epochs

sizes = [800, 150, 40, 10]        # web, books, code, wiki (billions of tokens)
print("alpha=1.0 (natural): ", [round(w, 3) for w in mixing_weights(sizes, 1.0)])
print("alpha=0.5 (tempered):", [round(w, 3) for w in mixing_weights(sizes, 0.5)])
print("alpha=0.0 (uniform): ", [round(w, 3) for w in mixing_weights(sizes, 0.0)])
alpha=1.0 (natural):  [0.8, 0.15, 0.04, 0.01]
alpha=0.5 (tempered): [0.565, 0.245, 0.126, 0.063]
alpha=0.0 (uniform):  [0.25, 0.25, 0.25, 0.25]

The catch: upsampling a small domain means repeating it. effective_epochs tells you how many times each domain is seen for a given token budget — and repetition past a few epochs is where memorization creeps back in.

eps = effective_epochs(sizes, target_tokens=1000, alpha=0.5)
for n, e in zip(["web", "books", "code", "wiki"], eps):
    print(f"  {n:6} seen {e:.2f}×")
  web    seen 0.71×
  books  seen 1.63×
  code   seen 3.16×
  wiki   seen 6.32×

At \(\alpha = 0.5\) the 10B-token wiki domain is repeated \(6.3\times\) to lift its share — a real risk you are trading against diversity.

Drive the mixing temperature

Slide \(\alpha\) from proportional (right) to uniform (left) and watch the domain shares reshape. The number under each bar is its effective epochs — when it climbs past a few, you’re re-showing the same tokens.

TipTry This
  1. Drag α to 0. Every domain gets an equal 25% — but wiki is now repeated many times over (its epoch count turns red). Uniform mixing trades diversity for memorization risk.
  2. Drag α to 1. Web swamps everything at 80%; the curated domains barely register. The truth is in between — most recipes use \(\alpha \approx 0.3\)–\(0.7\).

Interactive Exploration

Now run the whole funnel over the toy corpus. Toggle each stage on or off and watch the corpus shrink and clean. Each document is a tile, colored by the stage that drops it (or green if it survives to training). This is demonstrate_pipeline in pretraining_data.py, bridged live.

TipTry This
  1. Turn off every stage. All 7 documents pass through — including the hashtag wall and both copies. This is training on raw web.
  2. Enable only “quality”. The spam, the stub, and the boilerplate vanish, but both copies of the clean doc survive — filtering can’t see duplication.
  3. Enable “exact-dedup” then “near-dedup”. The identical copy dies first; only near-dedup catches the reworded near-copy. Two clean documents remain.

Common Pitfalls

WarningOver-filtering strips the tail you wanted

Aggressive heuristics disproportionately remove exactly the rare, high-value text (technical prose, code, non-English) that makes a model capable. The symbol filter that kills spam also kills LaTeX and source code. Measure what a filter removes before trusting it — sample the rejects and read them.

WarningMinHash estimates, it does not certify

The signature gives an estimate of Jaccard with \(1/m\) variance. At a threshold, some true duplicates slip through and some distinct docs get dropped. Pick \(m\) and the LSH band/row split for the precision/recall you can tolerate; there is no setting that is exact and cheap at once.

WarningUpsampling is repetition in disguise

A high mixing weight on a small domain means many epochs over the same tokens. Past a handful of repeats the model memorizes rather than generalizes — the same failure duplication caused. effective_epochs is the number to watch, not the sampling weight.

WarningCuration choices are silent and compounding

Every threshold is a curriculum decision the model can never override. Drop all code and it can’t program; over-weight forums and it learns their tone. Unlike a loss curve, a bad data mix leaves no error message — only a weaker model.

Exercises

Exercise 1: A custom quality filter

Add a heuristic that rejects documents whose fraction of numeric tokens exceeds a threshold (catches tables and data dumps). Wire it into a report.

import re
from pretraining_data import CORPUS

def numeric_fraction(text: str) -> float:
    # Your implementation here: fraction of whitespace tokens that are numbers.
    words = text.split()
    if not words:
        return 0.0
    nums = sum(1 for w in words if re.fullmatch(r"\d[\d,.]*", w))
    return nums / len(words)

print("clean doc:", round(numeric_fraction(CORPUS[0]["text"]), 3))
print("digits:   ", round(numeric_fraction("2024 2025 42 3.14 100 7 99 5"), 3))
clean doc: 0.0
digits:    1.0

Exercise 2: Tune the LSH band/row split

For a fixed signature length \(m = b \cdot r\), the split \((b, r)\) sets the S-curve’s threshold. Plot the collision probability at a target similarity for a few splits and pick the one whose curve rises near your dedup threshold.

from pretraining_data import lsh_collision_probability

m = 128
for bands, rows in [(64, 2), (32, 4), (16, 8)]:
    assert bands * rows == m
    at_08 = lsh_collision_probability(0.8, bands, rows)
    at_04 = lsh_collision_probability(0.4, bands, rows)
    print(f"b={bands:2} r={rows}: P(0.8)={at_08:.3f}  P(0.4)={at_04:.3f}")
# More rows per band → sharper, higher threshold. Which split best separates
# 0.8 (duplicate) from 0.4 (distinct)?
b=64 r=2: P(0.8)=1.000  P(0.4)=1.000
b=32 r=4: P(0.8)=1.000  P(0.4)=0.564
b=16 r=8: P(0.8)=0.947  P(0.4)=0.010

Exercise 3: Back out α from a target epoch count

Given domain sizes and a token budget, find the temperature \(\alpha\) that shows your smallest domain a target number of epochs — a common way to set the mix.

from pretraining_data import effective_epochs

sizes = [800, 150, 40, 10]
target_tokens = 1000

def epochs_of_smallest(alpha):
    return effective_epochs(sizes, target_tokens, alpha)[-1]

# Your implementation here: scan alpha for the value giving ~2 epochs on wiki.
best = min([a / 100 for a in range(0, 101)],
           key=lambda a: abs(epochs_of_smallest(a) - 2.0))
print(f"alpha ≈ {best:.2f} → wiki seen {epochs_of_smallest(best):.2f}×")
alpha ≈ 0.82 → wiki seen 2.01×

Summary

Key takeaways:

  1. Data curation is the lever. Given fixed compute and architecture, filtering, deduplication, and mixing determine model quality more than any single modeling trick — and they run cheaply on CPUs as pure maps over documents.
  2. Quality filters are cheap heuristics, applied as a conjunction. Word count, mean word length, symbol ratio, stop words, and line repetition each catch a fingerprint of junk; a document must clear all of them. The thresholds are tuned guesses, not truths.
  3. MinHash turns similarity into a sketch. Because \(P[\text{min-hash equal}] = J(A,B)\), the fraction of matching signature positions is an unbiased Jaccard estimate with \(1/m\) variance, and LSH (\(1-(1-s^r)^b\)) finds candidates without the \(O(N^2)\) scan. Near-dedup catches the copies exact hashing misses.
  4. Mixing is a temperature dial with a repetition cost. \(p_i \propto n_i^\alpha\) interpolates from natural (\(\alpha{=}1\)) to uniform (\(\alpha{=}0\)); upsampling a small domain repeats it, so watch effective epochs, not just the weight.
  5. Every threshold is a silent curriculum choice. Curation shapes what the model can ever learn, leaves no error message when wrong, and must be measured — including against the eval sets to avoid contamination.

What’s Next

You can now build the model (m01–m06), feed it a curated corpus (this module), train it (m07), scale that training (m23), align it (m12), and serve it (m08–m09). The data frontier from here: a learned quality classifier (a small model scoring documents), synthetic data and self-distillation, data curriculum (easy-to-hard ordering), and decontamination at scale against every benchmark.

Going Deeper

Core Papers:

Practical Resources: