Module 24: Pretraining Data

Introduction

Every module so far has been about the model — how it tokenizes (m03), attends (m05), trains (m07), and scales (m23). But the corpus has always arrived pre-cleaned, as if by magic. In practice the data is the lever: given a fixed architecture and compute budget, the single biggest thing you control is what goes in. Pretraining data curation is the pipeline that turns trillions of raw, filthy web tokens into a training set worth learning from.

That pipeline is a funnel with three stages, and this module builds each from scratch:

  • Quality filtering — cheap, deterministic heuristics that throw out documents that are too short, too repetitive, symbol-spammy, or missing the function words real language always has.
  • Deduplication — the web is full of near-copies, and duplicated data makes models memorize instead of generalize. Exact dedup is a hash set; near-dedup needs MinHash, a sketch whose collision rate equals set overlap.
  • Data mixing — a corpus is many domains of unequal size. How much to sample from each is a temperature dial you will drive yourself.

Why it matters for LLMs:

  • Garbage in, garbage out — measurably. Deduplicating training data alone lowers perplexity and cuts memorization (Lee et al. 2021); quality filtering is why Gopher, Llama, and every frontier model spend more engineering on data than on the model.
  • It’s cheap and parallel. Every function here is a pure map over documents — no gradients, no GPUs — so it runs at corpus scale on CPUs.
  • The estimator is exact. MinHash isn’t a heuristic: the fraction of matching signature positions is an unbiased estimate of the Jaccard similarity, with a variance you control by making the signature longer.

What You’ll Learn

After this module, you can:

  • Implement the Gopher/C4 quality heuristics as pure functions and read a per-document pass/fail report.
  • Explain why duplicated data hurts, and build exact and near-dedup.
  • Derive the MinHash property P[\text{min-hash equal}] = J(A,B) and use it to estimate document overlap; scale it with LSH.
  • Compute temperature-weighted mixing p_i \propto n_i^\alpha and the effective epochs each domain is seen.
  • Train one elastic supernet whose prefix-sliced sub-models serve many deployment budgets from a single pretraining run.

Prerequisites

This module requires familiarity with:

Intuition: The Data Funnel

Start with Common Crawl: petabytes of raw HTML, most of it junk — SEO spam, navigation bars, boilerplate, machine translation, and endless near-duplicates. You cannot train on it directly. Each stage of the funnel is a filter that a document must survive:

raw web  ─▶  language ID  ─▶  quality filters  ─▶  dedup  ─▶  domain mixing  ─▶  training set
 (trillions)                   (drop junk)      (drop copies)   (reweight)      (what m07 sees)

The stages are cheap and composable: each is a map from documents to a keep/drop decision (or a weight). The art is in the thresholds — filter too little and you train on spam; filter too much and you strip exactly the rare, high-quality text you wanted. By the end of this lesson you will drive that whole funnel over a toy corpus and watch it shrink and clean itself.

NoteKey Insight

Data curation is not preprocessing you do once and forget — it is the model’s inductive bias about what language looks like. A filter that removes all code removes the model’s ability to write code. Every threshold is a curriculum choice.

The Math & Code: Quality Filtering

The first funnel stage is a set of heuristics from the Gopher paper (Rae et al. 2021) and C4 (Raffel et al. 2020). None of them look at meaning; they exploit cheap statistical signatures of junk. The real thresholds Gopher uses:

Heuristic Keep the document if… Catches
word count between 50 and 100,000 words stubs, dumps
mean word length between 3 and 10 characters menus, base64/URL soup
symbol-to-word ratio \le 0.1 (#, ellipses) hashtag walls, snippets
stop words \ge 2 of {the, be, to, of, and, that, have, with} keyword spam, tables
duplicate lines < 30\% of lines are repeats boilerplate footers

Each is a pure function in pretraining_data.py. They are exact and checkable:

import sys
sys.path.insert(0, ".")
from pretraining_data import (
    mean_word_length, symbol_to_word_ratio, stop_word_count,
    fraction_duplicate_lines, quality_report, CORPUS, DEMO_FILTERS,
)

spam = CORPUS[1]["text"]     # a hashtag wall
print("text:", spam)
print("mean word length:", round(mean_word_length(spam), 2))
print("symbol/word ratio:", round(symbol_to_word_ratio(spam), 2))
print("stop words:", stop_word_count(spam))
text: # # deals # # cheap # # buy now # # click # # # subscribe # # #
mean word length: 2.2
symbol/word ratio: 0.7
stop words: 0

quality_report runs every heuristic and returns which ones a document passes — so you see why it was rejected, not just that it was:

for name, ok in quality_report(spam, DEMO_FILTERS).items():
    print(f"  {name:16} {'PASS' if ok else 'FAIL'}")
  word_count       PASS
  mean_word_length FAIL
  symbol_ratio     FAIL
  stop_words       FAIL
  duplicate_lines  PASS

The hashtag wall fails three ways at once: its “words” are mostly single # characters (mean length too low), its symbol ratio is 0.7 (seven times the limit), and it has no stop words. That triple failure is the fingerprint of non-language.

WarningThese are thresholds, not truths

The 50-word minimum, the 0.1 symbol ratio — none are laws. They were tuned on English web text and will mis-fire: a valid haiku is too short, a code file is symbol-heavy, a non-English document has different stop words. Production pipelines pair these heuristics with a learned quality classifier and per-language thresholds. Filter conservatively; you can’t un-drop a document.

Walk a document through the filters

Step through the five heuristics applied to the symbol-spam document. Each row shows the measured value against the rule, and whether it passes (green) or the document dies there (red).

TipTry This
  1. Step to “symbol / word ratio”. The measured 0.7 dwarfs the 0.1 rule — one heuristic alone would have caught this.
  2. Note “word count” passes. A junk document can clear some filters; you need the whole battery. This is why quality filtering is a conjunction, not a vote.

The Math & Code: Deduplication

Near-duplicate documents are everywhere on the web — reposts, templated pages, scraped mirrors. Training on them is actively harmful: the model spends capacity memorizing repeated spans instead of learning general structure, and duplicated text inflates benchmark contamination. Lee et al. (2021) showed that deduplicating the training set lowers perplexity and cuts verbatim memorization.

Exact dedup is easy — normalize whitespace and case, hash, keep the first of each:

from pretraining_data import exact_dedup, normalize_text

docs = ["The Transformer", "the   transformer", "a different doc"]
print("normalized:", [normalize_text(d) for d in docs])
print("kept indices:", exact_dedup(docs))   # the second is a normalized copy
normalized: ['the transformer', 'the transformer', 'a different doc']
kept indices: [0, 2]

But exact dedup misses the near-copy — the same article with one word changed hashes to something completely different. For that we need a similarity that is cheap to estimate over millions of documents.

Shingles and Jaccard

Represent a document as its set of shingles — overlapping word k-grams. Two documents are near-duplicates when their shingle sets overlap heavily, measured by the Jaccard similarity:

J(A, B) = \frac{|A \cap B|}{|A \cup B|} 

from pretraining_data import shingles, jaccard

a = shingles("the cat sat on the mat", k=3)
b = shingles("the cat sat on the rug", k=3)
print("shingles A:", sorted(a))
print("Jaccard(A, B):", jaccard(a, b))
shingles A: ['cat sat on', 'on the mat', 'sat on the', 'the cat sat']
Jaccard(A, B): 0.6

Exact Jaccard is correct but expensive: comparing every pair of N documents is O(N^2) set operations, hopeless at web scale. MinHash makes each comparison O(1) over a tiny fixed-size sketch.

MinHash: the estimator that is the similarity

Pick a random hash function h and, for a set A, keep only the minimum hash value over its shingles, \min_{x \in A} h(x). The magic property (Broder, 1997):

P\big[\, \min_{x \in A} h(x) = \min_{x \in B} h(x) \,\big] \;=\; J(A, B)

The two minima coincide exactly when the overall-minimum element lies in the intersection — which happens with probability |A\cap B| / |A \cup B|. So repeat with m independent hash functions to get an m-value signature, and the fraction of matching positions is an unbiased estimate of the Jaccard similarity — with variance shrinking as 1/m. A short signature (say 128 ints) replaces the whole document.

from pretraining_data import minhash_signature, estimated_jaccard

sig_a = minhash_signature(a, num_perm=128, seed=0)
sig_b = minhash_signature(b, num_perm=128, seed=0)
print("signature length:", len(sig_a))
print("estimated Jaccard:", estimated_jaccard(sig_a, sig_b))
print("true Jaccard:     ", jaccard(a, b))
signature length: 128
estimated Jaccard: 0.6328125
true Jaccard:      0.6

Watch the estimate converge

Two documents with a genuine partial overlap (the clean web doc and its near-duplicate from the toy corpus, true J \approx 0.51). Slide the signature length: with few hash functions the estimate is noisy; as m grows it locks onto the true value — the 1/m variance made visible.

Scaling with LSH

MinHash makes each comparison cheap, but you still can’t compare all N^2 pairs. Locality-Sensitive Hashing finds the candidate near-duplicates without looking at most pairs: split the length-m signature into b bands of r rows (m = b \cdot r), and hash each band. Two documents become a candidate if they collide in any band. A pair with similarity s survives with probability

P(\text{candidate}) = 1 - (1 - s^{\,r})^{b}

an S-curve: near-0 below a tunable threshold, near-1 above it. near_dedup ties it together — LSH proposes candidates, MinHash scores them, and the later document of any near-duplicate pair is dropped:

from pretraining_data import near_dedup

corpus_texts = [d["text"] for d in CORPUS]
kept = near_dedup(corpus_texts, threshold=0.4, num_perm=128, bands=32, rows=4)
print("kept indices:", kept)
print("dropped:", [i for i in range(len(corpus_texts)) if i not in kept])
kept indices: [0, 1, 2, 5, 6]
dropped: [3, 4]

The exact duplicate (index 3) and the near-duplicate (index 4) are both gone — the near-copy that exact-dedup could never catch.

WarningDedup against the eval set, or you’ll fool yourself

The most dangerous duplicates are the ones shared between your training data and your evaluation benchmarks. If a test question leaks into pretraining, the model “solves” it by memorization and your scores are fiction (m17 calls this contamination). Always dedup the training corpus against the eval sets, not just within itself.

The Math & Code: Data Mixing

After filtering and dedup you have a clean corpus — but it is imbalanced. Web text dwarfs curated sources (books, code, Wikipedia). Train in natural proportion and the model barely sees the high-quality tail; train uniformly and you repeat tiny domains until it memorizes them. The standard lever is temperature-weighted sampling: give domain i (with n_i tokens) a sampling weight

p_i \;=\; \frac{n_i^{\,\alpha}}{\sum_j n_j^{\,\alpha}}

\alpha = 1 samples in natural proportion; \alpha = 0 is uniform over domains (maximal upsampling of small ones); 0 < \alpha < 1 interpolates.

from pretraining_data import mixing_weights, effective_epochs

sizes = [800, 150, 40, 10]        # web, books, code, wiki (billions of tokens)
print("alpha=1.0 (natural): ", [round(w, 3) for w in mixing_weights(sizes, 1.0)])
print("alpha=0.5 (tempered):", [round(w, 3) for w in mixing_weights(sizes, 0.5)])
print("alpha=0.0 (uniform): ", [round(w, 3) for w in mixing_weights(sizes, 0.0)])
alpha=1.0 (natural):  [0.8, 0.15, 0.04, 0.01]
alpha=0.5 (tempered): [0.565, 0.245, 0.126, 0.063]
alpha=0.0 (uniform):  [0.25, 0.25, 0.25, 0.25]

The catch: upsampling a small domain means repeating it. effective_epochs tells you how many times each domain is seen for a given token budget — and repetition past a few epochs is where memorization creeps back in.

eps = effective_epochs(sizes, target_tokens=1000, alpha=0.5)
for n, e in zip(["web", "books", "code", "wiki"], eps):
    print(f"  {n:6} seen {e:.2f}×")
  web    seen 0.71×
  books  seen 1.63×
  code   seen 3.16×
  wiki   seen 6.32×

At \alpha = 0.5 the 10B-token wiki domain is repeated 6.3\times to lift its share — a real risk you are trading against diversity.

Drive the mixing temperature

Slide \alpha from proportional (right) to uniform (left) and watch the domain shares reshape. The number under each bar is its effective epochs — when it climbs past a few, you’re re-showing the same tokens.

TipTry This
  1. Drag α to 0. Every domain gets an equal 25% — but wiki is now repeated many times over (its epoch count turns red). Uniform mixing trades diversity for memorization risk.
  2. Drag α to 1. Web swamps everything at 80%; the curated domains barely register. The truth is in between — most recipes use \alpha \approx 0.3–0.7.

The Recipe Upgrade: Train Once, Serve Many

Train a frontier model once and you get one model — or, with the elastic recipe, you get a family. An elastic supernet is a single set of weights trained so that its prefixes are themselves good models: take the first k channels of every matrix, or the first D layers, and you hold a smaller model that works with zero retraining. You pretrain once, then slice a model per deployment budget — phone, laptop, server — from the same run.

This is not Matryoshka Representation Learning: MRL nests the feature vector itself with a weighted sum over prefixes \sum_m c_m\,L(W^{(m)} z_{1:m}; y) (c_m{=}1, M at powers of 2, up to 14\times smaller embeddings at matching ImageNet-1K accuracy — Kusupati et al. 2205.13147), while elastic training nests deployable sub-models — full networks with their own depth, width, and sparsity that each incur the task loss. MRL shrinks one embedding; elastic training ships many models.

The math: three axes, a sandwich, and one backward pass

ERNIE 5.0 trains elasticity along three axes: depth (how many of the first layers run), width (how many channels — or, in MoE, how many experts are available), and sparsity (the top-k routed per token). Each training step samples a subnet configuration rather than always training full-size: with probability 0.75 depth stays full, with probability 0.80 width stays full, and with probability 0.80 sparsity stays full (arXiv 2602.04705 §3.3).

NoteKey Insight

The ERNIE 5.0 report disagrees with itself on the depth rate: §3.3 says full-depth with probability 0.75 (25% elastic), while §6.4.2 describes an 80/20 split. The code in elastic.py defaults to the §3.3 rates (0.75 / 0.80 / 0.80) — if you replicate the paper, note which section you followed.

The training machinery around those axes is classical — and worth naming honestly. Each step trains the sandwich set: the smallest subnet, the full supernet, plus a few random subnets, so the min and max are never starved. The subnets learn from a detach-teacher cross-entropy: the full net’s logits are detached (stopgrad) and the subnet matches them, L_{\mathrm{sub}} = \mathrm{CE}(z_{\mathrm{sub}}, \mathrm{stopgrad}(z_{\mathrm{full}})). Both of these — sandwich sampling and the detach-teacher loss — are the US-Nets machinery (1903.05134), not ERNIE findings; ERNIE’s contribution is proving the recipe at frontier scale with a shared autoregressive loss on subnet and full net in a single backward pass.

One more piece of classical machinery you should know about, even though the demo below sidesteps it: BatchNorm. A subnet’s activations have different statistics than the full net’s, so shared BatchNorm running stats are wrong for every slice — the fix in the literature is switchable BatchNorm (S-BN) or a post-training statistics pass. The demo uses LayerNorm, which keeps no running stats by design, so there is nothing stale to fix.

Code: the elastic step in elastic.py

The implementation lives in elastic.py. An ElasticMLP slices width by prefix (weight[:k_out, :k_in]) and depth by taking the first D layers — so a small net’s weights literally are the large net’s prefix:

import torch
from elastic import ElasticMLP

torch.manual_seed(0)
mlp = ElasticMLP(d_in=16, d_hidden=64, d_out=16, n_layers=4, widths=[16, 32, 64])
x = torch.randn(8, 16)
full = mlp(x, width=64, depth=4)
sub = mlp(x, width=32, depth=2)
print(f"full subnet output:  {tuple(full.shape)}")
print(f"small subnet output: {tuple(sub.shape)}")
full subnet output:  (8, 16)
small subnet output: (8, 16)

An ElasticMoE adds the third axis: top-k routing over an available-expert subset (the width axis) with a reduced k (the sparsity axis) — the m11 routing idea, made sliceable:

import torch
from elastic import ElasticMoE

torch.manual_seed(1)
moe = ElasticMoE(d_model=16, d_hidden=32, num_experts=4)
x = torch.randn(4, 8, 16)
dense = moe(x, k=4)
sparse = moe(x, k=1)
print(f"dense (k=4) output:  {tuple(dense.shape)}")
print(f"sparse (k=1) output: {tuple(sparse.shape)}")
dense (k=4) output:  (4, 8, 16)
sparse (k=1) output: (4, 8, 16)

And elastic_step is the joint update: full forward against the ground truth, sampled-subnet forward against the detached teacher, one backward pass for both:

import torch
from elastic import ElasticMLP, elastic_step

torch.manual_seed(2)
mlp = ElasticMLP(d_in=16, d_hidden=64, d_out=16, n_layers=4, widths=[16, 32, 64])
opt = torch.optim.SGD(mlp.parameters(), lr=0.01)
x = torch.randn(32, 16)
y = torch.randint(0, 16, (32,))
full_loss, sub_loss = elastic_step(mlp, x, y, opt, widths=[16, 32, 64])
print(f"full loss: {full_loss:.4f}  subnet loss: {sub_loss:.4f}")
full loss: 2.7594  subnet loss: 2.7057

Walk through one elastic step

Step through a single training iteration — sample, two forwards, one backward, then the payoff:

TipTry This
  1. Step to 2 (subnet forward). The slice runs the same weights as the full net, just fewer of them — that prefix property is the whole trick.
  2. Step to 4 (deploy slice). Nothing is retrained here. The family was paid for during pretraining; deployment just picks a prefix.

Slice the trained supernet

Now drive the real thing. The hidden cell below trains the toy supernet once and bridges the slice grid into OJS; your dial picks a width, a depth, and a top-k, and the bars show each trained slice’s loss at that k (sorted by parameter count, dashed line for the full net):

TipTry This
  1. Drop width and depth to minimum. Loss climbs — but compare against the dashed full-net line: that gap is the price of the budget.
  2. Lower top-k to 1. Every bar shifts at once, because sparsity is a third axis, not a free lunch — the cheapest slice is rarely the one you want.

Interactive Exploration

Now run the whole funnel over the toy corpus. Toggle each stage on or off and watch the corpus shrink and clean. Each document is a tile, colored by the stage that drops it (or green if it survives to training). This is demonstrate_pipeline in pretraining_data.py, bridged live.

TipTry This
  1. Turn off every stage. All 7 documents pass through — including the hashtag wall and both copies. This is training on raw web.
  2. Enable only “quality”. The spam, the stub, and the boilerplate vanish, but both copies of the clean doc survive — filtering can’t see duplication.
  3. Enable “exact-dedup” then “near-dedup”. The identical copy dies first; only near-dedup catches the reworded near-copy. Two clean documents remain.

Common Pitfalls

WarningOver-filtering strips the tail you wanted

Aggressive heuristics disproportionately remove exactly the rare, high-value text (technical prose, code, non-English) that makes a model capable. The symbol filter that kills spam also kills LaTeX and source code. Measure what a filter removes before trusting it — sample the rejects and read them.

WarningMinHash estimates, it does not certify

The signature gives an estimate of Jaccard with 1/m variance. At a threshold, some true duplicates slip through and some distinct docs get dropped. Pick m and the LSH band/row split for the precision/recall you can tolerate; there is no setting that is exact and cheap at once.

WarningUpsampling is repetition in disguise

A high mixing weight on a small domain means many epochs over the same tokens. Past a handful of repeats the model memorizes rather than generalizes — the same failure duplication caused. effective_epochs is the number to watch, not the sampling weight.

WarningShared BatchNorm lies to every slice

A subnet’s activations have different means and variances than the full net’s, so one shared set of BatchNorm running statistics is wrong for all of them. The literature fix is switchable BatchNorm (a private statistics set per slice) or a post-training calibration pass — or sidestep it entirely with LayerNorm, which is what elastic.py does.

WarningHidden size is out of scope here

Width in this module means channels (or available experts), not the hidden size of an attention block — ERNIE lists hidden-size elasticity as future work. Do not extrapolate these slicing results to attention dimensions without new machinery.

WarningCuration choices are silent and compounding

Every threshold is a curriculum decision the model can never override. Drop all code and it can’t program; over-weight forums and it learns their tone. Unlike a loss curve, a bad data mix leaves no error message — only a weaker model.

Exercises

Exercise 1: A custom quality filter

Add a heuristic that rejects documents whose fraction of numeric tokens exceeds a threshold (catches tables and data dumps). Wire it into a report.

import re
from pretraining_data import CORPUS

def numeric_fraction(text: str) -> float:
    # Your implementation here: fraction of whitespace tokens that are numbers.
    words = text.split()
    if not words:
        return 0.0
    nums = sum(1 for w in words if re.fullmatch(r"\d[\d,.]*", w))
    return nums / len(words)

print("clean doc:", round(numeric_fraction(CORPUS[0]["text"]), 3))
print("digits:   ", round(numeric_fraction("2024 2025 42 3.14 100 7 99 5"), 3))
clean doc: 0.0
digits:    1.0

Exercise 2: Tune the LSH band/row split

For a fixed signature length m = b \cdot r, the split (b, r) sets the S-curve’s threshold. Plot the collision probability at a target similarity for a few splits and pick the one whose curve rises near your dedup threshold.

from pretraining_data import lsh_collision_probability

m = 128
for bands, rows in [(64, 2), (32, 4), (16, 8)]:
    assert bands * rows == m
    at_08 = lsh_collision_probability(0.8, bands, rows)
    at_04 = lsh_collision_probability(0.4, bands, rows)
    print(f"b={bands:2} r={rows}: P(0.8)={at_08:.3f}  P(0.4)={at_04:.3f}")
# More rows per band → sharper, higher threshold. Which split best separates
# 0.8 (duplicate) from 0.4 (distinct)?
b=64 r=2: P(0.8)=1.000  P(0.4)=1.000
b=32 r=4: P(0.8)=1.000  P(0.4)=0.564
b=16 r=8: P(0.8)=0.947  P(0.4)=0.010

Exercise 3: Back out α from a target epoch count

Given domain sizes and a token budget, find the temperature \alpha that shows your smallest domain a target number of epochs — a common way to set the mix.

from pretraining_data import effective_epochs

sizes = [800, 150, 40, 10]
target_tokens = 1000

def epochs_of_smallest(alpha):
    return effective_epochs(sizes, target_tokens, alpha)[-1]

# Your implementation here: scan alpha for the value giving ~2 epochs on wiki.
best = min([a / 100 for a in range(0, 101)],
           key=lambda a: abs(epochs_of_smallest(a) - 2.0))
print(f"alpha ≈ {best:.2f} → wiki seen {epochs_of_smallest(best):.2f}×")
alpha ≈ 0.82 → wiki seen 2.01×

Exercise 4: Slice at an unseen budget

demonstrate_elastic trains on a fixed width/depth grid — but deployment budgets don’t respect your grid. Pick a (width, depth, k) triple that is not in the demo grid, estimate its loss by interpolating between the nearest trained slices, and check how far a freshly evaluated slice lands from your estimate.

from elastic import demonstrate_elastic

demo = demonstrate_elastic(seed=0)
seen = {(g["width"], g["depth"], g["k"]) for g in demo["grid"]}
print(f"{len(seen)} trained slices; full net: {demo['full']}")
# Your implementation here: pick an unseen triple, interpolate, and compare.
8 trained slices; full net: {'width': 16, 'depth': 2, 'k': 2, 'params': 2327, 'loss': 2.8417342036846094e-05}

Summary

Key takeaways:

  1. Data curation is the lever. Given fixed compute and architecture, filtering, deduplication, and mixing determine model quality more than any single modeling trick — and they run cheaply on CPUs as pure maps over documents.
  2. Quality filters are cheap heuristics, applied as a conjunction. Word count, mean word length, symbol ratio, stop words, and line repetition each catch a fingerprint of junk; a document must clear all of them. The thresholds are tuned guesses, not truths.
  3. MinHash turns similarity into a sketch. Because P[\text{min-hash equal}] = J(A,B), the fraction of matching signature positions is an unbiased Jaccard estimate with 1/m variance, and LSH (1-(1-s^r)^b) finds candidates without the O(N^2) scan. Near-dedup catches the copies exact hashing misses.
  4. Mixing is a temperature dial with a repetition cost. p_i \propto n_i^\alpha interpolates from natural (\alpha{=}1) to uniform (\alpha{=}0); upsampling a small domain repeats it, so watch effective epochs, not just the weight.
  5. Every threshold is a silent curriculum choice. Curation shapes what the model can ever learn, leaves no error message when wrong, and must be measured — including against the eval sets to avoid contamination.
  6. One run ships many models. Prefix slicing plus the sandwich set and a detach-teacher loss train a supernet whose sub-models deploy at many budgets — with ERNIE’s depth/width/sparsity full-rates (0.75 / 0.80 / 0.80) as the frontier-tested defaults.
  7. Name the machinery honestly. Sandwich sampling and inplace distillation are US-Nets classics reused at scale, not ERNIE inventions — and BatchNorm statistics (fixed via S-BN, recalibration, or LayerNorm) are the silent killer of naive slicing.

What’s Next

You can now build the model (m01–m06), feed it a curated corpus (this module), train it (m07), scale that training (m23), align it (m12), and serve it (m08–m09). The data frontier from here: a learned quality classifier (a small model scoring documents), synthetic data and self-distillation, data curriculum (easy-to-hard ordering), and decontamination at scale against every benchmark.

Going Deeper

Core Papers:

Elastic Training:

  • ERNIE 5.0 — elastic depth/width/sparsity sampling (§3.3) proving one pretraining run serves many deployment sizes.
  • Universally Slimmable Networks (US-Nets) — the sandwich set plus detach-teacher loss this module’s demo reuses, honestly labeled as classical machinery.
  • Once-for-All (OFA) — progressive shrinking and elastic dimensions (depth, width, kernel) that scaled the sandwich idea to NAS supernets.
  • Slimmable Neural Networks — switchable widths in a single model via S-BN, the origin of the shared-statistics problem.
  • Matryoshka Representation Learning (MRL) — Kusupati et al.: nested feature prefixes (up to 14\times smaller embeddings at matching accuracy), the feature-level cousin of elastic sub-models.

Practical Resources: