LLM From Scratch

AN INTERACTIVE CURRICULUM · TENSORS → TRANSFORMER → TEXT

Build a language model from first principles. This tutorial takes you from basic tensor operations to a working code-generating model.

Modules

Foundations5

00IntroductionWhat is a language model? 01TensorsShapes, broadcasting, operations 02AutogradGradients, chain rule, backprop 03TokenizationBPE, byte-level BPE, chat templates, byte-latent patching (BLT) 04EmbeddingsVector representations

Transformer4

05AttentionSelf-attention, multi-head, GQA 06TransformerDecoder blocks, layer norm 07TrainingLoss, optimizers (AdamW, Muon, MuonClip / QK-Clip), scaling laws, μP 08GenerationGreedy, temperature, top-k, top-p, min-p; beam search (length-normalized); contrastive search (degeneration penalty); constrained decoding (regex→FSM, pushdown JSON), KV-cache

Scaling3

09Efficient AttentionMQA, GQA, MLA, FlashAttention, paging, native + DeepSeek sparse attention 10Long ContextContext extension, RoPE scaling (PI, NTK, YaRN) 11Mixture of ExpertsSparse FFN, top-k routing, load balancing

Alignment2

12AlignmentSFT, reward models, RLHF, DPO, PPO, IPO/KTO/ORPO/SimPO 13Reasoning & Test-Time ComputeChain-of-thought, self-consistency, best-of-N, GRPO, GSPO

Production5

14Quantizationint8/int4, per-channel, group-wise (g128), GPTQ, AWQ, MX (MXFP4/E8M0) 15Retrieval-Augmented GenerationTF-IDF, cosine retrieval, chunking, MMR, dense bi-encoder, BM25 + hybrid (RRF), cross-encoder re-ranking, ColBERT late interaction (MaxSim) 16Speculative DecodingDraft-and-verify, rejection sampling, 2–3× faster 17Evaluationpass@k, exact match, LLM-as-judge, contamination, emergence as a metric artifact 18Tool Use & AgentsReAct loop, tool calls, safe calculator, multi-step agents

Frontiers10

19State-Space Models (Mamba)Selective SSMs, recurrent↔︎convolutional, linear-time sequence modeling 20MultimodalViT patches, CLIP contrastive alignment, retrieval 21InterpretabilityLogit lens, residual stream, commit depth, rank/entropy 22Linear & Recurrent AttentionLinear attention, RetNet retention, chunkwise, the SSM duality, DeltaNet, gated DeltaNet, DeltaProduct, RWKV 23Distributed TrainingData parallelism, ring all-reduce, ZeRO memory partitioning, FSDP 24Pretraining DataQuality filtering, MinHash/LSH dedup, temperature-weighted mixing 25Parameter-Efficient Fine-TuningLoRA (low-rank BA update), zero-init identity, adapter merging, the r·(d+k) budget, QLoRA 26Diffusion Language ModelsAbsorbing-state masking, bidirectional mask predictor, 1/t-weighted loss, confidence-ordered denoising, steps↔︎quality 27Model MergingTask vectors, model soups, task arithmetic (add/negate), TIES-Merging, DARE drop-and-rescale, SLERP 28Knowledge DistillationTemperature-softened soft targets, dark knowledge, the T² distillation loss, high-T logit matching, word/sequence-level KD, on-policy GKD, R1 reasoning distillation

Quick Start

# Install Quarto
brew install quarto

# Preview with live reload
quarto preview

# Or generate Jupyter notebooks
quarto render --to ipynb

--- title: "LLM From Scratch" subtitle: "AN INTERACTIVE CURRICULUM · TENSORS → TRANSFORMER → TEXT" --- ::: {.hero-lede} Build a language model from first principles. This tutorial takes you from basic tensor operations to a working code-generating model. ::: ::: {.index-section} ## Modules  ::: {.module-index} ::: {.arc-group .arc-foundations} [Foundations]{.arc-name}[5]{.arc-count} ::: {.arc-mods} [[00]{.module-num}[Introduction]{.module-title}[What is a language model?]{.module-desc}](modules/m00_intro/lesson.qmd){.module-row} [[01]{.module-num}[Tensors]{.module-title}[Shapes, broadcasting, operations]{.module-desc}](modules/m01_tensors/lesson.qmd){.module-row} [[02]{.module-num}[Autograd]{.module-title}[Gradients, chain rule, backprop]{.module-desc}](modules/m02_autograd/lesson.qmd){.module-row} [[03]{.module-num}[Tokenization]{.module-title}[BPE, byte-level BPE, chat templates, byte-latent patching (BLT)]{.module-desc}](modules/m03_tokenization/lesson.qmd){.module-row} [[04]{.module-num}[Embeddings]{.module-title}[Vector representations]{.module-desc}](modules/m04_embeddings/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-transformer} [Transformer]{.arc-name}[4]{.arc-count} ::: {.arc-mods} [[05]{.module-num}[Attention]{.module-title}[Self-attention, multi-head, GQA]{.module-desc}](modules/m05_attention/lesson.qmd){.module-row} [[06]{.module-num}[Transformer]{.module-title}[Decoder blocks, layer norm]{.module-desc}](modules/m06_transformer/lesson.qmd){.module-row} [[07]{.module-num}[Training]{.module-title}[Loss, optimizers (AdamW, Muon, MuonClip / QK-Clip), scaling laws, μP]{.module-desc}](modules/m07_training/lesson.qmd){.module-row} [[08]{.module-num}[Generation]{.module-title}[Greedy, temperature, top-k, top-p, min-p; beam search (length-normalized); contrastive search (degeneration penalty); constrained decoding (regex→FSM, pushdown JSON), KV-cache]{.module-desc}](modules/m08_generation/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-scaling} [Scaling]{.arc-name}[3]{.arc-count} ::: {.arc-mods} [[09]{.module-num}[Efficient Attention]{.module-title}[MQA, GQA, MLA, FlashAttention, paging, native + DeepSeek sparse attention]{.module-desc}](modules/m09_efficient_attention/lesson.qmd){.module-row} [[10]{.module-num}[Long Context]{.module-title}[Context extension, RoPE scaling (PI, NTK, YaRN)]{.module-desc}](modules/m10_long_context/lesson.qmd){.module-row} [[11]{.module-num}[Mixture of Experts]{.module-title}[Sparse FFN, top-k routing, load balancing]{.module-desc}](modules/m11_mixture_of_experts/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-alignment} [Alignment]{.arc-name}[2]{.arc-count} ::: {.arc-mods} [[12]{.module-num}[Alignment]{.module-title}[SFT, reward models, RLHF, DPO, PPO, IPO/KTO/ORPO/SimPO]{.module-desc}](modules/m12_alignment/lesson.qmd){.module-row} [[13]{.module-num}[Reasoning & Test-Time Compute]{.module-title}[Chain-of-thought, self-consistency, best-of-N, GRPO, GSPO]{.module-desc}](modules/m13_reasoning/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-production} [Production]{.arc-name}[5]{.arc-count} ::: {.arc-mods} [[14]{.module-num}[Quantization]{.module-title}[int8/int4, per-channel, group-wise (g128), GPTQ, AWQ, MX (MXFP4/E8M0)]{.module-desc}](modules/m14_quantization/lesson.qmd){.module-row} [[15]{.module-num}[Retrieval-Augmented Generation]{.module-title}[TF-IDF, cosine retrieval, chunking, MMR, dense bi-encoder, BM25 + hybrid (RRF), cross-encoder re-ranking, ColBERT late interaction (MaxSim)]{.module-desc}](modules/m15_rag/lesson.qmd){.module-row} [[16]{.module-num}[Speculative Decoding]{.module-title}[Draft-and-verify, rejection sampling, 2–3× faster]{.module-desc}](modules/m16_speculative_decoding/lesson.qmd){.module-row} [[17]{.module-num}[Evaluation]{.module-title}[pass@k, exact match, LLM-as-judge, contamination, emergence as a metric artifact]{.module-desc}](modules/m17_evaluation/lesson.qmd){.module-row} [[18]{.module-num}[Tool Use & Agents]{.module-title}[ReAct loop, tool calls, safe calculator, multi-step agents]{.module-desc}](modules/m18_agents/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-frontiers} [Frontiers]{.arc-name}[10]{.arc-count} ::: {.arc-mods} [[19]{.module-num}[State-Space Models (Mamba)]{.module-title}[Selective SSMs, recurrent↔convolutional, linear-time sequence modeling]{.module-desc}](modules/m19_state_space/lesson.qmd){.module-row} [[20]{.module-num}[Multimodal]{.module-title}[ViT patches, CLIP contrastive alignment, retrieval]{.module-desc}](modules/m20_multimodal/lesson.qmd){.module-row} [[21]{.module-num}[Interpretability]{.module-title}[Logit lens, residual stream, commit depth, rank/entropy]{.module-desc}](modules/m21_interpretability/lesson.qmd){.module-row} [[22]{.module-num}[Linear & Recurrent Attention]{.module-title}[Linear attention, RetNet retention, chunkwise, the SSM duality, DeltaNet, gated DeltaNet, DeltaProduct, RWKV]{.module-desc}](modules/m22_linear_attention/lesson.qmd){.module-row} [[23]{.module-num}[Distributed Training]{.module-title}[Data parallelism, ring all-reduce, ZeRO memory partitioning, FSDP]{.module-desc}](modules/m23_distributed/lesson.qmd){.module-row} [[24]{.module-num}[Pretraining Data]{.module-title}[Quality filtering, MinHash/LSH dedup, temperature-weighted mixing]{.module-desc}](modules/m24_pretraining_data/lesson.qmd){.module-row} [[25]{.module-num}[Parameter-Efficient Fine-Tuning]{.module-title}[LoRA (low-rank BA update), zero-init identity, adapter merging, the r·(d+k) budget, QLoRA]{.module-desc}](modules/m25_peft/lesson.qmd){.module-row} [[26]{.module-num}[Diffusion Language Models]{.module-title}[Absorbing-state masking, bidirectional mask predictor, 1/t-weighted loss, confidence-ordered denoising, steps↔quality]{.module-desc}](modules/m26_diffusion/lesson.qmd){.module-row} [[27]{.module-num}[Model Merging]{.module-title}[Task vectors, model soups, task arithmetic (add/negate), TIES-Merging, DARE drop-and-rescale, SLERP]{.module-desc}](modules/m27_model_merging/lesson.qmd){.module-row} [[28]{.module-num}[Knowledge Distillation]{.module-title}[Temperature-softened soft targets, dark knowledge, the T² distillation loss, high-T logit matching, word/sequence-level KD, on-policy GKD, R1 reasoning distillation]{.module-desc}](modules/m28_distillation/lesson.qmd){.module-row} ::: ::: ::: ::: ::: {.index-section} ## Quick Start ```bash # Install Quarto brew install quarto # Preview with live reload quarto preview # Or generate Jupyter notebooks quarto render --to ipynb ``` :::