LLM From Scratch

AN INTERACTIVE CURRICULUM · TENSORS → TRANSFORMER → TEXT

Build a language model from first principles. This tutorial takes you from basic tensor operations to a working code-generating model.

Modules

00IntroductionWhat is a language model? 01TensorsShapes, broadcasting, operations 02AutogradGradients, chain rule, backprop 03TokenizationBPE, byte-level BPE, chat templates 04EmbeddingsVector representations 05AttentionSelf-attention, multi-head, GQA 06TransformerDecoder blocks, layer norm 07TrainingLoss, optimizers, scaling laws 08GenerationGreedy, temperature, top-k, top-p, min-p; beam search (length-normalized); contrastive search (degeneration penalty); constrained decoding (regex→FSM, pushdown JSON), KV-cache 09Efficient AttentionMQA, GQA, MLA, FlashAttention, paging, native sparse attention 10Long ContextContext extension, RoPE scaling (PI, NTK, YaRN) 11Mixture of ExpertsSparse FFN, top-k routing, load balancing 12AlignmentSFT, reward models, RLHF, DPO, PPO, IPO/KTO/ORPO/SimPO 13Reasoning & Test-Time ComputeChain-of-thought, self-consistency, best-of-N, GRPO, GSPO 14Quantizationint8/int4, per-channel, group-wise (g128), GPTQ, AWQ 15Retrieval-Augmented GenerationTF-IDF, cosine retrieval, chunking, MMR, dense bi-encoder, BM25 + hybrid (RRF), cross-encoder re-ranking, ColBERT late interaction (MaxSim) 16Speculative DecodingDraft-and-verify, rejection sampling, 2–3× faster 17Evaluationpass@k, exact match, LLM-as-judge, contamination, emergence as a metric artifact 18Tool Use & AgentsReAct loop, tool calls, safe calculator, multi-step agents 19State-Space Models (Mamba)Selective SSMs, recurrent↔︎convolutional, linear-time sequence modeling 20MultimodalViT patches, CLIP contrastive alignment, retrieval 21InterpretabilityLogit lens, residual stream, commit depth, rank/entropy 22Linear & Recurrent AttentionLinear attention, RetNet retention, chunkwise, the SSM duality, DeltaNet, gated DeltaNet, DeltaProduct, RWKV 23Distributed TrainingData parallelism, ring all-reduce, ZeRO memory partitioning, FSDP 24Pretraining DataQuality filtering, MinHash/LSH dedup, temperature-weighted mixing 25Parameter-Efficient Fine-TuningLoRA (low-rank BA update), zero-init identity, adapter merging, the r·(d+k) budget, QLoRA

Quick Start

# Install Quarto
brew install quarto

# Preview with live reload
quarto preview

# Or generate Jupyter notebooks
quarto render --to ipynb