LLM From Scratch

AN INTERACTIVE CURRICULUM · TENSORS → TRANSFORMER → TEXT

Build a language model from first principles. This tutorial takes you from basic tensor operations to a working code-generating model.

Modules

Foundations5

00IntroductionWhat is a language model? 01TensorsShapes, broadcasting, operations 02AutogradGradients, chain rule, backprop 03TokenizationBPE, byte-level BPE, chat templates, byte-latent patching (BLT) 04EmbeddingsVector representations

Transformer4

05AttentionSelf-attention, multi-head, GQA 06TransformerDecoder blocks, layer norm 07TrainingLoss, optimizers (AdamW, Muon, MuonClip / QK-Clip), scaling laws, μP 08GenerationGreedy, temperature, top-k, top-p, min-p, typical (entropy-band), ε/η (desmoothing floor), Mirostat (perplexity feedback); beam search (length-normalized); contrastive search (degeneration penalty); constrained decoding (regex→FSM, pushdown JSON), KV-cache

Scaling3

09Efficient AttentionMQA, GQA, MLA, FlashAttention, paging, native + DeepSeek sparse + compressed (CSA/HCA) attention 10Long ContextContext extension, RoPE scaling (PI, NTK, YaRN) 11Mixture of ExpertsSparse FFN, top-k routing, load balancing

Alignment2

12AlignmentSFT, reward models, RLHF, DPO, PPO, IPO/KTO/ORPO/SimPO 13Reasoning & Test-Time ComputeChain-of-thought, self-consistency, best-of-N, GRPO, GSPO

Production5

14Quantizationint8/int4, per-channel, group-wise (g128), GPTQ, AWQ, MX (MXFP4/E8M0), NVFP4 (two-level fp4), SmoothQuant + per-token W8A8 15Retrieval-Augmented GenerationTF-IDF, cosine retrieval, chunking, MMR, dense bi-encoder, BM25 + hybrid (RRF), cross-encoder re-ranking, ColBERT late interaction (MaxSim), Matryoshka, HNSW approximate nearest neighbors 16Speculative DecodingDraft-and-verify, rejection sampling, 2–3× faster 17Evaluationpass@k, exact match, LLM-as-judge, calibration (ECE & temperature scaling), contamination, emergence as a metric artifact 18Tool Use & AgentsReAct loop, tool calls, safe calculator, multi-step agents

Frontiers12

19State-Space Models (Mamba)Selective SSMs, recurrent↔︎convolutional, linear-time sequence modeling 20MultimodalViT patches, CLIP contrastive alignment, retrieval 21InterpretabilityLogit lens, DoLa, DLA, induction heads, patching, SAEs, ICL-as-GD 22Linear & Recurrent AttentionLinear attention, RetNet retention, chunkwise, the SSM duality, gated linear attention (GLA), DeltaNet, gated DeltaNet, DeltaProduct, RWKV 23Distributed TrainingData parallelism, ring all-reduce, ZeRO memory partitioning, FSDP 24Pretraining DataQuality filtering, MinHash/LSH dedup, temperature-weighted mixing 25Parameter-Efficient Fine-TuningLoRA (low-rank BA update), zero-init identity, adapter merging, the r·(d+k) budget, QLoRA, DoRA 26Diffusion Language ModelsAbsorbing-state masking, bidirectional mask predictor, 1/t-weighted loss, confidence-ordered denoising, steps↔︎quality, self-correcting remasking (ReMDM), schedule family (cap/rescale/loop), stochastic vs deterministic remask 27Model MergingTask vectors, model soups, task arithmetic (add/negate), TIES-Merging, DARE drop-and-rescale, SLERP, Fisher-weighted averaging 28Knowledge DistillationTemperature-softened soft targets, dark knowledge, the T² distillation loss, high-T logit matching, word/sequence-level KD, on-policy GKD, R1 reasoning distillation 29Agentic Infrastructure & WorkflowsDurable runtimes (checkpoints, replay, idempotency), context budgets, memory write policies, MCP/A2A, model gateways, sandbox isolation, spans & agent evals, taint-tracked policy, stateful coding teams and scheduled repository improvements 30Local Inference Enginesllama.cpp, vLLM and SGLang on consumer hardware: GGUF and layer offload, prefill vs decode, the roofline, KV-cache managers (slots, paged, radix), bottlenecks and tuning

Quick Start

# Install Quarto
brew install quarto

# Preview with live reload
quarto preview

# Or generate Jupyter notebooks
quarto render --to ipynb