LLM From Scratch

AN INTERACTIVE CURRICULUM · TENSORS → TRANSFORMER → TEXT

Build a language model from first principles. This tutorial takes you from basic tensor operations to a working code-generating model.

Modules

Foundations5

00IntroductionWhat is a language model? 01TensorsShapes, broadcasting, operations 02AutogradGradients, chain rule, backprop 03TokenizationBPE, byte-level BPE, chat templates, byte-latent patching (BLT) 04EmbeddingsVector representations

Transformer4

05AttentionSelf-attention, multi-head, GQA 06TransformerDecoder blocks, layer norm 07TrainingLoss, optimizers (AdamW, Muon, MuonClip / QK-Clip), scaling laws, μP 08GenerationGreedy, temperature, top-k, top-p, min-p, typical (entropy-band), ε/η (desmoothing floor), Mirostat (perplexity feedback); beam search (length-normalized); contrastive search (degeneration penalty); constrained decoding (regex→FSM, pushdown JSON), KV-cache

Scaling3

09Efficient AttentionMQA, GQA, MLA, FlashAttention, paging, native + DeepSeek sparse + compressed (CSA/HCA) attention 10Long ContextContext extension, RoPE scaling (PI, NTK, YaRN) 11Mixture of ExpertsSparse FFN, top-k routing, load balancing

Alignment2

12AlignmentSFT, reward models, RLHF, DPO, PPO, IPO/KTO/ORPO/SimPO 13Reasoning & Test-Time ComputeChain-of-thought, self-consistency, best-of-N, GRPO, GSPO

Production5

14Quantizationint8/int4, per-channel, group-wise (g128), GPTQ, AWQ, MX (MXFP4/E8M0), NVFP4 (two-level fp4), SmoothQuant + per-token W8A8 15Retrieval-Augmented GenerationTF-IDF, cosine retrieval, chunking, MMR, dense bi-encoder, BM25 + hybrid (RRF), cross-encoder re-ranking, ColBERT late interaction (MaxSim), Matryoshka, HNSW approximate nearest neighbors 16Speculative DecodingDraft-and-verify, rejection sampling, 2–3× faster 17Evaluationpass@k, exact match, LLM-as-judge, calibration (ECE & temperature scaling), contamination, emergence as a metric artifact 18Tool Use & AgentsReAct loop, tool calls, safe calculator, multi-step agents

Frontiers12

19State-Space Models (Mamba)Selective SSMs, recurrent↔︎convolutional, linear-time sequence modeling 20MultimodalViT patches, CLIP contrastive alignment, retrieval 21InterpretabilityLogit lens, DoLa, DLA, induction heads, patching, SAEs, ICL-as-GD 22Linear & Recurrent AttentionLinear attention, RetNet retention, chunkwise, the SSM duality, gated linear attention (GLA), DeltaNet, gated DeltaNet, DeltaProduct, RWKV 23Distributed TrainingData parallelism, ring all-reduce, ZeRO memory partitioning, FSDP 24Pretraining DataQuality filtering, MinHash/LSH dedup, temperature-weighted mixing 25Parameter-Efficient Fine-TuningLoRA (low-rank BA update), zero-init identity, adapter merging, the r·(d+k) budget, QLoRA, DoRA 26Diffusion Language ModelsAbsorbing-state masking, bidirectional mask predictor, 1/t-weighted loss, confidence-ordered denoising, steps↔︎quality, self-correcting remasking (ReMDM), schedule family (cap/rescale/loop), stochastic vs deterministic remask 27Model MergingTask vectors, model soups, task arithmetic (add/negate), TIES-Merging, DARE drop-and-rescale, SLERP, Fisher-weighted averaging 28Knowledge DistillationTemperature-softened soft targets, dark knowledge, the T² distillation loss, high-T logit matching, word/sequence-level KD, on-policy GKD, R1 reasoning distillation 29Agentic Infrastructure & WorkflowsDurable runtimes (checkpoints, replay, idempotency), context budgets, memory write policies, MCP/A2A, model gateways, sandbox isolation, spans & traces, taint-tracked policy, stateful coding teams and scheduled repository improvements 30Production EvalsError analysis, correctness before tuning, instrument panels, the grader ladder, judges measured as classifiers, ground truth without labels, eval audits, agent and harness evals, and the improvement loop

Appendix1

ALocal Inference Enginesllama.cpp, vLLM and SGLang on consumer hardware: GGUF and layer offload, prefill vs decode, the roofline, KV-cache managers (slots, paged, radix), bottlenecks and tuning

Quick Start

# Install Quarto
brew install quarto

# Preview with live reload
quarto preview

# Or generate Jupyter notebooks
quarto render --to ipynb

--- title: "LLM From Scratch" subtitle: "AN INTERACTIVE CURRICULUM · TENSORS → TRANSFORMER → TEXT" --- ::: {.hero-lede} Build a language model from first principles. This tutorial takes you from basic tensor operations to a working code-generating model. ::: ::: {.index-section} ## Modules  ::: {.module-index} ::: {.arc-group .arc-foundations} [Foundations]{.arc-name}[5]{.arc-count} ::: {.arc-mods} [[00]{.module-num}[Introduction]{.module-title}[What is a language model?]{.module-desc}](modules/m00_intro/lesson.qmd){.module-row} [[01]{.module-num}[Tensors]{.module-title}[Shapes, broadcasting, operations]{.module-desc}](modules/m01_tensors/lesson.qmd){.module-row} [[02]{.module-num}[Autograd]{.module-title}[Gradients, chain rule, backprop]{.module-desc}](modules/m02_autograd/lesson.qmd){.module-row} [[03]{.module-num}[Tokenization]{.module-title}[BPE, byte-level BPE, chat templates, byte-latent patching (BLT)]{.module-desc}](modules/m03_tokenization/lesson.qmd){.module-row} [[04]{.module-num}[Embeddings]{.module-title}[Vector representations]{.module-desc}](modules/m04_embeddings/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-transformer} [Transformer]{.arc-name}[4]{.arc-count} ::: {.arc-mods} [[05]{.module-num}[Attention]{.module-title}[Self-attention, multi-head, GQA]{.module-desc}](modules/m05_attention/lesson.qmd){.module-row} [[06]{.module-num}[Transformer]{.module-title}[Decoder blocks, layer norm]{.module-desc}](modules/m06_transformer/lesson.qmd){.module-row} [[07]{.module-num}[Training]{.module-title}[Loss, optimizers (AdamW, Muon, MuonClip / QK-Clip), scaling laws, μP]{.module-desc}](modules/m07_training/lesson.qmd){.module-row} [[08]{.module-num}[Generation]{.module-title}[Greedy, temperature, top-k, top-p, min-p, typical (entropy-band), ε/η (desmoothing floor), Mirostat (perplexity feedback); beam search (length-normalized); contrastive search (degeneration penalty); constrained decoding (regex→FSM, pushdown JSON), KV-cache]{.module-desc}](modules/m08_generation/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-scaling} [Scaling]{.arc-name}[3]{.arc-count} ::: {.arc-mods} [[09]{.module-num}[Efficient Attention]{.module-title}[MQA, GQA, MLA, FlashAttention, paging, native + DeepSeek sparse + compressed (CSA/HCA) attention]{.module-desc}](modules/m09_efficient_attention/lesson.qmd){.module-row} [[10]{.module-num}[Long Context]{.module-title}[Context extension, RoPE scaling (PI, NTK, YaRN)]{.module-desc}](modules/m10_long_context/lesson.qmd){.module-row} [[11]{.module-num}[Mixture of Experts]{.module-title}[Sparse FFN, top-k routing, load balancing]{.module-desc}](modules/m11_mixture_of_experts/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-alignment} [Alignment]{.arc-name}[2]{.arc-count} ::: {.arc-mods} [[12]{.module-num}[Alignment]{.module-title}[SFT, reward models, RLHF, DPO, PPO, IPO/KTO/ORPO/SimPO]{.module-desc}](modules/m12_alignment/lesson.qmd){.module-row} [[13]{.module-num}[Reasoning & Test-Time Compute]{.module-title}[Chain-of-thought, self-consistency, best-of-N, GRPO, GSPO]{.module-desc}](modules/m13_reasoning/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-production} [Production]{.arc-name}[5]{.arc-count} ::: {.arc-mods} [[14]{.module-num}[Quantization]{.module-title}[int8/int4, per-channel, group-wise (g128), GPTQ, AWQ, MX (MXFP4/E8M0), NVFP4 (two-level fp4), SmoothQuant + per-token W8A8]{.module-desc}](modules/m14_quantization/lesson.qmd){.module-row} [[15]{.module-num}[Retrieval-Augmented Generation]{.module-title}[TF-IDF, cosine retrieval, chunking, MMR, dense bi-encoder, BM25 + hybrid (RRF), cross-encoder re-ranking, ColBERT late interaction (MaxSim), Matryoshka, HNSW approximate nearest neighbors]{.module-desc}](modules/m15_rag/lesson.qmd){.module-row} [[16]{.module-num}[Speculative Decoding]{.module-title}[Draft-and-verify, rejection sampling, 2–3× faster]{.module-desc}](modules/m16_speculative_decoding/lesson.qmd){.module-row} [[17]{.module-num}[Evaluation]{.module-title}[pass@k, exact match, LLM-as-judge, calibration (ECE & temperature scaling), contamination, emergence as a metric artifact]{.module-desc}](modules/m17_evaluation/lesson.qmd){.module-row} [[18]{.module-num}[Tool Use & Agents]{.module-title}[ReAct loop, tool calls, safe calculator, multi-step agents]{.module-desc}](modules/m18_agents/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-frontiers} [Frontiers]{.arc-name}[12]{.arc-count} ::: {.arc-mods} [[19]{.module-num}[State-Space Models (Mamba)]{.module-title}[Selective SSMs, recurrent↔convolutional, linear-time sequence modeling]{.module-desc}](modules/m19_state_space/lesson.qmd){.module-row} [[20]{.module-num}[Multimodal]{.module-title}[ViT patches, CLIP contrastive alignment, retrieval]{.module-desc}](modules/m20_multimodal/lesson.qmd){.module-row} [[21]{.module-num}[Interpretability]{.module-title}[Logit lens, DoLa, DLA, induction heads, patching, SAEs, ICL-as-GD]{.module-desc}](modules/m21_interpretability/lesson.qmd){.module-row} [[22]{.module-num}[Linear & Recurrent Attention]{.module-title}[Linear attention, RetNet retention, chunkwise, the SSM duality, gated linear attention (GLA), DeltaNet, gated DeltaNet, DeltaProduct, RWKV]{.module-desc}](modules/m22_linear_attention/lesson.qmd){.module-row} [[23]{.module-num}[Distributed Training]{.module-title}[Data parallelism, ring all-reduce, ZeRO memory partitioning, FSDP]{.module-desc}](modules/m23_distributed/lesson.qmd){.module-row} [[24]{.module-num}[Pretraining Data]{.module-title}[Quality filtering, MinHash/LSH dedup, temperature-weighted mixing]{.module-desc}](modules/m24_pretraining_data/lesson.qmd){.module-row} [[25]{.module-num}[Parameter-Efficient Fine-Tuning]{.module-title}[LoRA (low-rank BA update), zero-init identity, adapter merging, the r·(d+k) budget, QLoRA, DoRA]{.module-desc}](modules/m25_peft/lesson.qmd){.module-row} [[26]{.module-num}[Diffusion Language Models]{.module-title}[Absorbing-state masking, bidirectional mask predictor, 1/t-weighted loss, confidence-ordered denoising, steps↔quality, self-correcting remasking (ReMDM), schedule family (cap/rescale/loop), stochastic vs deterministic remask]{.module-desc}](modules/m26_diffusion/lesson.qmd){.module-row} [[27]{.module-num}[Model Merging]{.module-title}[Task vectors, model soups, task arithmetic (add/negate), TIES-Merging, DARE drop-and-rescale, SLERP, Fisher-weighted averaging]{.module-desc}](modules/m27_model_merging/lesson.qmd){.module-row} [[28]{.module-num}[Knowledge Distillation]{.module-title}[Temperature-softened soft targets, dark knowledge, the T² distillation loss, high-T logit matching, word/sequence-level KD, on-policy GKD, R1 reasoning distillation]{.module-desc}](modules/m28_distillation/lesson.qmd){.module-row} [[29]{.module-num}[Agentic Infrastructure & Workflows]{.module-title}[Durable runtimes (checkpoints, replay, idempotency), context budgets, memory write policies, MCP/A2A, model gateways, sandbox isolation, spans & traces, taint-tracked policy, stateful coding teams and scheduled repository improvements]{.module-desc}](modules/m29_agentic_infra/lesson.qmd){.module-row} [[30]{.module-num}[Production Evals]{.module-title}[Error analysis, correctness before tuning, instrument panels, the grader ladder, judges measured as classifiers, ground truth without labels, eval audits, agent and harness evals, and the improvement loop]{.module-desc}](modules/m30_production_evals/lesson.qmd){.module-row} ::: ::: ::: {.arc-group .arc-appendix} [Appendix]{.arc-name}[1]{.arc-count} ::: {.arc-mods} [[A]{.module-num}[Local Inference Engines]{.module-title}[llama.cpp, vLLM and SGLang on consumer hardware: GGUF and layer offload, prefill vs decode, the roofline, KV-cache managers (slots, paged, radix), bottlenecks and tuning]{.module-desc}](modules/appendix_a_local_inference/lesson.qmd){.module-row} ::: ::: ::: ::: ::: {.index-section} ## Quick Start ```bash # Install Quarto brew install quarto # Preview with live reload quarto preview # Or generate Jupyter notebooks quarto render --to ipynb ``` :::