LLM From Scratch

Build a language model from first principles. This tutorial takes you from basic tensor operations to a working code-generating model.

Modules

Module	Topic	Description
00	Introduction	What is a language model?
01	Tensors	Shapes, broadcasting, operations
02	Autograd	Gradients, chain rule, backprop
03	Tokenization	BPE, byte-level BPE, chat templates
04	Embeddings	Vector representations
05	Attention	Self-attention, multi-head, GQA
06	Transformer	Decoder blocks, layer norm
07	Training	Loss, optimizers, scaling laws
08	Generation	Greedy, temperature, top-k, top-p, min-p
09	Efficient Attention	MQA, GQA, FlashAttention
10	Long Context	Context extension, RoPE scaling (PI, NTK, YaRN)
11	Mixture of Experts	Sparse FFN, top-k routing, load balancing
12	Alignment	SFT, reward models, RLHF, DPO
13	Reasoning & Test-Time Compute	Chain-of-thought, self-consistency, best-of-N
14	Quantization	int8/int4, per-channel, QuantizedLinear
15	Retrieval-Augmented Generation	TF-IDF, cosine retrieval, chunking, MMR
16	Speculative Decoding	Draft-and-verify, rejection sampling, 2–3× faster
17	Evaluation	pass@k, exact match, LLM-as-judge, contamination
18	Tool Use & Agents	ReAct loop, tool calls, safe calculator, multi-step agents
19	State-Space Models (Mamba)	Selective SSMs, recurrent↔︎convolutional, linear-time sequence modeling
20	Multimodal	ViT patches, CLIP contrastive alignment, retrieval
21	Interpretability	Logit lens, residual stream, commit depth, rank/entropy
22	Linear & Recurrent Attention	Linear attention, RetNet retention, chunkwise, the SSM duality
23	Distributed Training	Data parallelism, ring all-reduce, ZeRO memory partitioning, FSDP

Quick Start

# Install Quarto
brew install quarto

# Preview with live reload
quarto preview

# Or generate Jupyter notebooks
quarto render --to ipynb