{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Module 30: Local Inference Engines" ], "id": "e68b58f7-d09e-4f91-acea-8f28ff4b8e67" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "[Module 8](../m08_generation/lesson.qmd) generated text with a loop of\n", "`model(tokens)` calls, and [Module 14](../m14_quantization/lesson.qmd)\n", "made the weights small. Running a real model on your own machine adds a\n", "question neither module answered: **where do the bytes live, and how\n", "fast can the hardware move them?** An **inference engine** is the\n", "program that answers it. It loads the weights into the memory you have,\n", "schedules the forward passes, manages the KV cache, and turns a\n", "sampler’s output into a stream of tokens.\n", "\n", "This module is about three engines that people run locally:\n", "**llama.cpp** (C/C++, the GGUF file format, runs on almost anything),\n", "**vLLM** (Python/PyTorch, PagedAttention, built for serving many\n", "requests on NVIDIA-class GPUs) and **SGLang** (Python/PyTorch,\n", "RadixAttention prefix caching and a fast scheduler, also GPU-first). It\n", "is written for consumer hardware: one GPU with 8–32 GB of VRAM, a\n", "desktop’s system RAM, or an Apple silicon laptop with unified memory.\n", "\n", "Why it matters for LLMs:\n", "\n", "- **Speed on a desktop is a memory problem, not a compute problem.**\n", " Generating one token reads every weight once. A 4.9 GB model on a GPU\n", " with 288 GB/s of bandwidth cannot exceed about 59 tokens/s no matter\n", " how many TFLOPs the card has.\n", "- **“Does it fit” decides almost everything.** A model that fits in VRAM\n", " runs at the GPU’s bandwidth; the layers that spill into system RAM run\n", " at the RAM’s bandwidth, often 3–10× slower, and the slow part\n", " dominates.\n", "- **The engines differ most in memory management.** llama.cpp, vLLM and\n", " SGLang run the same matrix multiplies. What separates them on a given\n", " machine is how they place weights, how they allocate the KV cache, and\n", " how they batch.\n", "\n", "### What You’ll Learn\n", "\n", "After this module, you can:\n", "\n", "- Compute a model’s **file size, KV-cache size and memory placement**\n", " for any quantization, context length and GPU/RAM split, from its\n", " config alone.\n", "- Step through what happens on the hardware when a model **loads**,\n", " during **prefill** and during each **decode** step, and say which of\n", " them is memory-bound and which is compute-bound, and why.\n", "- Use the **roofline model** to predict tokens/s and to explain why\n", " batching raises throughput for free until a ridge point.\n", "- Explain how **llama.cpp**, **vLLM** and **SGLang** work internally\n", " (GGUF and layer offload; PagedAttention and continuous batching;\n", " RadixAttention and cache-aware scheduling), where they differ, and\n", " which fits which job.\n", "- Diagnose the common **bottlenecks** (partial offload, KV-cache\n", " overflow, oversubscribed VRAM, CPU threads, prompt processing) and\n", " choose the **parameters** that fix them.\n", "- Serve a **fine-tuned model** (merged weights or a LoRA adapter) with\n", " each engine.\n", "\n", "### Prerequisites\n", "\n", "This module requires familiarity with:\n", "\n", "- [Module 6: Transformer](../m06_transformer/lesson.qmd) — the block\n", " structure whose weights we count\n", "- [Module 8: Generation](../m08_generation/lesson.qmd) — the KV cache\n", " and the decode loop\n", "- [Module 9: Efficient Attention](../m09_efficient_attention/lesson.qmd)\n", " — GQA, PagedAttention, continuous batching and chunked prefill, which\n", " this module uses rather than re-derives\n", "- [Module 14: Quantization](../m14_quantization/lesson.qmd) — group-wise\n", " quantization, which GGUF’s block formats are an instance of\n", "\n", "Helpful but optional: [Module 16: Speculative\n", "Decoding](../m16_speculative_decoding/lesson.qmd) and [Module 25:\n", "Parameter-Efficient Fine-Tuning](../m25_peft/lesson.qmd).\n", "\n", "## Intuition: A Token’s Journey Through Hardware\n", "\n", "A consumer machine has three places a weight can be, and one wire\n", "between two of them:\n", "\n", "| Tier | Typical size | Typical bandwidth | Who reads it |\n", "|---------|-------------------|--------------------------|-------------------|\n", "| NVMe drive | 1–4 TB | 3–7 GB/s | the loader, once |\n", "| System RAM (dual-channel DDR4/DDR5) | 16–128 GB | 50–90 GB/s | the CPU |\n", "| PCIe link (GPU ↔ host) | — | 16–64 GB/s per direction | copies between the two |\n", "| GPU VRAM (GDDR6/6X/7) | 8–32 GB | 270–1800 GB/s | the GPU |\n", "| Apple unified memory (LPDDR5/5X) | 16–128 GB | 120–546 GB/s | CPU and GPU both |\n", "\n", "Two facts about this table drive the rest of the module.\n", "\n", "1. **The bandwidth ladder is steep.** An RTX 4090 reads its VRAM at\n", " about 1,008 GB/s; dual-channel DDR5-5600 delivers 89.6 GB/s at best.\n", " A layer that lives in system RAM takes about 11× longer to read than\n", " the same layer in VRAM.\n", "2. **Generation reads everything, every token.** To produce one token\n", " the model multiplies the current hidden state by every weight matrix\n", " in every layer. Each weight is fetched from memory, used for one\n", " multiply-add, and discarded. The time per token is therefore\n", " approximately *(bytes of weights) ÷ (bandwidth)*.\n", "\n", "> **Key Insight**\n", ">\n", "> For one user generating one token at a time, an inference engine is a\n", "> machine for streaming the model file through the memory bus as fast as\n", "> the bus allows. Every optimization in this module either makes the\n", "> stream smaller (quantization, a shorter context), moves it onto a\n", "> faster bus (offloading layers to VRAM), or makes each byte do more\n", "> work (batching, prefix caching).\n", "\n", "### Loading a model, step by step\n", "\n", "Before any token is generated the engine has to put the weights\n", "somewhere. The figure steps through a llama.cpp-style load. Pick a\n", "scenario: an 8B model that fits entirely on a 24 GB GPU, a 32B model\n", "that has to be split between a 16 GB GPU and system RAM, the same 8B\n", "model on Apple unified memory, and a CPU-only machine." ], "id": "0d2e7710-bd0a-448b-a8dc-a2ff1350b3dc" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Three things are worth noticing in the partial-offload scenario.\n", "\n", "- **Memory-mapping is not loading.** `mmap` makes the file addressable\n", " without reading it. Pages arrive from the drive the first time\n", " something touches them, so the CPU-resident layers are read lazily,\n", " and a second launch finds them already in the OS page cache.\n", "- **Offloaded layers are copied.** Every layer chosen by `-ngl` is read\n", " from the file and copied across PCIe into VRAM. This is a one-time\n", " cost per launch, bounded by the slower of the drive and the link.\n", "- **The KV cache is allocated up front.** llama.cpp reserves the cache\n", " for the full context length (`-c`) at startup, next to each layer’s\n", " weights. A long context can take gigabytes before the first token is\n", " typed, and it competes with weights for the same VRAM. `-c` defaults\n", " to 0, which means “the context length the model was trained with”:\n", " 131,072 tokens for Llama 3.1, or 16 GiB of f16 cache for the 8B model.\n", " Current llama.cpp builds run with `--fit on` by default, which shrinks\n", " any setting you did not pass explicitly (context down to `--fit-ctx`,\n", " default 4,096, and the number of GPU layers) until the model fits in\n", " free device memory with a margin (`--fit-target`, default 1,024 MiB\n", " per device).\n", "\n", "## The Math: Bytes, FLOPs and Time\n", "\n", "### Weight bytes\n", "\n", "A quantized file stores each tensor in a **block format** (Module 14’s\n", "group-wise quantization with a fixed group size and a fixed layout). The\n", "bits per weight follow from the block layout:\n", "\n", "$$\n", "\\text{bpw} = \\frac{8 \\times \\text{bytes per block}}{\\text{weights per block}}\n", "$$\n", "\n", "`Q8_0` stores 32 int8 values and one fp16 scale in 34 bytes:\n", "$8 \\times 34 / 32 = 8.5$ bpw. `Q4_K` packs 256 weights with their\n", "sub-block scales into a 144-byte super-block: $8 \\times 144 / 256 = 4.5$\n", "bpw. The presets with an `_M` suffix are **mixes**: `Q4_K_M` stores most\n", "tensors as `Q4_K` but gives `attn_v` and `ffn_down` in about half of the\n", "layers, and the output head, the 6.5625-bpw `Q6_K` type. That is why a\n", "“4-bit” file averages closer to 4.9 bits per weight.\n", "\n", "The file size is the sum over tensors:\n", "\n", "$$\n", "\\text{bytes} = \\sum_{\\text{tensors } t} n_t \\cdot \\frac{\\text{bpw}_t}{8}\n", "$$\n", "\n", "### KV-cache bytes\n", "\n", "Every layer stores one key and one value vector per KV head for every\n", "token in the context ([Module 8](../m08_generation/lesson.qmd), [Module\n", "9](../m09_efficient_attention/lesson.qmd)):\n", "\n", "$$\n", "\\text{KV bytes} = 2 \\cdot L \\cdot n_{kv} \\cdot d_{head} \\cdot n_{ctx} \\cdot b\n", "$$\n", "\n", "with $L$ layers, $n_{kv}$ KV heads, head size $d_{head}$, context length\n", "$n_{ctx}$ and $b$ bytes per element (2 for f16). For Llama 3.1 8B,\n", "$2 \\cdot 32 \\cdot 8 \\cdot 128 \\cdot 2 = 131{,}072$ bytes: **128 KiB per\n", "token**, or 1 GiB for an 8,192-token context. Grouped-query attention is\n", "what keeps this manageable; with 32 KV heads instead of 8 it would be 4×\n", "larger.\n", "\n", "### Time per token\n", "\n", "Decode reads the weights that participate in the forward pass plus the\n", "cached K/V entries, once per token. If $B_d$ bytes are read from a\n", "device with bandwidth $\\beta_d$:\n", "\n", "$$\n", "t_{\\text{token}} \\approx \\sum_{d \\in \\{\\text{GPU}, \\text{CPU}\\}} \\frac{B_d}{\\eta\\,\\beta_d}\n", "\\qquad\\qquad\n", "\\text{tok/s} = \\frac{1}{t_{\\text{token}}}\n", "$$\n", "\n", "where $\\eta \\le 1$ is the fraction of peak bandwidth the kernels\n", "actually achieve. This module uses $\\eta = 0.7$ for decode estimates.\n", "That is an assumption, not a measured constant; well-tuned kernels on a\n", "quiet GPU get closer to peak, and small models or busy systems fall\n", "further from it.\n", "\n", "### Step by Step" ], "id": "a6ed81db-d3a2-477c-bdc5-29cda152c863" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Code: The Performance Model\n", "\n", "`perf_model.py` turns those formulas into functions. The per-tensor\n", "accounting is checked against real files: the test suite asserts that\n", "the computed sizes of the Llama 3.1 8B `Q8_0`, `Q6_K`, `Q5_K_M` and\n", "`Q4_K_M` GGUFs land within 1% of the published file sizes." ], "id": "e99a172d-b8ff-45a5-9ecc-5f740130dadf" }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Llama 3.1 8B: 8.03B parameters\n", " F16 16.00 bpw 16.06 GB\n", " Q8_0 8.50 bpw 8.53 GB\n", " Q6_K 6.56 bpw 6.59 GB\n", " Q5_K_M 5.70 bpw 5.73 GB\n", " Q4_K_M 4.89 bpw 4.91 GB\n", " Q3_K 3.44 bpw 3.45 GB\n", " Q2_K 2.63 bpw 2.64 GB\n", "KV cache: 128 KiB/token (f16), 68 KiB/token (q8_0)" ] } ], "source": [ "from perf_model import (MODELS, MACHINES, weight_bytes, effective_bpw,\n", " kv_bytes_per_token, place, decode_step_time, prefill_time, GB)\n", "\n", "llama8 = MODELS[\"llama-3.1-8b\"]\n", "print(f\"{llama8.name}: {llama8.total_params() / 1e9:.2f}B parameters\")\n", "for q in [\"F16\", \"Q8_0\", \"Q6_K\", \"Q5_K_M\", \"Q4_K_M\", \"Q3_K\", \"Q2_K\"]:\n", " print(f\" {q:7s} {effective_bpw(llama8, q):5.2f} bpw {weight_bytes(llama8, q)['total'] / GB:6.2f} GB\")\n", "print(f\"KV cache: {kv_bytes_per_token(llama8, 'f16') / 1024:.0f} KiB/token (f16), \"\n", " f\"{kv_bytes_per_token(llama8, 'q8_0') / 1024:.0f} KiB/token (q8_0)\")" ], "id": "a9f3ac49" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Placement follows llama.cpp’s rule: blocks `0 … ngl-1` go to the GPU\n", "together with their slice of the KV cache, the rest stay in RAM, and\n", "`ngl = n_layers + 1` also moves the output head. With\n", "`n_gpu_layers=None` the function picks the largest value that fits." ], "id": "13546ddc-3f79-469e-bdbb-f64e8eb39fa3" }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Llama 3.1 8B on RTX 4060 Ti 16GB -ngl 33/33 VRAM 6.0 GB decode 41.2 tok/s (CPU share 0%) prefill 1505 tok/s\n", "Qwen2.5 14B on RTX 3060 -ngl 49/49 VRAM 10.6 GB decode 28.2 tok/s (CPU share 0%) prefill 465 tok/s\n", "Qwen2.5 32B on RTX 4060 Ti 16GB -ngl 44/65 VRAM 14.8 GB decode 5.8 tok/s (CPU share 62%) prefill 342 tok/s\n", "Llama 3.1 70B on RTX 4090 -ngl 41/81 VRAM 22.8 GB decode 2.7 tok/s (CPU share 92%) prefill 464 tok/s" ] } ], "source": [ "for model_key, machine_key in [(\"llama-3.1-8b\", \"rtx4060ti\"), (\"qwen2.5-14b\", \"rtx3060\"),\n", " (\"qwen2.5-32b\", \"rtx4060ti\"), (\"llama-3.1-70b\", \"rtx4090\")]:\n", " m, mc = MODELS[model_key], MACHINES[machine_key]\n", " p = place(m, mc, \"Q4_K_M\", n_ctx=8192)\n", " d = decode_step_time(p, mc, context=2000)\n", " f = prefill_time(p, mc, 2000)\n", " print(f\"{m.name:14s} on {mc.gpu.name:16s} -ngl {p.n_gpu_layers:2d}/{m.n_layers + 1:<2d} \"\n", " f\"VRAM {p.gpu_total / GB:5.1f} GB decode {d['tok_per_s']:6.1f} tok/s \"\n", " f\"(CPU share {d['cpu_s'] / d['total_s']:4.0%}) prefill {f['tok_per_s']:6.0f} tok/s\")" ], "id": "dcd4e4e2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `CPU share` column is the partial-offload story in one number. When\n", "a quarter of the layers live in system RAM they can take most of each\n", "token’s time.\n", "\n", "## Prefill and Decode on the Hardware\n", "\n", "A request has two phases with opposite performance profiles.\n", "\n", "- **Prefill** processes the prompt. All prompt tokens are known, so the\n", " engine pushes them through the network together in **micro-batches**\n", " (llama.cpp’s `-ub`, default 512). One read of a weight matrix serves\n", " every token in the micro-batch.\n", "- **Decode** generates. Each new token depends on the previous one, so a\n", " single user’s request processes exactly one token per forward pass.\n", " Every pass re-reads all of the weights for one token’s worth of\n", " arithmetic.\n", "\n", "Step through both phases. The first scenario fits on the GPU; the second\n", "is the partially offloaded 32B model from the loading figure." ], "id": "f744ba45-acca-4a21-9aad-59fd43aa7d90" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The roofline\n", "\n", "The **arithmetic intensity** of a computation is the number of FLOPs it\n", "performs per byte it moves. A weight matmul over a batch of $B$ tokens\n", "does $2B$ FLOPs per weight (one multiply and one add per token) and\n", "reads $\\text{bpw}/8$ bytes per weight:\n", "\n", "$$\n", "I = \\frac{2B}{\\text{bpw}/8} = \\frac{16B}{\\text{bpw}} \\ \\text{FLOP/byte}\n", "$$\n", "\n", "The **roofline model** (Williams, Waterman and Patterson, 2009) says a\n", "kernel’s attainable throughput is the smaller of the hardware’s peak\n", "compute and intensity × bandwidth:\n", "\n", "$$\n", "P = \\min\\left(P_{\\text{peak}},\\; I \\cdot \\beta\\right)\n", "$$\n", "\n", "The two lines meet at the **ridge point**\n", "$I^* = P_{\\text{peak}} / \\beta$. Below it, a kernel is memory-bound and\n", "more bandwidth helps; above it, compute-bound and more FLOPs help.\n", "Solving $16B/\\text{bpw} = I^*$ for $B$ gives the batch size where a\n", "weight matmul stops being memory-bound. For an RTX 4090 (165 TFLOP/s\n", "dense FP16 with FP32 accumulation, 1,008 GB/s) at 4.5 bpw that is about\n", "46 tokens. A single decode stream at $B = 1$ has intensity 3.6: it uses\n", "roughly 2% of the GPU’s arithmetic." ], "id": "1f556984-a6e9-44a0-9789-181ad5cdd813" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Key Insight**\n", ">\n", "> Prefill and batched serving sit on the flat, compute-bound part of the\n", "> roofline; single-stream decode sits far down the slope. That is why a\n", "> GPU processes a prompt at thousands of tokens per second and then\n", "> generates at tens, and why serving engines such as vLLM and SGLang get\n", "> their throughput by decoding many requests in the same forward pass.\n", "> It is also why speculative decoding ([Module\n", "> 16](../m16_speculative_decoding/lesson.qmd)) works: verifying several\n", "> drafted tokens in one pass moves decode up the slope.\n", "\n", "> **Try This**\n", ">\n", "> 1. **Find your GPU’s ridge batch.** Select the RTX 3060 and Q4_K_M.\n", "> How many concurrent streams can it decode before the weights stop\n", "> being the bottleneck?\n", "> 2. **Quantization moves the ridge.** Switch from F16 to Q4_K_M at\n", "> batch 1. The intensity rises 3.3× because the same FLOPs come from\n", "> fewer bytes, which is the second reason quantization speeds up\n", "> decode (the first is that the model fits).\n", "> 3. **Apple silicon.** Compare the M4 Max ridge with the RTX 4090’s\n", "> (the M4 Max compute figure is an approximation; Apple does not\n", "> publish one). Its bandwidth-to-compute ratio is higher, so batches\n", "> become compute-bound sooner and prompt processing is relatively\n", "> slower.\n", "\n", "## Fit and Speed Calculator\n", "\n", "Choose a model, a machine, a quantization, a KV-cache type and a context\n", "length. The calculator places the model as llama.cpp would, draws the\n", "VRAM and RAM it uses, and estimates decode and prompt-processing speed.\n", "Move `-ngl` off auto to see what each extra layer on the GPU buys, and\n", "what happens when you ask for more than fits." ], "id": "dcc1b837-77dd-470b-864c-46baf65f6b87" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **These are estimates**\n", ">\n", "> The calculator is a roofline model with assumed efficiencies (70% of\n", "> peak bandwidth for decode, 50% of peak for prefill), assumed CPU\n", "> throughput, and an approximate compute-buffer size. Bandwidths and the\n", "> RTX 3090/4090 tensor throughputs are vendor figures; the FP16\n", "> throughputs used for the RTX 3060, RTX 4060 Ti and M4 Max are\n", "> approximations, which only affect the prefill estimates. It predicts\n", "> the shape of the trade-offs and the right order of magnitude. Measure\n", "> your own machine with `llama-bench` before relying on a specific\n", "> number.\n", "\n", "## How the Three Engines Work\n", "\n", "All three engines run the same transformer forward pass. They differ in\n", "what they are built around: llama.cpp around **running anywhere with\n", "whatever memory there is**, vLLM and SGLang around **keeping a GPU busy\n", "with many requests at once**.\n", "\n", "### llama.cpp\n", "\n", "**llama.cpp** is a C/C++ inference engine built on **ggml**, a tensor\n", "library with a backend per hardware family: CUDA (NVIDIA), HIP (AMD),\n", "Metal (Apple), Vulkan, SYCL (Intel), and a CPU backend with\n", "AVX/AVX2/AVX-512/AMX on x86 and NEON on ARM. Its main properties for\n", "local use:\n", "\n", "- **One file, memory-mapped.** Models ship as **GGUF**: a header with a\n", " magic number and version, typed key-value metadata (architecture,\n", " hyperparameters, tokenizer, chat template), a table of tensor\n", " descriptors (name, shape, type, offset), then the aligned tensor data.\n", " The layout is designed so the file can be `mmap`ed and used in place.\n", " Loading is controlled by `--load-mode` (`auto` = mmap where the device\n", " allows it; `mlock` pins the pages in RAM).\n", "- **Block-quantized weights from 1.5 to 8 bits.** The `Q*_0`, `Q*_K` and\n", " `IQ*` types are group-wise formats (Module 14) whose kernels\n", " dequantize on the fly inside the matmul. `llama-quantize --imatrix`\n", " uses activation statistics from calibration text to weight the\n", " quantization error, which matters most for the low-bit `IQ` types.\n", "- **Layer offload.** `-ngl N` puts the first `N` layers and their KV\n", " cache on the GPU and runs the rest on the CPU. This is the feature\n", " that lets a 32B model run on a 16 GB card at all. `-ot`\n", " (override-tensor) places individual tensors by name pattern;\n", " `--n-cpu-moe N` keeps the expert weights of the first `N` layers of a\n", " mixture-of-experts model in RAM while attention stays on the GPU,\n", " which works well because each token only touches a few experts.\n", "- **Op offload during prefill.** When a micro-batch is large (32 tokens\n", " or more in the CUDA backend), the scheduler copies a CPU-resident\n", " layer’s weights to the GPU and runs that matmul there, because a PCIe\n", " transfer is cheaper than CPU arithmetic at that batch size.\n", " `--no-op-offload` turns this off.\n", "- **A server with slots.** `llama-server` exposes an OpenAI-compatible\n", " HTTP API. It runs up to `-np` sequences at once (automatic by default,\n", " which resolves to\n", " 1. with continuous batching. With an automatic slot count the slots\n", " share one **unified** KV buffer, so a single request can use the\n", " whole `-c`; with `--no-kv-unified` the context is split evenly\n", " across slots. Each slot keeps its last sequence, and a new request\n", " goes to the idle slot whose prompt is most similar\n", " (`--slot-prompt-similarity`), so a follow-up turn only prefills\n", " the new suffix.\n", "\n", "### vLLM\n", "\n", "**vLLM** is a Python serving engine on PyTorch with custom CUDA kernels.\n", "It started as the reference implementation of **PagedAttention** (Kwon\n", "et al., SOSP 2023) and is designed for throughput under concurrent load.\n", "\n", "- **Paged KV cache.** The cache is a pool of fixed-size blocks (16\n", " tokens by default) addressed through per-request block tables ([Module\n", " 9](../m09_efficient_attention/lesson.qmd) builds one). A request holds\n", " at most one partially filled block, so almost no memory is wasted. The\n", " paper measured that earlier systems used only 20.4–38.2% of their KV\n", " memory for actual token states, against 96.3% for vLLM.\n", "- **Preallocation by profiling.** At startup vLLM asks for\n", " `gpu_memory_utilization × total GPU memory` (default 0.92), runs a\n", " dummy forward pass to measure the weights and peak activations,\n", " subtracts an estimate for CUDA graphs, and turns everything left into\n", " KV blocks. The log line\n", " `GPU KV cache size: N tokens, Maximum concurrency for M tokens per request: X`\n", " tells you the result.\n", "- **Continuous batching and chunked prefill.** The scheduler admits and\n", " retires requests every iteration (Orca’s iteration-level scheduling),\n", " caps each step at `max_num_batched_tokens`, and splits long prompts\n", " into chunks so that running decodes are not stalled (Sarathi-Serve).\n", " Both are on by default.\n", "- **Automatic prefix caching.** Full blocks are hashed by their entire\n", " prefix; a new request whose prompt hashes to cached blocks reuses\n", " them. On by default.\n", "- **Preemption by recomputation.** When blocks run out, the most\n", " recently admitted request is evicted, its blocks freed, and it is\n", " recomputed later. Current vLLM has no swap-to-CPU path.\n", "- **CUDA graphs.** Decode steps for common batch sizes are captured as\n", " CUDA graphs so the Python and kernel-launch overhead disappears.\n", " `--enforce-eager` turns graphs (and torch.compile) off, saving memory\n", " and startup time at a per-token cost.\n", "- **Quantization from Hugging Face checkpoints.** AWQ, GPTQ (with Marlin\n", " kernels), FP8, compressed-tensors and others load directly. GGUF and\n", " bitsandbytes now require separate plugins (`vllm-gguf-plugin`,\n", " `vllm-bnb-plugin`), and the GGUF path is documented as experimental.\n", "\n", "### SGLang\n", "\n", "**SGLang** is also a Python/PyTorch serving engine (Zheng et al.,\n", "NeurIPS 2024). Its runtime shares vLLM’s goals and adds two ideas.\n", "\n", "- **RadixAttention.** Instead of hashing whole blocks, SGLang keeps\n", " every cached token sequence in a **radix tree** with one token per\n", " page. A new request walks the tree and reuses its longest cached\n", " prefix at token granularity. Nodes in use by running requests are\n", " reference-counted and pinned; when memory is needed, unreferenced\n", " leaves are evicted least-recently-used first. Cached and running\n", " tokens share one pool, so there is no separate cache budget.\n", "- **Cache-aware scheduling.** The paper’s scheduler orders waiting\n", " requests by matched prefix length (longest first), which maximizes\n", " reuse. In the current release this is `--schedule-policy lpm`; the\n", " default is `fcfs`.\n", "- **Overlap scheduling.** The CPU-side scheduling of step *t+1* runs\n", " while the GPU executes step *t*, hiding scheduler overhead (on by\n", " default, `--disable-overlap-schedule` turns it off).\n", "- **Structured generation.** SGLang was designed around programs that\n", " make many related calls (few-shot prompts, agents, JSON-constrained\n", " output), which is exactly the workload where prefix reuse pays off.\n", "\n", "### The same workload, three KV managers\n", "\n", "The figure runs one chat workload through simplified versions of all\n", "three managers (`kv_managers.py`) with the same 384 token cells each:\n", "four users, two turns each, every request starting with the same\n", "48-token system prompt, and each user’s second turn repeating the first\n", "turn plus the reply." ], "id": "64475924-ce5b-41de-a24c-05ad0dafbcaf" }, { "cell_type": "markdown", "metadata": { "learnllm-ojs-note": true }, "source": [ "> **Interactive figure**\n", ">\n", "> This visualization runs in the browser. Open the [HTML lesson](lesson.html) to explore it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What to look for:\n", "\n", "- **Slots** cap concurrency at `-np`, and reuse only what is still in\n", " the slot a request lands in. That is ideal for one user continuing one\n", " conversation and weaker for many users sharing a system prompt.\n", "- **Paged blocks** admit as many requests as there are blocks, waste\n", " less than one block per request, and reuse prefixes only in whole\n", " 16-token blocks.\n", "- **The radix tree** reuses prefixes at token granularity and keeps\n", " finished conversations until the memory is needed, so it computes the\n", " fewest prefill tokens on this workload." ], "id": "293ebb6d-99c9-4863-8d48-31511b47edbe" }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "slots prefill computed 376 / 652 prompt tokens (42% saved), peak concurrency 4, preemptions 0\n", "paged prefill computed 268 / 652 prompt tokens (59% saved), peak concurrency 5, preemptions 0\n", "radix prefill computed 208 / 652 prompt tokens (68% saved), peak concurrency 5, preemptions 0" ] } ], "source": [ "from kv_managers import compare\n", "for name, run in compare(capacity_tokens=384).items():\n", " s = run[\"summary\"]\n", " print(f\"{name:6s} prefill computed {s['prefill_computed']:4d} / {s['prompt_tokens']} prompt tokens \"\n", " f\"({s['prefill_saved_frac']:.0%} saved), peak concurrency {s['peak_concurrency']}, \"\n", " f\"preemptions {s['preemptions']}\")" ], "id": "f5311afa" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Where they differ\n", "\n", "| | llama.cpp | vLLM | SGLang |\n", "|------------------|------------------|------------------|------------------|\n", "| Language / runtime | C/C++ (ggml) | Python + PyTorch + CUDA kernels | Python + PyTorch + CUDA kernels |\n", "| Hardware | CPU, NVIDIA, AMD, Apple Metal, Vulkan, Intel, … | NVIDIA (compute capability ≥ 7.5), AMD ROCm incl. Radeon RX 7900 / 9000; Apple and CPU experimental; Windows via WSL | NVIDIA RTX 30/40/50 and datacenter; AMD Instinct; Apple via MLX (`SGLANG_USE_MLX=1`) |\n", "| Model format | GGUF | Hugging Face safetensors (AWQ, GPTQ, FP8, …) | Hugging Face safetensors; GGUF and bitsandbytes in tree |\n", "| Weights larger than VRAM | Layer offload to CPU (`-ngl`, `-ot`, `--n-cpu-moe`); CPU layers compute on the CPU | Weight streaming over PCIe each pass (`--cpu-offload-gb`) | Layer offload to RAM (`--cpu-offload-gb`, `--offload-group-size`) |\n", "| KV-cache allocation | Contiguous buffer sized by `-c`; slots share it | Paged blocks from a pool sized by profiling | Token-granular pool, radix tree |\n", "| Prefix reuse | Per slot (`--cache-prompt`), plus a RAM prompt cache | Automatic prefix caching, block granularity | RadixAttention, token granularity |\n", "| Batching | Continuous batching across `-np` slots | Continuous batching, chunked prefill | Continuous batching, chunked prefill, overlap scheduler |\n", "| Best at | One or a few users on any machine, models that do not fit in VRAM | Many concurrent requests on a GPU that holds the model | Many requests that share prefixes (agents, RAG, few-shot) |\n", "\n", "> **Key Insight**\n", ">\n", "> On a single consumer GPU with one user, all three engines are limited\n", "> by the same thing: bytes of weights per token over VRAM bandwidth. The\n", "> differences show up at the edges. When the model does not fit, only\n", "> llama.cpp’s layer offload keeps decode tolerable. When many requests\n", "> arrive at once, vLLM’s and SGLang’s schedulers turn the idle\n", "> arithmetic of the roofline into throughput.\n", "\n", "## Where Bottlenecks Happen\n", "\n", "The same few causes account for most slow or failing local setups.\n", "\n", "| Symptom | Cause | Fix |\n", "|------------------------|------------------------|------------------------|\n", "| Decode far below `bandwidth ÷ model size` | Some layers run from system RAM (partial offload) | More layers on the GPU: smaller quant, shorter `-c`, quantized KV cache, `--n-cpu-moe` for MoE models |\n", "| Fine at first, slower as the chat grows | Each token also reads the whole KV cache; at long context it rivals the weights | Shorter context, `q8_0` KV cache, a model with fewer KV heads |\n", "| Long wait before the first token | Prefill of a long prompt; on partial offload, prefill bounded by PCIe | Keep op offload on, raise `-ub`/`-b` if VRAM allows, enable prompt/prefix caching so repeated prefixes are not recomputed |\n", "| Load fails with an allocation error | Asked for more VRAM than is free (`-ngl` too high, `-c` too long, vLLM `gpu_memory_utilization` too high next to a desktop) | Let llama.cpp’s `--fit` choose, lower `-ngl`/`-c`; in vLLM lower `--gpu-memory-utilization` or `--max-model-len` |\n", "| Everything slow on Windows, GPU memory “full” | The NVIDIA driver’s system-memory fallback (driver 536.40+) moved allocations into shared system memory instead of failing | Fit the model properly; the fallback can be disabled per application in the NVIDIA Control Panel (driver 546.01+) |\n", "| Machine stutters, disk busy during generation | Weights + KV exceed RAM; the OS pages the model from disk | Smaller model or quant; `--load-mode mlock` only helps if it fits |\n", "| CPU-only decode slower than expected | Too many threads (hyperthreads, efficiency cores) contending for the same memory bus | Use physical performance cores (`-t`; the default already counts them) |\n", "| Throughput flat as users are added | Concurrency capped (`-np` in llama.cpp, `max_num_seqs`), or KV pool too small, causing preemptions | Raise the cap; give the KV pool more memory; check the vLLM concurrency log line |\n", "\n", "### Decode slows down as context grows\n", "\n", "At short context the weights dominate the bytes read per token. At long\n", "context the KV cache catches up. For Llama 3.1 8B at `Q4_K_M` the\n", "weights are 4.6 GB per token; the f16 KV cache adds 0.13 GB per 1,000\n", "tokens of context, so at 32k tokens it is almost as large as the\n", "weights." ], "id": "fa6a9567-4f3b-4ae2-b246-789933b17b75" }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "KV f16 : 0k → 151.6 tok/s 4k → 136.0 tok/s 16k → 103.8 tok/s 32k → 78.9 tok/s 64k → 53.3 tok/s\n", "KV q8_0 : 0k → 151.6 tok/s 4k → 142.9 tok/s 16k → 121.8 tok/s 32k → 101.7 tok/s 64k → 76.6 tok/s" ] } ], "source": [ "from inference_trace import decode_curve\n", "mc = MACHINES[\"rtx4090\"]\n", "for kv in [\"f16\", \"q8_0\"]:\n", " p = place(MODELS[\"llama-3.1-8b\"], mc, \"Q4_K_M\", n_ctx=65536, kv_type=kv)\n", " curve = decode_curve(p, mc, [0, 4096, 16384, 32768, 65536])\n", " print(f\"KV {kv:5s}: \" + \" \".join(f\"{c['context'] // 1024:>3d}k → {c['tok_s']:5.1f} tok/s\" for c in curve))" ], "id": "e8dff2af" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Configuration Reference\n", "\n", "The parameters that matter most on consumer hardware, with current\n", "defaults (checked against each project’s main branch in October 2026;\n", "defaults change, so confirm with `--help` on your installed version).\n", "\n", "### llama.cpp (`llama-server`, `llama-cli`)\n", "\n", "| Parameter | Default | What it controls |\n", "|------------------------|------------------------|------------------------|\n", "| `-ngl, --n-gpu-layers` | `auto` | Layers whose weights and KV live in VRAM (`all` or a number) |\n", "| `--fit` / `--fit-target` / `--fit-ctx` | `on` / 1024 MiB / 4096 | Shrink unset arguments until the model fits, keeping a margin |\n", "| `-c, --ctx-size` | 0 (model’s trained length) | KV-cache size in tokens, shared by all slots |\n", "| `-b, --batch-size` / `-ub, --ubatch-size` | 2048 / 512 | Logical and physical prompt batch; `-ub` sizes the compute buffer |\n", "| `-t` / `-tb` | physical cores | CPU threads for decode / for prompt processing |\n", "| `-fa, --flash-attn` | `auto` | Fused attention; required for a quantized V cache |\n", "| `-ctk` / `-ctv` | `f16` | KV-cache types: `f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1` |\n", "| `--load-mode` | `auto` (mmap) | `mmap`, `mlock`, `mmap+mlock`, `none`, `dio` |\n", "| `-np, --parallel` | auto (4) | Server slots: concurrent sequences |\n", "| `-kvu, --kv-unified` | on when `-np` is auto | One KV pool shared by all slots |\n", "| `--cache-prompt` / `--cache-reuse` / `--cache-ram` | on / 0 / 8192 MiB | Reuse of previous prompts in slots and in a RAM cache |\n", "| `-ot, --override-tensor` | — | Place tensors matching a regex on a given device |\n", "| `--cpu-moe` / `--n-cpu-moe N` | — | Keep MoE expert weights (all, or of the first N layers) in RAM |\n", "| `-sm` / `-ts` | `layer` / even | Multi-GPU split mode and per-GPU proportions |\n", "| `--op-offload` | on | Run large-batch matmuls of CPU layers on the GPU |\n", "| `-nkvo, --no-kv-offload` | off | Keep the KV cache in RAM even for GPU layers |\n", "| `--spec-draft-model, -md` / `--spec-draft-n-max` | — / 3 | Speculative decoding with a draft model |\n", "\n", "### vLLM (`vllm serve`)\n", "\n", "| Parameter | Default | What it controls |\n", "|------------------------|------------------------|------------------------|\n", "| `--gpu-memory-utilization` | 0.92 | Fraction of total GPU memory vLLM may use (weights + activations + KV) |\n", "| `--max-model-len` | from model config; `auto` fits to memory | Longest sequence; also bounds memory per request |\n", "| `--max-num-seqs` / `--max-num-batched-tokens` | 256 / 2048 (server, GPU \\< 70 GiB) | Concurrency and tokens per scheduler step |\n", "| `--kv-cache-dtype` | `auto` | `fp8` (e4m3) / `fp8_e5m2` halve the cache |\n", "| `--block-size` | 16 | Tokens per KV block |\n", "| `--enable-prefix-caching` | on | Block-hash prefix reuse |\n", "| `--enforce-eager` | off | Disable CUDA graphs (less memory, slower decode) |\n", "| `--cpu-offload-gb` | 0 | GiB of weights streamed from RAM every forward pass |\n", "| `--quantization, -q` | from checkpoint | `awq`, `gptq`, `fp8`, `compressed-tensors`, … |\n", "| `--tensor-parallel-size, -tp` | 1 | Split each layer across GPUs |\n", "| `--enable-lora` / `--max-loras` / `--max-lora-rank` | off / 1 / 16 | Serving LoRA adapters |\n", "| `--speculative-config` | — | JSON: `ngram`, `draft_model`, `eagle`, `eagle3`, … |\n", "\n", "### SGLang (`python -m sglang.launch_server`)\n", "\n", "| Parameter | Default | What it controls |\n", "|------------------------|------------------------|------------------------|\n", "| `--mem-fraction-static` | auto | Fraction of GPU memory for weights + KV pool |\n", "| `--context-length` | from model config | Longest sequence |\n", "| `--max-running-requests` | auto | Concurrency |\n", "| `--chunked-prefill-size` | auto (2048 on GPUs \\< 20 GB) | Prefill chunk; `-1` disables chunking |\n", "| `--schedule-policy` | `fcfs` | `lpm` = longest prefix match (the paper’s cache-aware policy) |\n", "| `--disable-radix-cache` / `--radix-eviction-policy` | off / `lru` | Radix prefix cache and its eviction policy |\n", "| `--kv-cache-dtype` | `auto` | `fp8_e4m3`, `fp8_e5m2`, … |\n", "| `--cuda-graph-max-bs-decode` | auto (8 on GPUs \\< 20 GB) | Largest decode batch captured as a CUDA graph |\n", "| `--quantization` | from checkpoint | `awq`, `gptq`, `fp8`, `gguf`, `bitsandbytes`, … |\n", "| `--lora-paths` / `--max-loras-per-batch` | — / 8 | Serving LoRA adapters |\n", "| `--speculative-algorithm` | — | `EAGLE`, `EAGLE3`, `NGRAM`, `STANDALONE`, … |\n", "\n", "## Tuning Recipes for Consumer Hardware\n", "\n", "Each recipe starts from a measurable goal. Measure before and after with\n", "`llama-bench` (llama.cpp) or the benchmark scripts shipped with vLLM and\n", "SGLang.\n", "\n", "**One user, model fits in VRAM (for example an 8B model on a 12–16 GB\n", "GPU).** Any of the three engines works; llama.cpp is the simplest. Put\n", "everything on the GPU, keep flash attention on, and set `-c` to what you\n", "actually use rather than the model’s maximum. Speed is then\n", "`VRAM bandwidth ÷ model bytes`. To go faster, use a smaller quant or\n", "speculative decoding (`--spec-draft-model` with a small model from the\n", "same family).\n", "\n", "**One user, model larger than VRAM (a 32B model on 16 GB, a 70B model on\n", "24 GB).** Use llama.cpp. Let `--fit` choose `-ngl`, then trade context\n", "for layers: a `q8_0` KV cache halves its size, and every layer moved to\n", "VRAM removes its bytes from the slow RAM path. For MoE models, keep\n", "attention and shared weights on the GPU and move experts to RAM with\n", "`--n-cpu-moe`. Expect decode speed to be set by system RAM bandwidth.\n", "\n", "**Apple silicon.** llama.cpp with Metal (on by default) uses unified\n", "memory, so there is no offload decision: the limit is how much of the\n", "unified memory macOS lets the GPU use. Decode speed follows the chip’s\n", "memory bandwidth (for example 273 GB/s on M4 Pro, 546 GB/s on the\n", "40-core M4 Max). vLLM and SGLang run on Apple silicon only through\n", "experimental or MLX-based paths.\n", "\n", "**CPU only.** Decode speed is `RAM bandwidth ÷ model bytes`: about 13\n", "tok/s upper bound for an 8B `Q4_K_M` on dual-channel DDR5 at 70%\n", "efficiency. Prompt processing is compute-bound and much slower than on a\n", "GPU. Use physical cores only, and prefer the smallest quant that keeps\n", "quality acceptable.\n", "\n", "**Several users or an agent making parallel calls, model fits in VRAM.**\n", "Use vLLM or SGLang. Set the context length to what the workload needs\n", "(`--max-model-len` / `--context-length`) so more requests fit in the KV\n", "pool, use an `fp8` KV cache if quality allows, and leave headroom for\n", "the desktop by lowering `--gpu-memory-utilization` /\n", "`--mem-fraction-static` on a GPU that also drives displays. If requests\n", "share long prefixes (a system prompt, tool definitions, retrieved\n", "documents), SGLang with `--schedule-policy lpm` reuses them at token\n", "granularity.\n", "\n", "> **Try This**\n", ">\n", "> In the calculator, take Qwen2.5 14B on the RTX 3060 at 32k context: 1.\n", "> Note the auto `-ngl` and decode speed with an `f16` KV cache. 2.\n", "> Switch the KV cache to `q8_0`. How many more layers fit, and what does\n", "> that do to tokens/s? 3. Drop the context to 8k instead. Which change\n", "> bought more speed, and what did each cost you?\n", "\n", "## Running a Fine-Tuned Model\n", "\n", "A fine-tuned model ([Module 25](../m25_peft/lesson.qmd)) reaches an\n", "engine in one of two forms.\n", "\n", "- **Merged weights.** The LoRA update $\\Delta W = BA$ is added into the\n", " base weights and saved as a normal checkpoint. It runs at exactly the\n", " base model’s speed. For llama.cpp, convert the merged Hugging Face\n", " checkpoint to GGUF with `convert_hf_to_gguf.py` and quantize it with\n", " `llama-quantize` like any other model.\n", "- **A runtime adapter.** The base model is loaded once and the small\n", " $A$, $B$ matrices are applied at inference time. llama.cpp loads GGUF\n", " adapters with `--lora` / `--lora-scaled`; vLLM serves adapters with\n", " `--enable-lora` (`--max-loras`, `--max-lora-rank`, default rank limit\n", " 16); SGLang with `--lora-paths`. Adapters let one GPU serve several\n", " fine-tunes of the same base, at the cost of extra matmuls per layer.\n", "\n", "Merge in full precision, then quantize. Adding $BA$ to weights that are\n", "already quantized would round the update away along with the rest of the\n", "weight.\n", "\n", "## Common Pitfalls\n", "\n", "1. **Leaving the context at the model’s maximum.** `-c 0` (the default)\n", " and vLLM’s default `--max-model-len` use the model’s trained length.\n", " For Llama 3.1 8B that is a 16 GiB f16 cache in llama.cpp.\n", " llama.cpp’s `--fit` shrinks it only if you did not set it yourself.\n", "2. **Forcing `-ngl 99` on a model that does not fit.** Setting `-ngl`\n", " explicitly turns off the automatic fitting for that setting. Expect\n", " an allocation error, or on Windows a silent spill into shared\n", " memory.\n", "3. **Comparing prompt processing with generation.** Benchmarks report\n", " both (`pp` and `tg` in `llama-bench`). They differ by one to two\n", " orders of magnitude and respond to different settings.\n", "4. **Quantizing the V cache with flash attention off.** llama.cpp\n", " refuses: a quantized V cache requires flash attention.\n", "5. **Expecting a serving engine to speed up one user.** vLLM’s and\n", " SGLang’s advantage is concurrency. For one stream on a model that\n", " fits, the bandwidth limit is the same for every engine.\n", "6. **Using GGUF files with vLLM by default.** vLLM’s GGUF support is\n", " now an experimental plugin; use an AWQ, GPTQ or FP8 checkpoint\n", " instead.\n", "7. **Starting vLLM on the GPU that drives your displays at the default\n", " memory fraction.** It requests 92% of total memory at startup. Lower\n", " `--gpu-memory-utilization` to leave room for the desktop.\n", "8. **Too many CPU threads.** Decode on the CPU is memory-bound; threads\n", " beyond the physical performance cores add contention, not bandwidth.\n", "\n", "## Exercises\n", "\n", "### Exercise 1: Size a deployment\n", "\n", "Compute, without the calculator, the file size, the KV cache at 16k\n", "context, and the decode upper bound of Qwen2.5 32B at `Q5_K_M` on an RTX\n", "5090 (32 GB, 1,792 GB/s). Does it fit with an f16 cache? With `q8_0`?" ], "id": "f5815085-c061-498e-bf41-91474a06da27" }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from perf_model import Device, Machine, DDR5_DUAL\n", "\n", "rtx5090 = Machine(\"RTX 5090 32 GB + DDR5\", Device(\"RTX 5090\", 32, 1792, 209.5), DDR5_DUAL, 63.0)\n", "qwen32 = MODELS[\"qwen2.5-32b\"]\n", "# Your implementation here: use weight_bytes, kv_cache_bytes and place(...)" ], "id": "bc08c48b" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2: The offload break-even\n", "\n", "For Qwen2.5 14B `Q4_K_M` on the RTX 3060, plot decode tok/s against\n", "`-ngl` from 0 to 49 using `place(..., n_gpu_layers=n)` and\n", "`decode_step_time`. Explain the shape: why does the curve rise slowly at\n", "first and steeply near full offload?" ], "id": "2ab1314d-b610-4006-a286-270f9388a453" }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# Your implementation here" ], "id": "c94c7869" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3: Make the radix cache lose\n", "\n", "Write a workload (a list of `Request`) on which `RadixManager` computes\n", "**more** prefill tokens than `PagedManager` with prefix caching. Hint:\n", "what happens when memory is tight and the scheduler’s order differs from\n", "arrival order?" ], "id": "ac9f5e33-8429-4bbf-8c83-0fb442dc9c25" }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "from kv_managers import Request, RadixManager, PagedManager, simulate\n", "# Your implementation here" ], "id": "f572c45d" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4: Batch size and the ridge\n", "\n", "Extend `decode_step_time` with a `batch` argument: the weights are read\n", "once, the KV cache once per sequence, and compute grows with the batch.\n", "At what batch size does an RTX 4060 Ti stop gaining throughput for Llama\n", "3.1 8B `Q4_K_M`? Compare with `ridge_batch`.\n", "\n", "## Summary\n", "\n", "Key takeaways:\n", "\n", "1. **Decode is memory-bound** — every generated token reads every\n", " weight once, so tok/s ≈ effective bandwidth ÷ bytes read, and\n", " quantization helps because it reads fewer bytes.\n", "2. **Prefill is compute-bound** — a micro-batch shares one read of the\n", " weights, which puts prompt processing on the flat part of the\n", " roofline.\n", "3. **Placement decides speed** — layers in VRAM run at VRAM bandwidth,\n", " layers in RAM at RAM bandwidth, and the slow part dominates; the KV\n", " cache competes with weights for VRAM and grows with context.\n", "4. **The engines differ in memory management** — llama.cpp offloads\n", " layers and runs anywhere; vLLM pages the KV cache and batches\n", " continuously; SGLang adds a token-granular radix cache for shared\n", " prefixes.\n", "5. **Most slow setups have one of a few causes** — partial offload, an\n", " oversized context, oversubscribed VRAM, or concurrency limits — and\n", " each maps to a specific parameter.\n", "\n", "### Going Deeper\n", "\n", "**Core Papers:**\n", "\n", "- [Efficient Memory Management for Large Language Model Serving with\n", " PagedAttention](https://arxiv.org/abs/2309.06180) — Kwon et al.,\n", " SOSP 2023. The vLLM paper: paged KV cache, 2–4× throughput over\n", " earlier systems.\n", "- [SGLang: Efficient Execution of Structured Language Model\n", " Programs](https://arxiv.org/abs/2312.07104) — Zheng et al.,\n", " NeurIPS 2024. RadixAttention and cache-aware scheduling, up to 6.4×\n", " throughput.\n", "- [Orca: A Distributed Serving System for Transformer-Based Generative\n", " Models](https://www.usenix.org/conference/osdi22/presentation/yu) — Yu\n", " et al., OSDI 2022. Iteration-level scheduling, the origin of\n", " continuous batching.\n", "- [Taming Throughput-Latency Tradeoff in LLM Inference with\n", " Sarathi-Serve](https://arxiv.org/abs/2403.02310) — Agrawal et al.,\n", " OSDI 2024. Chunked prefill and stall-free scheduling.\n", "- [FlashAttention: Fast and Memory-Efficient Exact Attention with\n", " IO-Awareness](https://arxiv.org/abs/2205.14135) — Dao et al., NeurIPS\n", " 2022.\n", "- [Roofline: An Insightful Visual Performance Model for Multicore\n", " Architectures](https://doi.org/10.1145/1498765.1498785) — Williams,\n", " Waterman and Patterson, CACM 2009.\n", "\n", "**Practical Resources:**\n", "\n", "- [llama.cpp](https://github.com/ggml-org/llama.cpp) — source, the\n", " server README (every flag with its default), and\n", " `tools/quantize/README.md` (bits per weight and perplexity cost of\n", " each quant type).\n", "- [GGUF\n", " specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)\n", " — the file format.\n", "- [vLLM documentation](https://docs.vllm.ai/) — engine arguments,\n", " supported hardware, quantization methods.\n", "- [SGLang documentation](https://docs.sglang.io/) — server arguments,\n", " hardware platforms.\n", "- NVIDIA\n", " [Ada](https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf)\n", " and [Ampere\n", " GA102](https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.pdf)\n", " whitepapers — bandwidth and tensor throughput figures used here.\n", "- [NVIDIA: System Memory\n", " Fallback](https://nvidia.custhelp.com/app/answers/detail/a_id/5490) —\n", " the Windows driver behavior behind silent slowdowns.\n", "\n", "## What’s Next\n", "\n", "This is the last module. [Module 29: Agentic\n", "Infrastructure](../m29_agentic_infra/lesson.qmd) builds the platform\n", "that calls an engine like these, and the\n", "[playground](../../playground.qmd) runs a small GPT entirely in your\n", "browser, where the same memory-bound decode loop is visible token by\n", "token." ], "id": "8348ac02-6cad-4a6d-84ea-d87967891ca9" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3 (ipykernel)", "language": "python", "path": "/usr/local/share/jupyter/kernels/python3" }, "language_info": { "name": "python", "codemirror_mode": { "name": "ipython", "version": "3" }, "file_extension": ".py", "mimetype": "text/x-python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.16" } } }