Module 30: Local Inference Engines

Introduction

Module 8 generated text with a loop of model(tokens) calls, and Module 14 made the weights small. Running a real model on your own machine adds a question neither module answered: where do the bytes live, and how fast can the hardware move them? An inference engine is the program that answers it. It loads the weights into the memory you have, schedules the forward passes, manages the KV cache, and turns a sampler’s output into a stream of tokens.

This module is about three engines that people run locally: llama.cpp (C/C++, the GGUF file format, runs on almost anything), vLLM (Python/PyTorch, PagedAttention, built for serving many requests on NVIDIA-class GPUs) and SGLang (Python/PyTorch, RadixAttention prefix caching and a fast scheduler, also GPU-first). It is written for consumer hardware: one GPU with 8–32 GB of VRAM, a desktop’s system RAM, or an Apple silicon laptop with unified memory.

Why it matters for LLMs:

  • Speed on a desktop is a memory problem, not a compute problem. Generating one token reads every weight once. A 4.9 GB model on a GPU with 288 GB/s of bandwidth cannot exceed about 59 tokens/s no matter how many TFLOPs the card has.
  • “Does it fit” decides almost everything. A model that fits in VRAM runs at the GPU’s bandwidth; the layers that spill into system RAM run at the RAM’s bandwidth, often 3–10× slower, and the slow part dominates.
  • The engines differ most in memory management. llama.cpp, vLLM and SGLang run the same matrix multiplies. What separates them on a given machine is how they place weights, how they allocate the KV cache, and how they batch.

What You’ll Learn

After this module, you can:

  • Compute a model’s file size, KV-cache size and memory placement for any quantization, context length and GPU/RAM split, from its config alone.
  • Step through what happens on the hardware when a model loads, during prefill and during each decode step, and say which of them is memory-bound and which is compute-bound, and why.
  • Use the roofline model to predict tokens/s and to explain why batching raises throughput for free until a ridge point.
  • Explain how llama.cpp, vLLM and SGLang work internally (GGUF and layer offload; PagedAttention and continuous batching; RadixAttention and cache-aware scheduling), where they differ, and which fits which job.
  • Diagnose the common bottlenecks (partial offload, KV-cache overflow, oversubscribed VRAM, CPU threads, prompt processing) and choose the parameters that fix them.
  • Serve a fine-tuned model (merged weights or a LoRA adapter) with each engine.

Prerequisites

This module requires familiarity with:

Helpful but optional: Module 16: Speculative Decoding and Module 25: Parameter-Efficient Fine-Tuning.

Intuition: A Token’s Journey Through Hardware

A consumer machine has three places a weight can be, and one wire between two of them:

Tier Typical size Typical bandwidth Who reads it
NVMe drive 1–4 TB 3–7 GB/s the loader, once
System RAM (dual-channel DDR4/DDR5) 16–128 GB 50–90 GB/s the CPU
PCIe link (GPU ↔︎ host) — 16–64 GB/s per direction copies between the two
GPU VRAM (GDDR6/6X/7) 8–32 GB 270–1800 GB/s the GPU
Apple unified memory (LPDDR5/5X) 16–128 GB 120–546 GB/s CPU and GPU both

Two facts about this table drive the rest of the module.

  1. The bandwidth ladder is steep. An RTX 4090 reads its VRAM at about 1,008 GB/s; dual-channel DDR5-5600 delivers 89.6 GB/s at best. A layer that lives in system RAM takes about 11× longer to read than the same layer in VRAM.
  2. Generation reads everything, every token. To produce one token the model multiplies the current hidden state by every weight matrix in every layer. Each weight is fetched from memory, used for one multiply-add, and discarded. The time per token is therefore approximately (bytes of weights) ÷ (bandwidth).
NoteKey Insight

For one user generating one token at a time, an inference engine is a machine for streaming the model file through the memory bus as fast as the bus allows. Every optimization in this module either makes the stream smaller (quantization, a shorter context), moves it onto a faster bus (offloading layers to VRAM), or makes each byte do more work (batching, prefix caching).

Loading a model, step by step

Before any token is generated the engine has to put the weights somewhere. The figure steps through a llama.cpp-style load. Pick a scenario: an 8B model that fits entirely on a 24 GB GPU, a 32B model that has to be split between a 16 GB GPU and system RAM, the same 8B model on Apple unified memory, and a CPU-only machine.

Three things are worth noticing in the partial-offload scenario.

  • Memory-mapping is not loading. mmap makes the file addressable without reading it. Pages arrive from the drive the first time something touches them, so the CPU-resident layers are read lazily, and a second launch finds them already in the OS page cache.
  • Offloaded layers are copied. Every layer chosen by -ngl is read from the file and copied across PCIe into VRAM. This is a one-time cost per launch, bounded by the slower of the drive and the link.
  • The KV cache is allocated up front. llama.cpp reserves the cache for the full context length (-c) at startup, next to each layer’s weights. A long context can take gigabytes before the first token is typed, and it competes with weights for the same VRAM. -c defaults to 0, which means “the context length the model was trained with”: 131,072 tokens for Llama 3.1, or 16 GiB of f16 cache for the 8B model. Current llama.cpp builds run with --fit on by default, which shrinks any setting you did not pass explicitly (context down to --fit-ctx, default 4,096, and the number of GPU layers) until the model fits in free device memory with a margin (--fit-target, default 1,024 MiB per device).

The Math: Bytes, FLOPs and Time

Weight bytes

A quantized file stores each tensor in a block format (Module 14’s group-wise quantization with a fixed group size and a fixed layout). The bits per weight follow from the block layout:

\text{bpw} = \frac{8 \times \text{bytes per block}}{\text{weights per block}}

Q8_0 stores 32 int8 values and one fp16 scale in 34 bytes: 8 \times 34 / 32 = 8.5 bpw. Q4_K packs 256 weights with their sub-block scales into a 144-byte super-block: 8 \times 144 / 256 = 4.5 bpw. The presets with an _M suffix are mixes: Q4_K_M stores most tensors as Q4_K but gives attn_v and ffn_down in about half of the layers, and the output head, the 6.5625-bpw Q6_K type. That is why a “4-bit” file averages closer to 4.9 bits per weight.

The file size is the sum over tensors:

\text{bytes} = \sum_{\text{tensors } t} n_t \cdot \frac{\text{bpw}_t}{8}

KV-cache bytes

Every layer stores one key and one value vector per KV head for every token in the context (Module 8, Module 9):

\text{KV bytes} = 2 \cdot L \cdot n_{kv} \cdot d_{head} \cdot n_{ctx} \cdot b

with L layers, n_{kv} KV heads, head size d_{head}, context length n_{ctx} and b bytes per element (2 for f16). For Llama 3.1 8B, 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072 bytes: 128 KiB per token, or 1 GiB for an 8,192-token context. Grouped-query attention is what keeps this manageable; with 32 KV heads instead of 8 it would be 4× larger.

Time per token

Decode reads the weights that participate in the forward pass plus the cached K/V entries, once per token. If B_d bytes are read from a device with bandwidth \beta_d:

t_{\text{token}} \approx \sum_{d \in \{\text{GPU}, \text{CPU}\}} \frac{B_d}{\eta\,\beta_d} \qquad\qquad \text{tok/s} = \frac{1}{t_{\text{token}}}

where \eta \le 1 is the fraction of peak bandwidth the kernels actually achieve. This module uses \eta = 0.7 for decode estimates. That is an assumption, not a measured constant; well-tuned kernels on a quiet GPU get closer to peak, and small models or busy systems fall further from it.

Step by Step

Code: The Performance Model

perf_model.py turns those formulas into functions. The per-tensor accounting is checked against real files: the test suite asserts that the computed sizes of the Llama 3.1 8B Q8_0, Q6_K, Q5_K_M and Q4_K_M GGUFs land within 1% of the published file sizes.

from perf_model import (MODELS, MACHINES, weight_bytes, effective_bpw,
                        kv_bytes_per_token, place, decode_step_time, prefill_time, GB)

llama8 = MODELS["llama-3.1-8b"]
print(f"{llama8.name}: {llama8.total_params() / 1e9:.2f}B parameters")
for q in ["F16", "Q8_0", "Q6_K", "Q5_K_M", "Q4_K_M", "Q3_K", "Q2_K"]:
    print(f"  {q:7s} {effective_bpw(llama8, q):5.2f} bpw  {weight_bytes(llama8, q)['total'] / GB:6.2f} GB")
print(f"KV cache: {kv_bytes_per_token(llama8, 'f16') / 1024:.0f} KiB/token (f16), "
      f"{kv_bytes_per_token(llama8, 'q8_0') / 1024:.0f} KiB/token (q8_0)")
Llama 3.1 8B: 8.03B parameters
  F16     16.00 bpw   16.06 GB
  Q8_0     8.50 bpw    8.53 GB
  Q6_K     6.56 bpw    6.59 GB
  Q5_K_M   5.70 bpw    5.73 GB
  Q4_K_M   4.89 bpw    4.91 GB
  Q3_K     3.44 bpw    3.45 GB
  Q2_K     2.63 bpw    2.64 GB
KV cache: 128 KiB/token (f16), 68 KiB/token (q8_0)

Placement follows llama.cpp’s rule: blocks 0 … ngl-1 go to the GPU together with their slice of the KV cache, the rest stay in RAM, and ngl = n_layers + 1 also moves the output head. With n_gpu_layers=None the function picks the largest value that fits.

for model_key, machine_key in [("llama-3.1-8b", "rtx4060ti"), ("qwen2.5-14b", "rtx3060"),
                               ("qwen2.5-32b", "rtx4060ti"), ("llama-3.1-70b", "rtx4090")]:
    m, mc = MODELS[model_key], MACHINES[machine_key]
    p = place(m, mc, "Q4_K_M", n_ctx=8192)
    d = decode_step_time(p, mc, context=2000)
    f = prefill_time(p, mc, 2000)
    print(f"{m.name:14s} on {mc.gpu.name:16s} -ngl {p.n_gpu_layers:2d}/{m.n_layers + 1:<2d} "
          f"VRAM {p.gpu_total / GB:5.1f} GB  decode {d['tok_per_s']:6.1f} tok/s  "
          f"(CPU share {d['cpu_s'] / d['total_s']:4.0%})  prefill {f['tok_per_s']:6.0f} tok/s")
Llama 3.1 8B   on RTX 4060 Ti 16GB -ngl 33/33 VRAM   6.0 GB  decode   41.2 tok/s  (CPU share   0%)  prefill   1505 tok/s
Qwen2.5 14B    on RTX 3060         -ngl 49/49 VRAM  10.6 GB  decode   28.2 tok/s  (CPU share   0%)  prefill    465 tok/s
Qwen2.5 32B    on RTX 4060 Ti 16GB -ngl 44/65 VRAM  14.8 GB  decode    5.8 tok/s  (CPU share  62%)  prefill    342 tok/s
Llama 3.1 70B  on RTX 4090         -ngl 41/81 VRAM  22.8 GB  decode    2.7 tok/s  (CPU share  92%)  prefill    464 tok/s

The CPU share column is the partial-offload story in one number. When a quarter of the layers live in system RAM they can take most of each token’s time.

Prefill and Decode on the Hardware

A request has two phases with opposite performance profiles.

  • Prefill processes the prompt. All prompt tokens are known, so the engine pushes them through the network together in micro-batches (llama.cpp’s -ub, default 512). One read of a weight matrix serves every token in the micro-batch.
  • Decode generates. Each new token depends on the previous one, so a single user’s request processes exactly one token per forward pass. Every pass re-reads all of the weights for one token’s worth of arithmetic.

Step through both phases. The first scenario fits on the GPU; the second is the partially offloaded 32B model from the loading figure.

The roofline

The arithmetic intensity of a computation is the number of FLOPs it performs per byte it moves. A weight matmul over a batch of B tokens does 2B FLOPs per weight (one multiply and one add per token) and reads \text{bpw}/8 bytes per weight:

I = \frac{2B}{\text{bpw}/8} = \frac{16B}{\text{bpw}} \ \text{FLOP/byte}

The roofline model (Williams, Waterman and Patterson, 2009) says a kernel’s attainable throughput is the smaller of the hardware’s peak compute and intensity × bandwidth:

P = \min\left(P_{\text{peak}},\; I \cdot \beta\right)

The two lines meet at the ridge point I^* = P_{\text{peak}} / \beta. Below it, a kernel is memory-bound and more bandwidth helps; above it, compute-bound and more FLOPs help. Solving 16B/\text{bpw} = I^* for B gives the batch size where a weight matmul stops being memory-bound. For an RTX 4090 (165 TFLOP/s dense FP16 with FP32 accumulation, 1,008 GB/s) at 4.5 bpw that is about 46 tokens. A single decode stream at B = 1 has intensity 3.6: it uses roughly 2% of the GPU’s arithmetic.

NoteKey Insight

Prefill and batched serving sit on the flat, compute-bound part of the roofline; single-stream decode sits far down the slope. That is why a GPU processes a prompt at thousands of tokens per second and then generates at tens, and why serving engines such as vLLM and SGLang get their throughput by decoding many requests in the same forward pass. It is also why speculative decoding (Module 16) works: verifying several drafted tokens in one pass moves decode up the slope.

TipTry This
  1. Find your GPU’s ridge batch. Select the RTX 3060 and Q4_K_M. How many concurrent streams can it decode before the weights stop being the bottleneck?
  2. Quantization moves the ridge. Switch from F16 to Q4_K_M at batch 1. The intensity rises 3.3× because the same FLOPs come from fewer bytes, which is the second reason quantization speeds up decode (the first is that the model fits).
  3. Apple silicon. Compare the M4 Max ridge with the RTX 4090’s (the M4 Max compute figure is an approximation; Apple does not publish one). Its bandwidth-to-compute ratio is higher, so batches become compute-bound sooner and prompt processing is relatively slower.

Fit and Speed Calculator

Choose a model, a machine, a quantization, a KV-cache type and a context length. The calculator places the model as llama.cpp would, draws the VRAM and RAM it uses, and estimates decode and prompt-processing speed. Move -ngl off auto to see what each extra layer on the GPU buys, and what happens when you ask for more than fits.

WarningThese are estimates

The calculator is a roofline model with assumed efficiencies (70% of peak bandwidth for decode, 50% of peak for prefill), assumed CPU throughput, and an approximate compute-buffer size. Bandwidths and the RTX 3090/4090 tensor throughputs are vendor figures; the FP16 throughputs used for the RTX 3060, RTX 4060 Ti and M4 Max are approximations, which only affect the prefill estimates. It predicts the shape of the trade-offs and the right order of magnitude. Measure your own machine with llama-bench before relying on a specific number.

How the Three Engines Work

All three engines run the same transformer forward pass. They differ in what they are built around: llama.cpp around running anywhere with whatever memory there is, vLLM and SGLang around keeping a GPU busy with many requests at once.

llama.cpp

llama.cpp is a C/C++ inference engine built on ggml, a tensor library with a backend per hardware family: CUDA (NVIDIA), HIP (AMD), Metal (Apple), Vulkan, SYCL (Intel), and a CPU backend with AVX/AVX2/AVX-512/AMX on x86 and NEON on ARM. Its main properties for local use:

  • One file, memory-mapped. Models ship as GGUF: a header with a magic number and version, typed key-value metadata (architecture, hyperparameters, tokenizer, chat template), a table of tensor descriptors (name, shape, type, offset), then the aligned tensor data. The layout is designed so the file can be mmaped and used in place. Loading is controlled by --load-mode (auto = mmap where the device allows it; mlock pins the pages in RAM).
  • Block-quantized weights from 1.5 to 8 bits. The Q*_0, Q*_K and IQ* types are group-wise formats (Module 14) whose kernels dequantize on the fly inside the matmul. llama-quantize --imatrix uses activation statistics from calibration text to weight the quantization error, which matters most for the low-bit IQ types.
  • Layer offload. -ngl N puts the first N layers and their KV cache on the GPU and runs the rest on the CPU. This is the feature that lets a 32B model run on a 16 GB card at all. -ot (override-tensor) places individual tensors by name pattern; --n-cpu-moe N keeps the expert weights of the first N layers of a mixture-of-experts model in RAM while attention stays on the GPU, which works well because each token only touches a few experts.
  • Op offload during prefill. When a micro-batch is large (32 tokens or more in the CUDA backend), the scheduler copies a CPU-resident layer’s weights to the GPU and runs that matmul there, because a PCIe transfer is cheaper than CPU arithmetic at that batch size. --no-op-offload turns this off.
  • A server with slots. llama-server exposes an OpenAI-compatible HTTP API. It runs up to -np sequences at once (automatic by default, which resolves to
    1. with continuous batching. With an automatic slot count the slots share one unified KV buffer, so a single request can use the whole -c; with --no-kv-unified the context is split evenly across slots. Each slot keeps its last sequence, and a new request goes to the idle slot whose prompt is most similar (--slot-prompt-similarity), so a follow-up turn only prefills the new suffix.

vLLM

vLLM is a Python serving engine on PyTorch with custom CUDA kernels. It started as the reference implementation of PagedAttention (Kwon et al., SOSP 2023) and is designed for throughput under concurrent load.

  • Paged KV cache. The cache is a pool of fixed-size blocks (16 tokens by default) addressed through per-request block tables (Module 9 builds one). A request holds at most one partially filled block, so almost no memory is wasted. The paper measured that earlier systems used only 20.4–38.2% of their KV memory for actual token states, against 96.3% for vLLM.
  • Preallocation by profiling. At startup vLLM asks for gpu_memory_utilization × total GPU memory (default 0.92), runs a dummy forward pass to measure the weights and peak activations, subtracts an estimate for CUDA graphs, and turns everything left into KV blocks. The log line GPU KV cache size: N tokens, Maximum concurrency for M tokens per request: X tells you the result.
  • Continuous batching and chunked prefill. The scheduler admits and retires requests every iteration (Orca’s iteration-level scheduling), caps each step at max_num_batched_tokens, and splits long prompts into chunks so that running decodes are not stalled (Sarathi-Serve). Both are on by default.
  • Automatic prefix caching. Full blocks are hashed by their entire prefix; a new request whose prompt hashes to cached blocks reuses them. On by default.
  • Preemption by recomputation. When blocks run out, the most recently admitted request is evicted, its blocks freed, and it is recomputed later. Current vLLM has no swap-to-CPU path.
  • CUDA graphs. Decode steps for common batch sizes are captured as CUDA graphs so the Python and kernel-launch overhead disappears. --enforce-eager turns graphs (and torch.compile) off, saving memory and startup time at a per-token cost.
  • Quantization from Hugging Face checkpoints. AWQ, GPTQ (with Marlin kernels), FP8, compressed-tensors and others load directly. GGUF and bitsandbytes now require separate plugins (vllm-gguf-plugin, vllm-bnb-plugin), and the GGUF path is documented as experimental.

SGLang

SGLang is also a Python/PyTorch serving engine (Zheng et al., NeurIPS 2024). Its runtime shares vLLM’s goals and adds two ideas.

  • RadixAttention. Instead of hashing whole blocks, SGLang keeps every cached token sequence in a radix tree with one token per page. A new request walks the tree and reuses its longest cached prefix at token granularity. Nodes in use by running requests are reference-counted and pinned; when memory is needed, unreferenced leaves are evicted least-recently-used first. Cached and running tokens share one pool, so there is no separate cache budget.
  • Cache-aware scheduling. The paper’s scheduler orders waiting requests by matched prefix length (longest first), which maximizes reuse. In the current release this is --schedule-policy lpm; the default is fcfs.
  • Overlap scheduling. The CPU-side scheduling of step t+1 runs while the GPU executes step t, hiding scheduler overhead (on by default, --disable-overlap-schedule turns it off).
  • Structured generation. SGLang was designed around programs that make many related calls (few-shot prompts, agents, JSON-constrained output), which is exactly the workload where prefix reuse pays off.

The same workload, three KV managers

The figure runs one chat workload through simplified versions of all three managers (kv_managers.py) with the same 384 token cells each: four users, two turns each, every request starting with the same 48-token system prompt, and each user’s second turn repeating the first turn plus the reply.

What to look for:

  • Slots cap concurrency at -np, and reuse only what is still in the slot a request lands in. That is ideal for one user continuing one conversation and weaker for many users sharing a system prompt.
  • Paged blocks admit as many requests as there are blocks, waste less than one block per request, and reuse prefixes only in whole 16-token blocks.
  • The radix tree reuses prefixes at token granularity and keeps finished conversations until the memory is needed, so it computes the fewest prefill tokens on this workload.
from kv_managers import compare
for name, run in compare(capacity_tokens=384).items():
    s = run["summary"]
    print(f"{name:6s} prefill computed {s['prefill_computed']:4d} / {s['prompt_tokens']} prompt tokens "
          f"({s['prefill_saved_frac']:.0%} saved), peak concurrency {s['peak_concurrency']}, "
          f"preemptions {s['preemptions']}")
slots  prefill computed  376 / 652 prompt tokens (42% saved), peak concurrency 4, preemptions 0
paged  prefill computed  268 / 652 prompt tokens (59% saved), peak concurrency 5, preemptions 0
radix  prefill computed  208 / 652 prompt tokens (68% saved), peak concurrency 5, preemptions 0

Where they differ

llama.cpp vLLM SGLang
Language / runtime C/C++ (ggml) Python + PyTorch + CUDA kernels Python + PyTorch + CUDA kernels
Hardware CPU, NVIDIA, AMD, Apple Metal, Vulkan, Intel, … NVIDIA (compute capability ≥ 7.5), AMD ROCm incl. Radeon RX 7900 / 9000; Apple and CPU experimental; Windows via WSL NVIDIA RTX 30/40/50 and datacenter; AMD Instinct; Apple via MLX (SGLANG_USE_MLX=1)
Model format GGUF Hugging Face safetensors (AWQ, GPTQ, FP8, …) Hugging Face safetensors; GGUF and bitsandbytes in tree
Weights larger than VRAM Layer offload to CPU (-ngl, -ot, --n-cpu-moe); CPU layers compute on the CPU Weight streaming over PCIe each pass (--cpu-offload-gb) Layer offload to RAM (--cpu-offload-gb, --offload-group-size)
KV-cache allocation Contiguous buffer sized by -c; slots share it Paged blocks from a pool sized by profiling Token-granular pool, radix tree
Prefix reuse Per slot (--cache-prompt), plus a RAM prompt cache Automatic prefix caching, block granularity RadixAttention, token granularity
Batching Continuous batching across -np slots Continuous batching, chunked prefill Continuous batching, chunked prefill, overlap scheduler
Best at One or a few users on any machine, models that do not fit in VRAM Many concurrent requests on a GPU that holds the model Many requests that share prefixes (agents, RAG, few-shot)
NoteKey Insight

On a single consumer GPU with one user, all three engines are limited by the same thing: bytes of weights per token over VRAM bandwidth. The differences show up at the edges. When the model does not fit, only llama.cpp’s layer offload keeps decode tolerable. When many requests arrive at once, vLLM’s and SGLang’s schedulers turn the idle arithmetic of the roofline into throughput.

Where Bottlenecks Happen

The same few causes account for most slow or failing local setups.

Symptom Cause Fix
Decode far below bandwidth ÷ model size Some layers run from system RAM (partial offload) More layers on the GPU: smaller quant, shorter -c, quantized KV cache, --n-cpu-moe for MoE models
Fine at first, slower as the chat grows Each token also reads the whole KV cache; at long context it rivals the weights Shorter context, q8_0 KV cache, a model with fewer KV heads
Long wait before the first token Prefill of a long prompt; on partial offload, prefill bounded by PCIe Keep op offload on, raise -ub/-b if VRAM allows, enable prompt/prefix caching so repeated prefixes are not recomputed
Load fails with an allocation error Asked for more VRAM than is free (-ngl too high, -c too long, vLLM gpu_memory_utilization too high next to a desktop) Let llama.cpp’s --fit choose, lower -ngl/-c; in vLLM lower --gpu-memory-utilization or --max-model-len
Everything slow on Windows, GPU memory “full” The NVIDIA driver’s system-memory fallback (driver 536.40+) moved allocations into shared system memory instead of failing Fit the model properly; the fallback can be disabled per application in the NVIDIA Control Panel (driver 546.01+)
Machine stutters, disk busy during generation Weights + KV exceed RAM; the OS pages the model from disk Smaller model or quant; --load-mode mlock only helps if it fits
CPU-only decode slower than expected Too many threads (hyperthreads, efficiency cores) contending for the same memory bus Use physical performance cores (-t; the default already counts them)
Throughput flat as users are added Concurrency capped (-np in llama.cpp, max_num_seqs), or KV pool too small, causing preemptions Raise the cap; give the KV pool more memory; check the vLLM concurrency log line

Decode slows down as context grows

At short context the weights dominate the bytes read per token. At long context the KV cache catches up. For Llama 3.1 8B at Q4_K_M the weights are 4.6 GB per token; the f16 KV cache adds 0.13 GB per 1,000 tokens of context, so at 32k tokens it is almost as large as the weights.

from inference_trace import decode_curve
mc = MACHINES["rtx4090"]
for kv in ["f16", "q8_0"]:
    p = place(MODELS["llama-3.1-8b"], mc, "Q4_K_M", n_ctx=65536, kv_type=kv)
    curve = decode_curve(p, mc, [0, 4096, 16384, 32768, 65536])
    print(f"KV {kv:5s}: " + "  ".join(f"{c['context'] // 1024:>3d}k → {c['tok_s']:5.1f} tok/s" for c in curve))
KV f16  :   0k → 151.6 tok/s    4k → 136.0 tok/s   16k → 103.8 tok/s   32k →  78.9 tok/s   64k →  53.3 tok/s
KV q8_0 :   0k → 151.6 tok/s    4k → 142.9 tok/s   16k → 121.8 tok/s   32k → 101.7 tok/s   64k →  76.6 tok/s

Configuration Reference

The parameters that matter most on consumer hardware, with current defaults (checked against each project’s main branch in October 2026; defaults change, so confirm with --help on your installed version).

llama.cpp (llama-server, llama-cli)

Parameter Default What it controls
-ngl, --n-gpu-layers auto Layers whose weights and KV live in VRAM (all or a number)
--fit / --fit-target / --fit-ctx on / 1024 MiB / 4096 Shrink unset arguments until the model fits, keeping a margin
-c, --ctx-size 0 (model’s trained length) KV-cache size in tokens, shared by all slots
-b, --batch-size / -ub, --ubatch-size 2048 / 512 Logical and physical prompt batch; -ub sizes the compute buffer
-t / -tb physical cores CPU threads for decode / for prompt processing
-fa, --flash-attn auto Fused attention; required for a quantized V cache
-ctk / -ctv f16 KV-cache types: f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
--load-mode auto (mmap) mmap, mlock, mmap+mlock, none, dio
-np, --parallel auto (4) Server slots: concurrent sequences
-kvu, --kv-unified on when -np is auto One KV pool shared by all slots
--cache-prompt / --cache-reuse / --cache-ram on / 0 / 8192 MiB Reuse of previous prompts in slots and in a RAM cache
-ot, --override-tensor — Place tensors matching a regex on a given device
--cpu-moe / --n-cpu-moe N — Keep MoE expert weights (all, or of the first N layers) in RAM
-sm / -ts layer / even Multi-GPU split mode and per-GPU proportions
--op-offload on Run large-batch matmuls of CPU layers on the GPU
-nkvo, --no-kv-offload off Keep the KV cache in RAM even for GPU layers
--spec-draft-model, -md / --spec-draft-n-max — / 3 Speculative decoding with a draft model

vLLM (vllm serve)

Parameter Default What it controls
--gpu-memory-utilization 0.92 Fraction of total GPU memory vLLM may use (weights + activations + KV)
--max-model-len from model config; auto fits to memory Longest sequence; also bounds memory per request
--max-num-seqs / --max-num-batched-tokens 256 / 2048 (server, GPU < 70 GiB) Concurrency and tokens per scheduler step
--kv-cache-dtype auto fp8 (e4m3) / fp8_e5m2 halve the cache
--block-size 16 Tokens per KV block
--enable-prefix-caching on Block-hash prefix reuse
--enforce-eager off Disable CUDA graphs (less memory, slower decode)
--cpu-offload-gb 0 GiB of weights streamed from RAM every forward pass
--quantization, -q from checkpoint awq, gptq, fp8, compressed-tensors, …
--tensor-parallel-size, -tp 1 Split each layer across GPUs
--enable-lora / --max-loras / --max-lora-rank off / 1 / 16 Serving LoRA adapters
--speculative-config — JSON: ngram, draft_model, eagle, eagle3, …

SGLang (python -m sglang.launch_server)

Parameter Default What it controls
--mem-fraction-static auto Fraction of GPU memory for weights + KV pool
--context-length from model config Longest sequence
--max-running-requests auto Concurrency
--chunked-prefill-size auto (2048 on GPUs < 20 GB) Prefill chunk; -1 disables chunking
--schedule-policy fcfs lpm = longest prefix match (the paper’s cache-aware policy)
--disable-radix-cache / --radix-eviction-policy off / lru Radix prefix cache and its eviction policy
--kv-cache-dtype auto fp8_e4m3, fp8_e5m2, …
--cuda-graph-max-bs-decode auto (8 on GPUs < 20 GB) Largest decode batch captured as a CUDA graph
--quantization from checkpoint awq, gptq, fp8, gguf, bitsandbytes, …
--lora-paths / --max-loras-per-batch — / 8 Serving LoRA adapters
--speculative-algorithm — EAGLE, EAGLE3, NGRAM, STANDALONE, …

Tuning Recipes for Consumer Hardware

Each recipe starts from a measurable goal. Measure before and after with llama-bench (llama.cpp) or the benchmark scripts shipped with vLLM and SGLang.

One user, model fits in VRAM (for example an 8B model on a 12–16 GB GPU). Any of the three engines works; llama.cpp is the simplest. Put everything on the GPU, keep flash attention on, and set -c to what you actually use rather than the model’s maximum. Speed is then VRAM bandwidth ÷ model bytes. To go faster, use a smaller quant or speculative decoding (--spec-draft-model with a small model from the same family).

One user, model larger than VRAM (a 32B model on 16 GB, a 70B model on 24 GB). Use llama.cpp. Let --fit choose -ngl, then trade context for layers: a q8_0 KV cache halves its size, and every layer moved to VRAM removes its bytes from the slow RAM path. For MoE models, keep attention and shared weights on the GPU and move experts to RAM with --n-cpu-moe. Expect decode speed to be set by system RAM bandwidth.

Apple silicon. llama.cpp with Metal (on by default) uses unified memory, so there is no offload decision: the limit is how much of the unified memory macOS lets the GPU use. Decode speed follows the chip’s memory bandwidth (for example 273 GB/s on M4 Pro, 546 GB/s on the 40-core M4 Max). vLLM and SGLang run on Apple silicon only through experimental or MLX-based paths.

CPU only. Decode speed is RAM bandwidth ÷ model bytes: about 13 tok/s upper bound for an 8B Q4_K_M on dual-channel DDR5 at 70% efficiency. Prompt processing is compute-bound and much slower than on a GPU. Use physical cores only, and prefer the smallest quant that keeps quality acceptable.

Several users or an agent making parallel calls, model fits in VRAM. Use vLLM or SGLang. Set the context length to what the workload needs (--max-model-len / --context-length) so more requests fit in the KV pool, use an fp8 KV cache if quality allows, and leave headroom for the desktop by lowering --gpu-memory-utilization / --mem-fraction-static on a GPU that also drives displays. If requests share long prefixes (a system prompt, tool definitions, retrieved documents), SGLang with --schedule-policy lpm reuses them at token granularity.

TipTry This

In the calculator, take Qwen2.5 14B on the RTX 3060 at 32k context: 1. Note the auto -ngl and decode speed with an f16 KV cache. 2. Switch the KV cache to q8_0. How many more layers fit, and what does that do to tokens/s? 3. Drop the context to 8k instead. Which change bought more speed, and what did each cost you?

Running a Fine-Tuned Model

A fine-tuned model (Module 25) reaches an engine in one of two forms.

  • Merged weights. The LoRA update \Delta W = BA is added into the base weights and saved as a normal checkpoint. It runs at exactly the base model’s speed. For llama.cpp, convert the merged Hugging Face checkpoint to GGUF with convert_hf_to_gguf.py and quantize it with llama-quantize like any other model.
  • A runtime adapter. The base model is loaded once and the small A, B matrices are applied at inference time. llama.cpp loads GGUF adapters with --lora / --lora-scaled; vLLM serves adapters with --enable-lora (--max-loras, --max-lora-rank, default rank limit 16); SGLang with --lora-paths. Adapters let one GPU serve several fine-tunes of the same base, at the cost of extra matmuls per layer.

Merge in full precision, then quantize. Adding BA to weights that are already quantized would round the update away along with the rest of the weight.

Common Pitfalls

  1. Leaving the context at the model’s maximum. -c 0 (the default) and vLLM’s default --max-model-len use the model’s trained length. For Llama 3.1 8B that is a 16 GiB f16 cache in llama.cpp. llama.cpp’s --fit shrinks it only if you did not set it yourself.
  2. Forcing -ngl 99 on a model that does not fit. Setting -ngl explicitly turns off the automatic fitting for that setting. Expect an allocation error, or on Windows a silent spill into shared memory.
  3. Comparing prompt processing with generation. Benchmarks report both (pp and tg in llama-bench). They differ by one to two orders of magnitude and respond to different settings.
  4. Quantizing the V cache with flash attention off. llama.cpp refuses: a quantized V cache requires flash attention.
  5. Expecting a serving engine to speed up one user. vLLM’s and SGLang’s advantage is concurrency. For one stream on a model that fits, the bandwidth limit is the same for every engine.
  6. Using GGUF files with vLLM by default. vLLM’s GGUF support is now an experimental plugin; use an AWQ, GPTQ or FP8 checkpoint instead.
  7. Starting vLLM on the GPU that drives your displays at the default memory fraction. It requests 92% of total memory at startup. Lower --gpu-memory-utilization to leave room for the desktop.
  8. Too many CPU threads. Decode on the CPU is memory-bound; threads beyond the physical performance cores add contention, not bandwidth.

Exercises

Exercise 1: Size a deployment

Compute, without the calculator, the file size, the KV cache at 16k context, and the decode upper bound of Qwen2.5 32B at Q5_K_M on an RTX 5090 (32 GB, 1,792 GB/s). Does it fit with an f16 cache? With q8_0?

from perf_model import Device, Machine, DDR5_DUAL

rtx5090 = Machine("RTX 5090 32 GB + DDR5", Device("RTX 5090", 32, 1792, 209.5), DDR5_DUAL, 63.0)
qwen32 = MODELS["qwen2.5-32b"]
# Your implementation here: use weight_bytes, kv_cache_bytes and place(...)

Exercise 2: The offload break-even

For Qwen2.5 14B Q4_K_M on the RTX 3060, plot decode tok/s against -ngl from 0 to 49 using place(..., n_gpu_layers=n) and decode_step_time. Explain the shape: why does the curve rise slowly at first and steeply near full offload?

# Your implementation here

Exercise 3: Make the radix cache lose

Write a workload (a list of Request) on which RadixManager computes more prefill tokens than PagedManager with prefix caching. Hint: what happens when memory is tight and the scheduler’s order differs from arrival order?

from kv_managers import Request, RadixManager, PagedManager, simulate
# Your implementation here

Exercise 4: Batch size and the ridge

Extend decode_step_time with a batch argument: the weights are read once, the KV cache once per sequence, and compute grows with the batch. At what batch size does an RTX 4060 Ti stop gaining throughput for Llama 3.1 8B Q4_K_M? Compare with ridge_batch.

Summary

Key takeaways:

  1. Decode is memory-bound — every generated token reads every weight once, so tok/s ≈ effective bandwidth ÷ bytes read, and quantization helps because it reads fewer bytes.
  2. Prefill is compute-bound — a micro-batch shares one read of the weights, which puts prompt processing on the flat part of the roofline.
  3. Placement decides speed — layers in VRAM run at VRAM bandwidth, layers in RAM at RAM bandwidth, and the slow part dominates; the KV cache competes with weights for VRAM and grows with context.
  4. The engines differ in memory management — llama.cpp offloads layers and runs anywhere; vLLM pages the KV cache and batches continuously; SGLang adds a token-granular radix cache for shared prefixes.
  5. Most slow setups have one of a few causes — partial offload, an oversized context, oversubscribed VRAM, or concurrency limits — and each maps to a specific parameter.

Going Deeper

Core Papers:

Practical Resources:

What’s Next

This is the last module. Module 29: Agentic Infrastructure builds the platform that calls an engine like these, and the playground runs a small GPT entirely in your browser, where the same memory-bound decode loop is visible token by token.