Module 25: Parameter-Efficient Fine-Tuning

Introduction

Every module so far taught you to build a model and then train all of its weights — pretraining (m07) and alignment (m12) both step every parameter. That is fine when you own the compute to update a 175-billion-parameter model. Almost nobody does. In practice you take a frozen pretrained model and adapt it with a tiny number of new parameters — a technique so dominant that “fine-tuning” today usually means this. Parameter-efficient fine-tuning (PEFT) learns a small add-on that steers a frozen model, and its flagship is LoRA — Low-Rank Adaptation.

LoRA freezes the pretrained weight \(W_0\) and learns a low-rank update \(\Delta W = BA\) beside it, so instead of retraining a \(d \times k\) matrix you train two thin ones with rank \(r \ll \min(d, k)\).

Why it matters for LLMs:

  • You can afford it. LoRA reported 10,000× fewer trainable parameters and 3× less GPU memory to fine-tune GPT-3 175B — the difference between a data center and a single GPU.
  • It costs nothing at inference. The adapter merges back into the weight (\(W = W_0 + \frac{\alpha}{r} BA\)), so a deployed LoRA model is an ordinary linear layer — no extra latency, unlike bolt-on adapter modules.
  • It’s swappable. One frozen base + many small adapters = one model that wears many hats (a code adapter, a chat adapter, a domain adapter), each a few MB.
  • It bridges to quantization. Freeze the base in 4-bit and train the adapter in full precision and you have QLoRA — fine-tuning a 65B model on one GPU (ties m14).

What You’ll Learn

After this module, you can:

  • Explain why a fine-tuning update lives in a low-rank subspace, and why that makes it cheap
  • Write the LoRA forward pass \(h = W_0 x + \frac{\alpha}{r} BA\,x\) from the shapes up
  • Build LoRALinear from scratch, wrapping a frozen nn.Linear
  • Prove the two facts that make LoRA trustworthy — the zero-init identity and merge equivalence — on a real GPTModel
  • Compute the parameter budget \(r(d+k)\) vs \(dk\) and read the savings
  • Fine-tune a model with only the adapter, leaving the base bit-for-bit frozen

Prerequisites

This module requires familiarity with:

Intuition: The Update Is Low-Rank

Start with the question full fine-tuning ignores: how much does a weight matrix actually need to change to specialize a pretrained model? A \(4096 \times 4096\) attention projection has 16.8M numbers. Fine-tuning nudges all of them — but the net change \(\Delta W\) it learns turns out to have very low intrinsic rank. The directions that matter live in a handful of dimensions, not thousands.

LoRA takes that observation and makes it structural. Rather than let \(\Delta W\) be any \(d \times k\) matrix, it forces it to be low rank by writing it as a product of two thin factors:

        full fine-tune                     LoRA
        ──────────────                     ────
   ΔW  =  ┌──────────┐            ΔW  =  ┌─┐   ┌──────────┐
          │          │                  │ │ × │    A     │   ← r × k  (down)
          │   d×k    │                  │B│   └──────────┘
          │          │                  │ │   r
          │          │                  │d│
          └──────────┘                  └─┘   ← d × r  (up)
      d·k trainable numbers        r·(d + k) trainable numbers

Every signal the adapter adds has to squeeze through the \(r\)-dimensional bottleneck in the middle. A projects the \(d\)-wide input down to \(r\) dimensions; B projects that back up to \(k\). With \(r = 8\) and \(d = k = 4096\), that is \(8 \times (4096 + 4096) = 65{,}536\) numbers instead of \(16{,}777{,}216\) — 256× fewer, and the same expressive ceiling as any rank-8 update.

Step through the data flow. Watch the wide input collapse into the thin bottleneck and re-expand, running alongside the frozen \(W_0 x\) path — never touching it.

NoteKey Insight

LoRA doesn’t add a smaller dense update — it adds a structurally low-rank one. That is what buys the parameter savings and what limits the adapter: it can only write changes of rank \(\le r\). The bet is that fine-tuning updates were low-rank to begin with.

The Math: \(h = W_0 x + \frac{\alpha}{r} BA\,x\)

For a linear layer with frozen weight \(W_0 \in \mathbb{R}^{k \times d}\) (PyTorch stores it as (out, in)), LoRA parameterizes the update as

\[ h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B A\, x, \qquad B \in \mathbb{R}^{k \times r},\; A \in \mathbb{R}^{r \times d},\; r \ll \min(d, k). \]

Three details carry all the weight:

  1. Initialization. \(A\) gets a random Gaussian init, \(B\) is initialized to zero — so \(\Delta W = BA = 0\) at the start of training. Fine-tuning begins as an exact copy of the pretrained model. There is no random perturbation to recover from; the adapter grows from nothing.

  2. The scaling \(\alpha/r\). The update is scaled by a constant \(\alpha/r\). Following the paper, \(\alpha\) behaves like a learning rate and is not tuned separately — set it to the first \(r\) you try (with \(\alpha = r\) the scale is \(1\)). Dividing by \(r\) keeps the update magnitude roughly stable as you change the rank.

  3. What trains. Only \(A\) and \(B\) receive gradients. \(W_0\) is frozen, so its optimizer state (the momentum and variance buffers that dominate Adam’s memory) never has to exist — this, not the parameter count alone, is where the 3× memory win comes from.

Code: LoRALinear From Scratch

The whole method is a few lines. We wrap an existing nn.Linear, freeze it, and add the two matrices. This follows the implementation in lora.py.

import sys
sys.path.insert(0, '..')  # make sibling modules (m06_transformer, …) importable

import torch
import torch.nn as nn
from lora import LoRALinear, lora_update

torch.manual_seed(0)
base = nn.Linear(16, 8, bias=False)      # a pretrained layer: d=16 → k=8
lora = LoRALinear(base, rank=2, alpha=2.0)

print("A shape (r × d):", tuple(lora.A.shape))
print("B shape (k × r):", tuple(lora.B.shape))
print("B is all zeros at init:", bool(torch.count_nonzero(lora.B) == 0))
A shape (r × d): (2, 16)
B shape (k × r): (8, 2)
B is all zeros at init: True

The forward pass runs the input through the bottleneck rather than forming the full \(\Delta W\): base(x) + scaling * (x @ A.T) @ B.T. Algebraically identical, but it only ever multiplies by the thin matrices.

Anchor 1 — the zero-init identity

Because \(B = 0\), the adapter contributes exactly nothing at initialization. The LoRA layer is bit-for-bit the frozen layer:

x = torch.randn(4, 16)
print("LoRA(x) == base(x) exactly:", bool(torch.equal(lora(x), base(x))))
LoRA(x) == base(x) exactly: True

This is the property that makes LoRA safe to attach: you never degrade the model by adding an adapter. Let’s confirm it on a real transformer — wrap a GPTModel’s output projection and check the logits are unchanged:

from m06_transformer.transformer import GPTModel
from lora import inject_lora, trainable_parameters, frozen_parameters

model = GPTModel(vocab_size=100, embed_dim=32, num_heads=2, num_layers=2,
                 max_seq_len=64, dropout=0.0)
model.eval()
tokens = torch.randint(0, 100, (2, 16))
with torch.no_grad():
    before = model(tokens)

inject_lora(model, ["lm_head"], rank=4, alpha=8.0)   # freeze all, adapt lm_head
with torch.no_grad():
    after = model(tokens)

print("logits identical after attaching LoRA:", bool(torch.equal(before, after)))
print("trainable params:", trainable_parameters(model))
print("frozen params:   ", frozen_parameters(model))
logits identical after attaching LoRA: True
trainable params: 528
frozen params:    30720

Only the two adapter matrices train; the entire transformer is frozen.

Anchor 2 — merging away the adapter (no inference latency)

Since \(h = (W_0 + \frac{\alpha}{r} BA)\,x\), we can fold the update into a single weight once training is done. The merged layer is a plain nn.Linear that reproduces the adapter’s output — so LoRA adds zero inference cost:

# give the adapter a real (non-zero) update to merge
with torch.no_grad():
    lora.B.copy_(torch.randn_like(lora.B))

merged = lora.merge()                       # a standalone nn.Linear
diff = (lora(x) - merged(x)).detach().abs().max()
print("max |LoRA(x) − merged(x)|:", f"{diff.item():.2e}")
print("merged is a plain nn.Linear:", isinstance(merged, nn.Linear))
max |LoRA(x) − merged(x)|: 1.79e-07
merged is a plain nn.Linear: True

The difference is floating-point noise. At deployment you merge, ship one matrix, and the adapter has vanished.

The Parameter Budget

The whole point is the count. A full fine-tune of one \((k, d)\) layer trains \(d \cdot k\) numbers; LoRA trains \(r \cdot (d + k)\). Drive the two dials — the layer width \(d\) and the rank \(r\) — and watch the gap open on a log scale. The reduction factor is \(\dfrac{dk}{r(d+k)}\), which for a square layer is \(\dfrac{d}{2r}\).

from lora import (count_full_parameters, count_lora_parameters,
                  parameter_reduction)

# A GPT-3-scale attention projection, adapted at rank 8.
anchor = {
    "d": 4096, "r": 8,
    "full": count_full_parameters(4096, 4096),
    "lora": count_lora_parameters(4096, 4096, 8),
    "reduction": parameter_reduction(4096, 4096, 8),
}
ojs_define(loraAnchor = anchor)
TipTry This
  1. Push \(r\) to 64. The savings shrink — a fat adapter approaches a full fine-tune. LoRA’s win is the small rank.
  2. Halve \(d\) from 4096 to 2048 at \(r = 8\). The reduction halves too: for a square layer it is exactly \(d/2r\).
  3. Set \(r = 1\). The extreme rank-one adapter — one down-vector, one up-vector — still adapts the layer, at \(2d\) parameters.

Fine-Tuning With Only the Adapter

Put it together: freeze a model, attach an adapter, and train only \(A\) and \(B\). The demo in lora.py fits an adapter so the frozen layer plus its update matches a target that is genuinely rank-\(r\) — exactly what a rank-\(r\) adapter can represent. Watch three things at once: the loss falls, the update norm \(\lVert \Delta W \rVert\) rises from exactly zero, and the frozen base weight never moves.

from lora import demonstrate_lora

demo = demonstrate_lora()          # frozen base, SGD on A and B only
curve = [{"step": s.step, "loss": s.loss,
          "delta": s.delta_norm, "base": s.base_norm} for s in demo["steps"]]
ojs_define(loraCurve = curve)
ojs_define(loraMeta = {"trainable": demo["trainable"], "frozen": demo["frozen"],
                       "base_unchanged": demo["base_unchanged"]})
print("trainable (A + B):", demo["trainable"])
print("frozen (W₀):      ", demo["frozen"])
print("first-step ‖ΔW‖:  ", f"{demo['steps'][0].delta_norm:.4f}")
print("last-step  ‖ΔW‖:  ", f"{demo['steps'][-1].delta_norm:.2f}")
print("base weight bit-for-bit unchanged:", demo["base_unchanged"])
trainable (A + B): 64
frozen (W₀):       256
first-step ‖ΔW‖:   0.0044
last-step  ‖ΔW‖:   23.20
base weight bit-for-bit unchanged: True
NoteKey Insight

The base is not “mostly” frozen — it is exactly frozen. demonstrate_lora checks the base weight is bit-for-bit identical before and after training. All the learning lives in \(A\) and \(B\); the pretrained knowledge is untouched and shared.

Beyond LoRA

LoRA is the foundation of a whole family. The book builds the core here; these are the load-bearing extensions (roadmap follow-ups):

  • QLoRA (Dettmers et al., 2023) — freeze the base in 4-bit (NF4 quantization, m14) and keep the LoRA adapter in full precision. Because the frozen base carries no optimizer state, quantizing it is nearly free — this is what fits a 65B fine-tune on a single 48 GB GPU.
  • DoRA (Liu et al., 2024) — decompose the weight into magnitude and direction, and apply LoRA only to the direction; closes much of the gap to full fine-tuning at the same parameter budget.
  • Adapters (Houlsby et al., 2019) — the predecessor: small bottleneck MLPs inserted between layers. They work, but unlike LoRA they cannot be merged, so they add permanent inference latency.
  • Prefix / prompt tuning (Li & Liang; Lester et al., 2021) — freeze the whole model and learn a handful of virtual “prefix” key/value vectors instead of touching weights at all.

Common Pitfalls

Pitfall Why it bites Fix
Initializing both \(A\) and \(B\) randomly \(\Delta W \ne 0\) at step 0 — you start by corrupting the pretrained model Keep \(B = 0\) (Gaussian \(A\)) so training begins as the identity
Treating rank \(r\) as “bigger is better” A large \(r\) erases the savings and can overfit the small fine-tune set Start small (\(r = 8\)–\(16\)); raise only if the task underfits
Forgetting to freeze the base You silently full-fine-tune — the memory win evaporates requires_grad_(False) on the base (what inject_lora does)
Merging, then continuing to train Once merged, there is no separate adapter to update Merge only for deployment; keep the adapter while training
Adapting the wrong layers The update has to land where the task needs it The paper adapts attention \(W_q, W_v\); that is the usual first choice

Exercises

Exercise 1: The rank bound

Show that no matter what \(A\) and \(B\) contain, the update \(\Delta W\) has rank at most \(r\).

from lora import LoRALinear

base = nn.Linear(32, 24, bias=False)
lora = LoRALinear(base, rank=3, alpha=3.0)
with torch.no_grad():
    lora.A.copy_(torch.randn_like(lora.A))
    lora.B.copy_(torch.randn_like(lora.B))

rank = torch.linalg.matrix_rank(lora.delta_weight())
print("rank(ΔW):", int(rank), "≤ r =", lora.rank)
rank(ΔW): 3 ≤ r = 3

Exercise 2: Only the adapter gets gradients

Run one backward pass and confirm the base weight has no gradient while \(A\) and \(B\) do.

base = nn.Linear(8, 8, bias=False)
lora = LoRALinear(base, rank=2)
with torch.no_grad():
    lora.B.copy_(torch.randn_like(lora.B))     # leave the zero-init so B gets a grad
lora(torch.randn(3, 8)).pow(2).mean().backward()

print("base.weight.grad is None:", base.weight.grad is None)
print("A.grad set:", lora.A.grad is not None, "| B.grad set:", lora.B.grad is not None)
base.weight.grad is None: True
A.grad set: True | B.grad set: True

Exercise 3: Swap two adapters on one base

One frozen model, two adapters. Attach B₁, then a different B₂, and confirm the merged weights differ — the “many hats, one base” story.

# Your implementation here:
# 1. build a base nn.Linear and two LoRALinear wrappers around copies
# 2. set different B matrices
# 3. compare merged_weight() — they should differ, base unchanged

Summary

Key takeaways:

  1. Fine-tuning updates are low-rank. LoRA exploits this by writing \(\Delta W = BA\) with rank \(r \ll \min(d, k)\) — the same expressive ceiling as any rank-\(r\) update, at a fraction of the parameters.
  2. The forward pass is \(h = W_0 x + \frac{\alpha}{r} BA\,x\). Signal squeezes through the \(r\)-dim bottleneck; \(A\) down-projects, \(B\) up-projects, \(\alpha/r\) scales.
  3. Zero-init identity. \(B = 0\) at start, so \(\Delta W = 0\) and the adapted layer is exactly the pretrained one — verified bit-for-bit on a real GPTModel.
  4. No inference latency. \(W = W_0 + \frac{\alpha}{r} BA\) merges the adapter into one matrix — unlike inserted adapter modules.
  5. The budget: \(r(d+k)\) vs \(dk\). For a square layer that is a \(d/2r\) reduction — 256× at \(d = 4096\), \(r = 8\); 10,000× fewer trainable params for GPT-3 175B.
  6. The base stays frozen. Only \(A\) and \(B\) train, so the base carries no optimizer state (the real memory win) and is shared across many swappable adapters.

What’s Next

You now have the technique that makes every other fine-tuning method in this book affordable — plug it into the SFT and DPO objectives of Module 12: Alignment, or freeze the base in 4-bit with Module 14: Quantization to reconstruct QLoRA from the two halves.

Going Deeper

Core Papers:

Practical Resources: