Module 25: Parameter-Efficient Fine-Tuning

Introduction

Every module so far taught you to build a model and then train all of its weights — pretraining (m07) and alignment (m12) both step every parameter. That is fine when you own the compute to update a 175-billion-parameter model. Almost nobody does. In practice you take a frozen pretrained model and adapt it with a tiny number of new parameters — a technique so dominant that “fine-tuning” today usually means this. Parameter-efficient fine-tuning (PEFT) learns a small add-on that steers a frozen model, and its flagship is LoRA — Low-Rank Adaptation.

LoRA freezes the pretrained weight W_0 and learns a low-rank update \Delta W = BA beside it, so instead of retraining a d \times k matrix you train two thin ones with rank r \ll \min(d, k).

Why it matters for LLMs:

  • You can afford it. LoRA reported 10,000× fewer trainable parameters and 3× less GPU memory to fine-tune GPT-3 175B — the difference between a data center and a single GPU.
  • It costs nothing at inference. The adapter merges back into the weight (W = W_0 + \frac{\alpha}{r} BA), so a deployed LoRA model is an ordinary linear layer — no extra latency, unlike bolt-on adapter modules.
  • It’s swappable. One frozen base + many small adapters = one model that wears many hats (a code adapter, a chat adapter, a domain adapter), each a few MB.
  • It bridges to quantization. Freeze the base in 4-bit and train the adapter in full precision and you have QLoRA — fine-tuning a 65B model on one GPU (ties m14).

What You’ll Learn

After this module, you can:

  • Explain why a fine-tuning update lives in a low-rank subspace, and why that makes it cheap
  • Write the LoRA forward pass h = W_0 x + \frac{\alpha}{r} BA\,x from the shapes up
  • Build LoRALinear from scratch, wrapping a frozen nn.Linear
  • Prove the two facts that make LoRA trustworthy — the zero-init identity and merge equivalence — on a real GPTModel
  • Compute the parameter budget r(d+k) vs dk and read the savings
  • Fine-tune a model with only the adapter, leaving the base bit-for-bit frozen
  • Decompose a weight into magnitude and direction and build DoRA, which adapts the two separately to recover full fine-tuning’s update pattern

Prerequisites

This module requires familiarity with:

Intuition: The Update Is Low-Rank

Start with the question full fine-tuning ignores: how much does a weight matrix actually need to change to specialize a pretrained model? A 4096 \times 4096 attention projection has 16.8M numbers. Fine-tuning nudges all of them — but the net change \Delta W it learns turns out to have very low intrinsic rank. The directions that matter live in a handful of dimensions, not thousands.

LoRA takes that observation and makes it structural. Rather than let \Delta W be any d \times k matrix, it forces it to be low rank by writing it as a product of two thin factors:

        full fine-tune                     LoRA
        ──────────────                     ────
   ΔW  =  ┌──────────┐            ΔW  =  ┌─┐   ┌──────────┐
          │          │                  │ │ × │    A     │   ← r × k  (down)
          │   d×k    │                  │B│   └──────────┘
          │          │                  │ │   r
          │          │                  │d│
          └──────────┘                  └─┘   ← d × r  (up)
      d·k trainable numbers        r·(d + k) trainable numbers

Every signal the adapter adds has to squeeze through the r-dimensional bottleneck in the middle. A projects the d-wide input down to r dimensions; B projects that back up to k. With r = 8 and d = k = 4096, that is 8 \times (4096 + 4096) = 65{,}536 numbers instead of 16{,}777{,}216 — 256× fewer, and the same expressive ceiling as any rank-8 update.

Step through the data flow. Watch the wide input collapse into the thin bottleneck and re-expand, running alongside the frozen W_0 x path — never touching it.

NoteKey Insight

LoRA doesn’t add a smaller dense update — it adds a structurally low-rank one. That is what buys the parameter savings and what limits the adapter: it can only write changes of rank \le r. The bet is that fine-tuning updates were low-rank to begin with.

The Math: h = W_0 x + \frac{\alpha}{r} BA\,x

For a linear layer with frozen weight W_0 \in \mathbb{R}^{k \times d} (PyTorch stores it as (out, in)), LoRA parameterizes the update as

h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B A\, x, \qquad B \in \mathbb{R}^{k \times r},\; A \in \mathbb{R}^{r \times d},\; r \ll \min(d, k).

Three details carry all the weight:

  1. Initialization. A gets a random Gaussian init, B is initialized to zero — so \Delta W = BA = 0 at the start of training. Fine-tuning begins as an exact copy of the pretrained model. There is no random perturbation to recover from; the adapter grows from nothing.

  2. The scaling \alpha/r. The update is scaled by a constant \alpha/r. Following the paper, \alpha behaves like a learning rate and is not tuned separately — set it to the first r you try (with \alpha = r the scale is 1). Dividing by r keeps the update magnitude roughly stable as you change the rank.

  3. What trains. Only A and B receive gradients. W_0 is frozen, so its optimizer state (the momentum and variance buffers that dominate Adam’s memory) never has to exist — this, not the parameter count alone, is where the 3× memory win comes from.

Code: LoRALinear From Scratch

The whole method is a few lines. We wrap an existing nn.Linear, freeze it, and add the two matrices. This follows the implementation in lora.py.

import sys
sys.path.insert(0, '..')  # make sibling modules (m06_transformer, …) importable

import torch
import torch.nn as nn
from lora import LoRALinear, lora_update

torch.manual_seed(0)
base = nn.Linear(16, 8, bias=False)      # a pretrained layer: d=16 → k=8
lora = LoRALinear(base, rank=2, alpha=2.0)

print("A shape (r × d):", tuple(lora.A.shape))
print("B shape (k × r):", tuple(lora.B.shape))
print("B is all zeros at init:", bool(torch.count_nonzero(lora.B) == 0))
A shape (r × d): (2, 16)
B shape (k × r): (8, 2)
B is all zeros at init: True

The forward pass runs the input through the bottleneck rather than forming the full \Delta W: base(x) + scaling * (x @ A.T) @ B.T. Algebraically identical, but it only ever multiplies by the thin matrices.

Anchor 1 — the zero-init identity

Because B = 0, the adapter contributes exactly nothing at initialization. The LoRA layer is bit-for-bit the frozen layer:

x = torch.randn(4, 16)
print("LoRA(x) == base(x) exactly:", bool(torch.equal(lora(x), base(x))))
LoRA(x) == base(x) exactly: True

This is the property that makes LoRA safe to attach: you never degrade the model by adding an adapter. Let’s confirm it on a real transformer — wrap a GPTModel’s output projection and check the logits are unchanged:

from m06_transformer.transformer import GPTModel
from lora import inject_lora, trainable_parameters, frozen_parameters

model = GPTModel(vocab_size=100, embed_dim=32, num_heads=2, num_layers=2,
                 max_seq_len=64, dropout=0.0)
model.eval()
tokens = torch.randint(0, 100, (2, 16))
with torch.no_grad():
    before = model(tokens)

inject_lora(model, ["lm_head"], rank=4, alpha=8.0)   # freeze all, adapt lm_head
with torch.no_grad():
    after = model(tokens)

print("logits identical after attaching LoRA:", bool(torch.equal(before, after)))
print("trainable params:", trainable_parameters(model))
print("frozen params:   ", frozen_parameters(model))
logits identical after attaching LoRA: True
trainable params: 528
frozen params:    30720

Only the two adapter matrices train; the entire transformer is frozen.

Anchor 2 — merging away the adapter (no inference latency)

Since h = (W_0 + \frac{\alpha}{r} BA)\,x, we can fold the update into a single weight once training is done. The merged layer is a plain nn.Linear that reproduces the adapter’s output — so LoRA adds zero inference cost:

# give the adapter a real (non-zero) update to merge
with torch.no_grad():
    lora.B.copy_(torch.randn_like(lora.B))

merged = lora.merge()                       # a standalone nn.Linear
diff = (lora(x) - merged(x)).detach().abs().max()
print("max |LoRA(x) − merged(x)|:", f"{diff.item():.2e}")
print("merged is a plain nn.Linear:", isinstance(merged, nn.Linear))
max |LoRA(x) − merged(x)|: 1.34e-07
merged is a plain nn.Linear: True

The difference is floating-point noise. At deployment you merge, ship one matrix, and the adapter has vanished.

The Parameter Budget

The whole point is the count. A full fine-tune of one (k, d) layer trains d \cdot k numbers; LoRA trains r \cdot (d + k). Drive the two dials — the layer width d and the rank r — and watch the gap open on a log scale. The reduction factor is \dfrac{dk}{r(d+k)}, which for a square layer is \dfrac{d}{2r}.

from lora import (count_full_parameters, count_lora_parameters,
                  parameter_reduction)

# A GPT-3-scale attention projection, adapted at rank 8.
anchor = {
    "d": 4096, "r": 8,
    "full": count_full_parameters(4096, 4096),
    "lora": count_lora_parameters(4096, 4096, 8),
    "reduction": parameter_reduction(4096, 4096, 8),
}
ojs_define(loraAnchor = anchor)
TipTry This
  1. Push r to 64. The savings shrink — a fat adapter approaches a full fine-tune. LoRA’s win is the small rank.
  2. Halve d from 4096 to 2048 at r = 8. The reduction halves too: for a square layer it is exactly d/2r.
  3. Set r = 1. The extreme rank-one adapter — one down-vector, one up-vector — still adapts the layer, at 2d parameters.

Fine-Tuning With Only the Adapter

Put it together: freeze a model, attach an adapter, and train only A and B. The demo in lora.py fits an adapter so the frozen layer plus its update matches a target that is genuinely rank-r — exactly what a rank-r adapter can represent. Watch three things at once: the loss falls, the update norm \lVert \Delta W \rVert rises from exactly zero, and the frozen base weight never moves.

from lora import demonstrate_lora

demo = demonstrate_lora()          # frozen base, SGD on A and B only
curve = [{"step": s.step, "loss": s.loss,
          "delta": s.delta_norm, "base": s.base_norm} for s in demo["steps"]]
ojs_define(loraCurve = curve)
ojs_define(loraMeta = {"trainable": demo["trainable"], "frozen": demo["frozen"],
                       "base_unchanged": demo["base_unchanged"]})
print("trainable (A + B):", demo["trainable"])
print("frozen (W₀):      ", demo["frozen"])
print("first-step ‖ΔW‖:  ", f"{demo['steps'][0].delta_norm:.4f}")
print("last-step  ‖ΔW‖:  ", f"{demo['steps'][-1].delta_norm:.2f}")
print("base weight bit-for-bit unchanged:", demo["base_unchanged"])
trainable (A + B): 64
frozen (W₀):       256
first-step ‖ΔW‖:   0.0044
last-step  ‖ΔW‖:   23.20
base weight bit-for-bit unchanged: True
NoteKey Insight

The base is not “mostly” frozen — it is exactly frozen. demonstrate_lora checks the base weight is bit-for-bit identical before and after training. All the learning lives in A and B; the pretrained knowledge is untouched and shared.

QLoRA: Fine-Tuning a 4-Bit Base

Here is the observation that turns LoRA into the way large models are actually fine-tuned. The base is frozen, so it carries no optimizer state. In a full fine-tune, every weight needs its 16-bit value plus an Adam momentum and variance — roughly 3\times the weight memory in optimizer state alone. LoRA already deleted that for the base: only the tiny adapter has an optimizer. But the base still sits in memory at 16 bits, and for a 65B model that is 130 GB before you have loaded a single gradient.

QLoRA (Dettmers et al., 2023) asks: if the base is read-only, why store it at full precision? Freeze it in 4-bit and keep the adapter in 16-bit. Quantizing a frozen weight is nearly free — there is no optimizer state to quantize, and the tiny error the 4-bit rounding introduces is exactly the kind of small offset the LoRA adapter learns to absorb. This is what fits a 65B fine-tune — a 780 GB full fine-tune — onto a single 48 GB GPU with no measurable quality loss.

We build it from the two halves you already have: LoRA (this module) and blockwise quantization (m14). The one genuinely new piece is the 4-bit datatype QLoRA invented for the job.

The Math: NormalFloat (NF4)

Naïve 4-bit quantization spaces its 16 levels evenly across [-\text{absmax}, \text{absmax}]. But pretrained weights are not spread evenly — they pile up near zero in a bell curve. Evenly-spaced levels waste most of their codes out in the tails where almost no weight lives, and quantize the crowded center too coarsely.

NormalFloat fixes the mismatch by placing the levels where the weights are. For zero-mean, normally-distributed data the optimal fixed codebook is the set of quantiles of a standard normal: split the probability mass into 2^k equal slices and put a code at the center of each. Every code is then used equally often — no code is wasted — and the levels bunch up near zero exactly where the weights are dense. For k = 4 bits that is 16 code values

q_i \;=\; \Phi^{-1}\!\left(p_i\right), \qquad p_i \text{ evenly spaced in probability}, \qquad \hat q_i = \frac{q_i}{\max_j |q_j|},

where \Phi^{-1} is the normal quantile function (the inverse CDF). QLoRA makes the split slightly asymmetric — 8 negative quantiles, then 0, then 7 positive — so that 0.0 is exactly representable (padding and genuinely-zero weights must round-trip). A weight is stored, as in m14, by blockwise absmax scaling: scale each block of 64 weights into [-1, 1] by its own |\max|, snap to the nearest code, and dequantize with \hat w = \hat q_{\text{idx}} \cdot \text{absmax}_{\text{block}}.

The quantile function has no elementary form, but it is one call away from the error function: \Phi^{-1}(p) = \sqrt{2}\,\operatorname{erf}^{-1}(2p - 1). So we can build NF4 from scratch with torch.erfinv — no scipy — and it reproduces the canonical values bit-for-bit.

Drive the explorer: the 16 NF4 levels sit under the normal curve, dense in the middle. Toggle the uniform code to see it ignore the distribution — equal width, not equal mass.

from qlora import nf4_level_data, nf4_code_values

ojs_define(nf4Levels = nf4_level_data())
print("NF4 code values (16 levels, normalized to [-1, 1]):")
print([round(v, 4) for v in nf4_code_values().tolist()])
print("zero exactly representable:", 0.0 in nf4_code_values().tolist())

Now let’s build the code and prove it reproduces the standard NF4 datatype:

import torch
from qlora import nf4_code_values, quantize_nf4, dequantize_nf4

# The canonical bitsandbytes NF4 code values.
canonical = torch.tensor([
    -1.0, -0.6961928, -0.5250731, -0.3949175, -0.2844414, -0.1847734,
    -0.0910500, 0.0, 0.0795803, 0.1609302, 0.2461123, 0.3379152,
    0.4407098, 0.5626170, 0.7229568, 1.0])
print("matches canonical NF4:", torch.allclose(nf4_code_values(), canonical, atol=1e-4))

# Quantize a weight to 4-bit and back.
torch.manual_seed(0)
W = torch.randn(64, 64)
q = quantize_nf4(W)                     # NF4Tensor: uint8 indices + fp32 block scales
recon = dequantize_nf4(q)
print("stored as:", q.indices.dtype, "indices +", q.absmax.numel(), "block scales")
print("mean abs reconstruction error:", round(float((recon - W).abs().mean()), 4))
matches canonical NF4: True
stored as: torch.uint8 indices + 64 block scales
mean abs reconstruction error: 0.0726

Why NF4, Not Uniform int4?

The claim is that matching the distribution pays. It does — but only for the distribution NF4 assumes. On normal weights (what pretrained models have) NF4’s equal-mass bins beat uniform int4 at the very same 4 bits. On uniform data the advantage reverses: a fixed codebook is optimal only for the shape it was built for. Teaching both is the honest version of the story.

from qlora import error_by_distribution

err = error_by_distribution()
ojs_define(nf4Err = err)
print("normal data  → NF4 MSE", round(err["normal"]["nf4"], 5),
      "< uniform MSE", round(err["normal"]["uniform"], 5))
print("uniform data → uniform MSE", round(err["uniform"]["uniform"], 5),
      "< NF4 MSE", round(err["uniform"]["nf4"], 5))
NoteKey Insight

NF4 is not “better quantization” in general — it is quantization tuned to a distribution. Because pretrained weights really are roughly normal, NF4 spends its 16 codes where the mass is and wins. Feed it uniform data and it loses to plain int4. The datatype encodes an assumption; state the assumption.

Double Quantization: Quantize the Scales Too

Blockwise NF4 has one lingering cost. Every block of 64 weights needs its own absmax, stored in fp32. That is 32 / 64 = 0.5 extra bits per parameter — on top of the 4 — just for the scales. Double Quantization quantizes those scales too: treat the fp32 absmax values as a second tensor and quantize them to 8-bit, in blocks of 256, with a small fp32 constant per second-level block.

\underbrace{b}_{\text{4-bit weight}} \;+\; \underbrace{\frac{32}{B}}_{\text{fp32 absmax}} \;\xrightarrow{\text{double-quant}}\; b \;+\; \underbrace{\frac{8}{B}}_{\text{int8 absmax}} \;+\; \underbrace{\frac{32}{B \cdot B_2}}_{\text{2nd-level fp32}}

With B = 64, B_2 = 256 the overhead falls from 0.5 to 0.127 bits/param — a 0.37 bit/param saving that, across 65B parameters, is ~3 GB back. Drive the memory ladder: pick a model size and read the frozen-weight footprint drop as you walk fp16 → int8 → NF4 → NF4 + double-quant.

from qlora import memory_ladder, bits_per_param

ojs_define(memLadder = memory_ladder())
print("bits/param  fp16=16  int8=8",
      " nf4=", round(bits_per_param(64, double_quant=False), 3),
      " nf4+dq=", round(bits_per_param(64, double_quant=True), 3))
print("double-quant saves",
      round(bits_per_param(64, False) - bits_per_param(64, True), 3), "bits/param")

The QLoRALinear Layer

Now assemble the object. QLoRALinear quantizes the base weight to NF4 once, stores it as buffers (no gradient, no optimizer state), and on every forward dequantizes it and adds the LoRA update through the r bottleneck:

h \;=\; \operatorname{dequant_{NF4}}(W_4)\,x \;+\; \frac{\alpha}{r} B A\, x .

Just like plain LoRA, B = 0 at initialization — so the adapter is an exact no-op and the layer is bit-for-bit the dequantized 4-bit base. That is the LoRA identity, now standing over a quantized base.

from qlora import QLoRALinear
import torch.nn as nn

torch.manual_seed(0)
base = nn.Linear(64, 64, bias=False)
ql = QLoRALinear(base, rank=8, alpha=8)        # base frozen in 4-bit NF4

x = torch.randn(4, 64)
# zero adapter (B=0) → exactly the dequantized 4-bit base, no adapter contribution
identity = torch.equal(ql(x), x @ ql.dequantized_weight().t())
print("zero-adapter identity holds:", identity)

trainable = {n for n, p in ql.named_parameters() if p.requires_grad}
print("trainable parameters:", trainable, "— the 4-bit base is buffers, not params")
print("base stored as:", ql.nf4_indices.dtype, "code indices +", ql.nf4_absmax.numel(),
      "block scales (no fp weight)")
zero-adapter identity holds: True
trainable parameters: {'A', 'B'} — the 4-bit base is buffers, not params
base stored as: torch.uint8 code indices + 64 block scales (no fp weight)

Fine-Tuning Over the Frozen 4-Bit Base

Put it in motion: quantize a base to NF4, freeze it, attach the adapter, and train only A and B to fit a target. Watch the loss fall while \lVert \Delta W \rVert climbs from exactly zero — and the 4-bit stored base (its uint8 code indices) stays byte-for-byte identical, start to finish. The gradient flows through the frozen 4-bit weights into the adapter; the weights themselves never move.

from qlora import demonstrate_qlora

demo = demonstrate_qlora()
curve = [{"step": s.step, "loss": s.loss, "delta": s.delta_norm} for s in demo["steps"]]
ojs_define(qloraCurve = curve)
ojs_define(qloraMeta = {"trainable": demo["trainable"], "bits": demo["base_bits"],
                        "unchanged": demo["base_unchanged"], "err": demo["dequant_error"]})
print("trainable adapter params:", demo["trainable"])
print("frozen base storage:", round(demo["base_bits"], 3), "bits/param (NF4 + double-quant)")
print("first-step ‖ΔW‖:", f"{demo['steps'][0].delta_norm:.4f}  (exactly 0)")
print("last-step  ‖ΔW‖:", f"{demo['steps'][-1].delta_norm:.2f}")
print("4-bit base bit-for-bit unchanged after training:", demo["base_unchanged"])
trainable adapter params: 256
frozen base storage: 4.127 bits/param (NF4 + double-quant)
first-step ‖ΔW‖: 0.0000  (exactly 0)
last-step  ‖ΔW‖: 60.88
4-bit base bit-for-bit unchanged after training: True
TipTry This
  1. Overlay the uniform code in the NF4 explorer. See how its levels ignore the bell curve — evenly spaced, most of them stranded in the empty tails.
  2. Switch the memory ladder to 7B. The nf4+dq bar sits far under the 48 GB line — a 7B fine-tune is comfortable on a laptop-class GPU; the 65B case is what QLoRA made newly possible.
  3. Read the two error bars. NF4 wins on normal data and loses on uniform — the datatype is an assumption about the weights, not a free lunch.
NoteKey Insight

QLoRA is LoRA’s memory story taken to its conclusion. LoRA removed the optimizer state for the base; QLoRA removes most of the weight memory too, by storing the read-only base in 4-bit NF4 (plus double-quantized scales). The adapter stays in 16-bit because it is the only thing that trains. Nothing about the low-rank math changes — the base just got 4× smaller.

DoRA: Split the Weight Into Magnitude and Direction

LoRA adds one low-rank update and stops there. DoRA (Liu et al., 2024) asks a sharper question about how a weight is allowed to move. Think of each output unit’s weight vector as having two independent properties: a length (how strongly it fires) and a heading (which input pattern it responds to). LoRA’s single BA update changes both at once — you cannot ask it to rotate a weight while holding its length fixed, because the length is just whatever \lVert W_0 + BA \rVert happens to come out to.

DoRA decomposes the weight first, then adapts the two pieces separately:

W = m \cdot \frac{V}{\lVert V \rVert_c},

where m is the magnitude — one trainable scalar per output unit — and V/\lVert V \rVert_c is the unit-norm direction. Here \lVert \cdot \rVert_c is the per-output-unit (row-wise) L2 norm. The direction gets a LoRA update; the magnitude gets its own full vector. Watch a single weight row split apart and recombine:

NoteKey Insight

LoRA and DoRA reach the same kind of weight — a single merged matrix — but DoRA gets there through two separate controls: a magnitude vector and a directional LoRA. That extra control is the whole method. It costs almost nothing (the m vector is one number per output unit, ~+0.01% params over LoRA) and it lets the adapter move a weight’s length and heading independently.

The Math: W' = m \cdot (W_0 + BA) / \lVert W_0 + BA \rVert_c

DoRA freezes the base direction V = W_0 and fine-tunes with

W' \;=\; m \cdot \frac{W_0 + \Delta V}{\lVert W_0 + \Delta V \rVert_c}, \qquad \Delta V = \frac{\alpha}{r} B A,

with three trainable pieces: the LoRA matrices A, B (the directional update) and the full magnitude vector m \in \mathbb{R}^{\text{out}}. Two details carry the method:

  1. Initialization keeps it a no-op. Set m = \lVert W_0 \rVert_c and (as in LoRA) B = 0 so \Delta V = 0. Then W' = \lVert W_0 \rVert_c \cdot W_0 / \lVert W_0 \rVert_c = W_0 — exactly. Fine-tuning begins as the pretrained model, the same identity LoRA has.

  2. The denominator is detached. In backprop, DoRA treats \lVert W_0 + \Delta V \rVert_c as a constant (stop-gradient). This changes nothing about the forward value — only the gradient path — and the paper reports it cuts training memory by ~24% on LLaMA-7B. We build it with a .detach().

Code: dora_decompose and DoRALinear

The decomposition is two lines, and it recomposes exactly. This follows dora.py.

import torch
from dora import dora_decompose, column_norm

torch.manual_seed(0)
W = torch.randn(4, 8)                      # 4 output units, 8 inputs
m, direction = dora_decompose(W)

print("magnitude m (one per output unit):", [round(v, 3) for v in m.tolist()])
print("directions are unit-norm:", torch.allclose(direction.norm(dim=1), torch.ones(4), atol=1e-6))
print("m · direction == W exactly:", torch.allclose(m[:, None] * direction, W, atol=1e-6))
magnitude m (one per output unit): [2.936, 2.184, 3.462, 2.65]
directions are unit-norm: True
m · direction == W exactly: True

The layer wraps a frozen nn.Linear, initializes m to the base’s row norms, and builds W' on every forward.

import torch.nn as nn
from dora import DoRALinear

torch.manual_seed(0)
base = nn.Linear(16, 8, bias=False)        # d=16 → k=8
dora = DoRALinear(base, rank=4, alpha=8.0)

print("m initialized to base row norms:", torch.allclose(dora.m.data, base.weight.data.norm(dim=1)))
print("B is zero at init:", bool(torch.count_nonzero(dora.B) == 0))
trainable = {n for n, p in dora.named_parameters() if p.requires_grad}
print("trainable pieces:", trainable, "— A, B (direction) + m (magnitude)")
m initialized to base row norms: True
B is zero at init: True
trainable pieces: {'A', 'B', 'm'} — A, B (direction) + m (magnitude)

Anchor 1 — the zero-init identity

Because m = \lVert W_0 \rVert_c and \Delta V = 0, DoRA is bit-for-bit the frozen layer at initialization — verified here on a real GPTModel:

from m06_transformer.transformer import GPTModel
from dora import inject_dora

model = GPTModel(vocab_size=100, embed_dim=32, num_heads=2, num_layers=2,
                 max_seq_len=64, dropout=0.0)
model.eval()
tokens = torch.randint(0, 100, (2, 16))
with torch.no_grad():
    before = model(tokens)

inject_dora(model, ["lm_head"], rank=4, alpha=8.0)   # freeze all, adapt lm_head
with torch.no_grad():
    after = model(tokens)

print("logits identical after attaching DoRA:", torch.allclose(before, after, atol=1e-4))
logits identical after attaching DoRA: True

Anchor 2 — merge equivalence (no inference latency)

Like LoRA, DoRA’s effective weight is a single (k, d) matrix, so a finished adapter merges into one plain nn.Linear and adds zero inference cost:

# give the adapter a real update to merge
with torch.no_grad():
    dora.B.copy_(torch.randn_like(dora.B))
    dora.m.add_(0.3 * torch.randn_like(dora.m))

x = torch.randn(5, 16)
merged = dora.merge()
diff = (dora(x) - merged(x)).detach().abs().max()
print(f"max |DoRA(x) − merged(x)|: {diff.item():.2e}")
print("merged is a plain nn.Linear:", isinstance(merged, nn.Linear))
max |DoRA(x) − merged(x)|: 0.00e+00
merged is a plain nn.Linear: True

The Decoupling LoRA Can’t Do

Here is the mechanism, made visible. Take one directional update \Delta V and ask each method: over that same \Delta V, what magnitude change \Delta M and direction change \Delta D can you produce, per output unit?

  • LoRA merges to exactly W_0 + \Delta V — a single point. Its magnitude change is pinned to \lVert W_0 + \Delta V \rVert_c - \lVert W_0 \rVert_c, and it correlates with the directional change: rows that rotate more also stretch more, together.
  • DoRA merges to m \cdot (W_0 + \Delta V) / \lVert W_0 + \Delta V \rVert_c — the same direction change, but m sets the magnitude freely. Here we drive m so that length moves against rotation.
from dora import demonstrate_decoupling

dec = demonstrate_decoupling()
ojs_define(doraDecoupling = dec)
print("DoRA reproduces LoRA's direction change exactly:", dec["same_direction"])
print(f"LoRA  corr(ΔM, ΔD) = {dec['lora']['corr']:+.2f}   (coupled — positive)")
print(f"DoRA  corr(ΔM, ΔD) = {dec['dora']['corr']:+.2f}   (magnitude set freely — negative)")
DoRA reproduces LoRA's direction change exactly: True
LoRA  corr(ΔM, ΔD) = +0.85   (coupled — positive)
DoRA  corr(ΔM, ΔD) = -1.00   (magnitude set freely — negative)
TipTry It!

Both point clouds sit at the same set of ΔD values — DoRA reuses LoRA’s exact directional update. Only the vertical axis differs: LoRA’s magnitude rides up with rotation (a positive slope it cannot avoid), while DoRA’s is set by m and here slopes down. That vertical freedom is the entire method.

Why It Matters: the Fine-Tuning Pattern

Why is decoupling worth it? Because it is how full fine-tuning actually moves weights. The DoRA paper measures, across layers, the correlation between each weight’s magnitude change \Delta M and direction change \Delta D:

Method corr(\Delta M, \Delta D) Pattern
Full fine-tuning -0.62 negative — big rotations come with small length changes
LoRA +0.83 positive — length and heading move together (coupled)
DoRA -0.31 negative — recovers the fine-tuning pattern
NoteKey Insight

LoRA’s single low-rank update forces magnitude and direction to move together (a +0.83 correlation), the opposite sign from full fine-tuning (-0.62). DoRA’s separate magnitude knob lets it produce the negative pattern, and that closes much of the gap: on commonsense reasoning DoRA lifts LLaMA-7B from LoRA’s 74.7 to 78.4 (+3.7), at ~+0.01% extra parameters. Same rank, one extra vector.

Fine-Tuning With DoRA

Put it in motion: freeze a base, attach DoRA, and train A, B, and m to fit a target whose rows have been both rotated and rescaled — a change that needs both knobs. Watch the loss fall while the directional update \lVert \Delta V \rVert and the magnitude shift \lVert m - m_0 \rVert both climb from exactly zero, and the frozen base never moves.

from dora import demonstrate_dora

demo = demonstrate_dora()
curve = [{"step": s.step, "loss": s.loss,
          "dir": s.delta_dir_norm, "mag": s.mag_shift} for s in demo["steps"]]
ojs_define(doraCurve = curve)
ojs_define(doraMeta = {"trainable": demo["trainable"], "frozen": demo["frozen"],
                       "m": demo["m_params"], "base_unchanged": demo["base_unchanged"]})
print("trainable (A + B + m):", demo["trainable"], "| frozen (W₀):", demo["frozen"])
print("magnitude vector m adds:", demo["m_params"], "params (one per output unit)")
print("first-step ‖ΔV‖:", f"{demo['steps'][0].delta_dir_norm:.4f}  (exactly 0)")
print("first-step ‖m−m₀‖:", f"{demo['steps'][0].mag_shift:.4f}  (exactly 0)")
print("base weight bit-for-bit unchanged:", demo["base_unchanged"])
trainable (A + B + m): 144 | frozen (W₀): 256
magnitude vector m adds: 16 params (one per output unit)
first-step ‖ΔV‖: 0.0000  (exactly 0)
first-step ‖m−m₀‖: 0.0000  (exactly 0)
base weight bit-for-bit unchanged: True
NoteKey Insight

DoRA is LoRA with the weight’s length peeled off into its own parameter. The LoRA update handles the direction; the magnitude vector m handles the length. Everything LoRA gives you survives — the zero-init identity, merge-away-the-adapter, the frozen shared base — and you gain an independent magnitude control that recovers full fine-tuning’s update pattern, for one extra number per output unit.

Beyond LoRA

LoRA is the foundation of a whole family. The book builds the core, QLoRA, and DoRA above; these are the remaining load-bearing extensions (roadmap follow-ups):

  • QLoRA (Dettmers et al., 2023) — built above: freeze the base in 4-bit (NF4 quantization, m14) and keep the LoRA adapter in full precision. Because the frozen base carries no optimizer state, quantizing it is nearly free — this is what fits a 65B fine-tune on a single 48 GB GPU.
  • DoRA (Liu et al., 2024) — built above: decompose the weight into magnitude and direction, and apply LoRA only to the direction; the separate magnitude knob recovers full fine-tuning’s update pattern and closes much of the gap at ~+0.01% extra parameters.
  • Adapters (Houlsby et al., 2019) — the predecessor: small bottleneck MLPs inserted between layers. They work, but unlike LoRA they cannot be merged, so they add permanent inference latency.
  • Prefix / prompt tuning (Li & Liang; Lester et al., 2021) — freeze the whole model and learn a handful of virtual “prefix” key/value vectors instead of touching weights at all.

Common Pitfalls

Pitfall Why it bites Fix
Initializing both A and B randomly \Delta W \ne 0 at step 0 — you start by corrupting the pretrained model Keep B = 0 (Gaussian A) so training begins as the identity
Treating rank r as “bigger is better” A large r erases the savings and can overfit the small fine-tune set Start small (r = 8–16); raise only if the task underfits
Forgetting to freeze the base You silently full-fine-tune — the memory win evaporates requires_grad_(False) on the base (what inject_lora does)
Merging, then continuing to train Once merged, there is no separate adapter to update Merge only for deployment; keep the adapter while training
Adapting the wrong layers The update has to land where the task needs it The paper adapts attention W_q, W_v; that is the usual first choice
Letting DoRA’s norm gradient blow up memory The per-row renorm makes the backward graph heavier than LoRA’s Detach the denominator \lVert W_0+\Delta V\rVert_c (what DoRALinear does) — same forward, ~24% less training memory

Exercises

Exercise 1: The rank bound

Show that no matter what A and B contain, the update \Delta W has rank at most r.

from lora import LoRALinear

base = nn.Linear(32, 24, bias=False)
lora = LoRALinear(base, rank=3, alpha=3.0)
with torch.no_grad():
    lora.A.copy_(torch.randn_like(lora.A))
    lora.B.copy_(torch.randn_like(lora.B))

rank = torch.linalg.matrix_rank(lora.delta_weight())
print("rank(ΔW):", int(rank), "≤ r =", lora.rank)
rank(ΔW): 3 ≤ r = 3

Exercise 2: Only the adapter gets gradients

Run one backward pass and confirm the base weight has no gradient while A and B do.

base = nn.Linear(8, 8, bias=False)
lora = LoRALinear(base, rank=2)
with torch.no_grad():
    lora.B.copy_(torch.randn_like(lora.B))     # leave the zero-init so B gets a grad
lora(torch.randn(3, 8)).pow(2).mean().backward()

print("base.weight.grad is None:", base.weight.grad is None)
print("A.grad set:", lora.A.grad is not None, "| B.grad set:", lora.B.grad is not None)
base.weight.grad is None: True
A.grad set: True | B.grad set: True

Exercise 3: Swap two adapters on one base

One frozen model, two adapters. Attach B₁, then a different B₂, and confirm the merged weights differ — the “many hats, one base” story.

# Your implementation here:
# 1. build a base nn.Linear and two LoRALinear wrappers around copies
# 2. set different B matrices
# 3. compare merged_weight() — they should differ, base unchanged

Summary

Key takeaways:

  1. Fine-tuning updates are low-rank. LoRA exploits this by writing \Delta W = BA with rank r \ll \min(d, k) — the same expressive ceiling as any rank-r update, at a fraction of the parameters.
  2. The forward pass is h = W_0 x + \frac{\alpha}{r} BA\,x. Signal squeezes through the r-dim bottleneck; A down-projects, B up-projects, \alpha/r scales.
  3. Zero-init identity. B = 0 at start, so \Delta W = 0 and the adapted layer is exactly the pretrained one — verified bit-for-bit on a real GPTModel.
  4. No inference latency. W = W_0 + \frac{\alpha}{r} BA merges the adapter into one matrix — unlike inserted adapter modules.
  5. The budget: r(d+k) vs dk. For a square layer that is a d/2r reduction — 256× at d = 4096, r = 8; 10,000× fewer trainable params for GPT-3 175B.
  6. The base stays frozen. Only A and B train, so the base carries no optimizer state (the real memory win) and is shared across many swappable adapters.
  7. DoRA decouples magnitude from direction. Decompose W = m \cdot V/\lVert V \rVert_c, LoRA the direction, give magnitude its own vector m. The zero-init identity and merge equivalence carry over; the extra knob (~+0.01% params) recovers full fine-tuning’s magnitude/direction pattern LoRA can’t.

What’s Next

You now have the technique that makes every other fine-tuning method in this book affordable — plug it into the SFT and DPO objectives of Module 12: Alignment. And you built QLoRA above, combining LoRA with the 4-bit NF4 quantization of Module 14: Quantization — the recipe that fits a 65B fine-tune on a single GPU.

Going Deeper

Core Papers:

Practical Resources: