Module 22: Linear & Recurrent Attention

Introduction

Two modules ago (m19) you built the leading alternative to attention — the state-space model, a linear recurrence over a fixed-size state. This module closes the loop by showing that the “alternative” was hiding inside attention all along.

Linear attention is self-attention with one thing removed: the softmax. That single deletion turns attention from an O(L^2) all-pairs comparison into a linear RNN — a recurrence over a fixed-size matrix state, exactly the shape of the SSM. The slogan from the paper that started this line of work says it plainly: transformers are RNNs.

From there, one small addition — a decay on the state — gives retention (the RetNet architecture), which comes with three provably-identical faces: a parallel form for training, a recurrent form for cheap decoding, and a chunkwise form that gets both at once. Then the payoff: linear attention, retention, and the SSMs of m19 are literally the same recurrence, differing only in one gate. Finally we open a second axis — not how the state forgets but how it writes — and build the delta rule (DeltaNet), an error-correcting write that can overwrite a stored memory, fixing the associative-recall weakness of the additive mixers. Knowing that one recurrence, and those two axes, is how you read the whole zoo of sub-quadratic sequence models.

Why it matters for LLMs:

  • Constant-memory decoding. Softmax attention’s KV cache (m09) grows with every token; linear attention keeps a fixed-size state, so each generated token costs the same no matter how long the context.
  • Linear-time training and inference. No L \times L score matrix — compute scales with L, not L^2.
  • One mental model for many architectures. RetNet, RWKV, gated linear attention, and Mamba are one idea with different gates. This module gives you that idea.

What You’ll Learn

After this module, you can:

  • Explain why the softmax is the only thing forcing attention to be O(L^2), and how a kernel feature map \varphi removes it.
  • Run linear attention as a masked matmul (parallel) and as a running-state recurrence, and see they are the same function.
  • Build retention (RetNet): a decayed state with a \gamma^{n-m} decay mask, in its parallel, recurrent, and chunkwise forms — all provably equal.
  • Write the one unifying recurrence S_t = A_t S_{t-1} + k_t^\top v_t that specializes to linear attention (A_t=1), retention (A_t=\gamma), and a diagonal SSM / gated linear attention (A_t input-dependent).
  • Build the delta rule (DeltaNet): an error-correcting write S_t = (I - \beta_t k_t k_t^\top)S_{t-1} + \beta_t k_t v_t^\top that can overwrite a stored value — and its parallel UT transform — to fix the associative-recall weakness of additive linear attention.
  • Compose both axes into Gated DeltaNet — a forget gate on top of the delta write (S_t = \alpha_t(I - \beta_t k_t k_t^\top)S_{t-1} + \beta_t k_t v_t^\top) — and parallelize it for free with a change of variables that reuses the UT transform; see how \alpha (global forget) and \beta (targeted overwrite) are orthogonal erasers.
  • Train the delta rule the way GPUs do — the chunkwise algorithm (parallel within chunks, a carried state between them) — and see why gating makes chunking a numerical necessity, not just a speed trick (the global rescale underflows to NaN; a per-chunk-local decay does not).
  • Build RWKV’s time-mixing block from scratch — a deployed, query-free time-decayed RNN with a token shift and a receptance gate — and watch its parallel training pass and O(1) RNN decode compute the same thing.
  • Quantify the win: linear vs quadratic compute, constant vs growing decode memory.

Prerequisites

This module requires familiarity with:

Intuition: Attention Without the Softmax

Recall causal self-attention for query position i:

y_i = \frac{\sum_{j \le i} \exp(q_i \cdot k_j)\, v_j}{\sum_{j \le i} \exp(q_i \cdot k_j)} .

The trouble is the \exp(q_i \cdot k_j): it entangles q_i and k_j inside one nonlinearity, so you cannot separate them. You are forced to compute a score for every pair (i, j) — the L \times L matrix — before you can sum. That matrix is the whole O(L^2) cost.

Now suppose the similarity factored — suppose we could write \text{sim}(q_i, k_j) = \varphi(q_i) \cdot \varphi(k_j) for some feature map \varphi. Then the numerator regroups by associativity:

\sum_{j \le i} \big(\varphi(q_i) \cdot \varphi(k_j)\big)\, v_j = \varphi(q_i) \underbrace{\sum_{j \le i} \varphi(k_j)\, v_j^\top}_{S_i} .

The sum S_i no longer depends on i except through its upper limit — it is a running total you can carry forward one token at a time. No L \times L matrix ever forms. The same regrouping is the difference between multiplying (QK^\top)V (build the big matrix first) and Q(K^\top V) (build a small state first). Step through both association orders and watch the intermediate shape:

NoteKey Insight

The softmax is the only thing that forces the L \times L matrix. Remove it and replace the pairwise score with a factored kernel \varphi(q)\cdot\varphi(k), and the associativity of matrix multiplication lets you keep a fixed-size state instead — the same move that makes an SSM linear-time.

The Math: Linear Attention

We need a \varphi that keeps similarities non-negative (so the normalizer never turns negative) and is cheap. Katharopoulos et al. (2020) use

\varphi(x) = \operatorname{elu}(x) + 1 ,

which is x+1 for x>0 and \exp(x)\in(0,1] for x\le 0 — positive for any realistic activation. With \text{sim}(q,k)=\varphi(q)\cdot\varphi(k), causal attention becomes a pair of running states:

S_i = \sum_{j\le i} \varphi(k_j)\, v_j^\top, \qquad Z_i = \sum_{j\le i} \varphi(k_j), \qquad y_i = \frac{\varphi(q_i)^\top S_i}{\varphi(q_i)^\top Z_i} .

S_i is the (m \times d_v) state matrix — a running, weighted memory of every value seen — and Z_i is the running normalizer. Both are fixed-size and updated by one addition per token. linear_attention.py implements exactly this:

import torch
from linear_attention import feature_map, linear_attention_recurrent

torch.manual_seed(0)
Q = torch.randn(6, 8)   # (length, d)
K = torch.randn(6, 8)
V = torch.randn(6, 8)   # (length, d_v)

y = linear_attention_recurrent(Q, K, V)
print("phi positive:", bool((feature_map(K) > 0).all()))
print("output shape:", tuple(y.shape))          # (length, d_v), no L×L matrix built
phi positive: True
output shape: (6, 8)

Code: The Two Faces of Linear Attention

Just like the SSM in m19 had a recurrent face and a convolutional face, linear attention has a recurrent face (the running state above, O(L), cheap to decode) and a parallel face that keeps the familiar attention layout — build the masked score matrix A_{ij}=\varphi(q_i)\cdot\varphi(k_j) for j\le i, normalize each row, and read out V:

y_i = \frac{\sum_{j\le i} A_{ij}\, v_j}{\sum_{j\le i} A_{ij}} .

The parallel face is one masked matmul (great for training on a GPU); the recurrent face never forms a matrix (great for decoding). For the same inputs they compute bit-for-bit the same function — the linear-attention analogue of m19’s recurrent≡convolutional identity:

from linear_attention import linear_attention_parallel

y_par = linear_attention_parallel(Q, K, V)
y_rec = linear_attention_recurrent(Q, K, V)
print("parallel == recurrent:", torch.allclose(y_par, y_rec, atol=1e-5))
parallel == recurrent: True

Bridge both to a plot: the recurrent dots land exactly on the parallel line, because they are the same function computed two ways.

For decoding, wrap the running state in a small object and fold in one token at a time — the constant-work-per-token generation that a growing KV cache cannot match. LinearAttentionState does this:

from linear_attention import LinearAttentionState

state = LinearAttentionState(m=8, d_v=8)
ys = torch.stack([state.step(Q[i], K[i], V[i]) for i in range(6)])
print("incremental == full recompute:", torch.allclose(ys, y_rec, atol=1e-5))
incremental == full recompute: True
NoteKey Insight

Train like attention, decode like an RNN. The parallel face gives fast, parallelizable training; the recurrent face gives constant-memory generation. The price is a matrix state of size m \times d_v (typically d \times d) instead of a growing cache — constant in L, but quadratic in the model width.

Retention: Add a Decay to the State

Linear attention keeps every past token with equal weight — its state S_i is an undamped running sum, which can saturate and blur. Retention (the RetNet architecture, Sun et al., 2023) adds one ingredient: a scalar decay \gamma \in (0,1] that shrinks the old state before each update:

S_n = \gamma\, S_{n-1} + k_n^\top v_n, \qquad o_n = q_n\, S_n .

Two things are different from linear attention. First, older tokens fade geometrically — the state prefers recent context. Second, retention drops the normalizer entirely (no softmax, no division); RetNet stabilizes the output with a GroupNorm and a swish gate instead, but the core operation is this bare decayed state. Unrolling the recurrence, S_n = \sum_{m\le n} \gamma^{\,n-m} k_m^\top v_m, so the output is

o_n = \sum_{m\le n} \gamma^{\,n-m} (q_n\cdot k_m)\, v_m .

That is ordinary QK^\top scores, causally masked, weighted by a decay \gamma^{n-m} — a fixed lower-triangular decay matrix D. Drive \gamma and watch D interpolate between two familiar limits:

TipTry It!
  1. Slide γ → 1. Every causal entry becomes 1: the decay vanishes and retention is uniform causal attention (unnormalized). This is exactly linear attention’s state with \varphi=\text{identity} and no forgetting.
  2. Slide γ → 0. Only the diagonal survives (0^0=1): each token sees only itself. The decay is the dial between “remember everything” and “remember only now”.
  3. Mid-range γ. A soft, fixed locality window — and note there is no positional encoding anywhere. The decay is the position information.

The Three Faces of Retention

Retention’s real selling point is that the same function has three computational forms, and you pick whichever the situation wants:

  • Parallel — \text{Retention}(X) = (QK^\top \odot D)\,V. One masked matmul, fully parallel. Best for training short-to-medium sequences. O(L^2).
  • Recurrent — S_n = \gamma S_{n-1} + k_n^\top v_n, o_n = q_n S_n. Constant state, one step at a time. Best for decoding. O(L), O(1) memory per token.
  • Chunkwise — split into chunks of size B; compute each chunk internally by the parallel form, and carry a fixed-size state R between chunks by the recurrent form. Best for training on long sequences: parallelism inside a chunk, linear scaling across chunks. O(LB).

The chunkwise form is the one production systems use. For a token at chunk-local index t it adds an intra-chunk term (parallel) to a cross-chunk term that reads the carried state R, decayed by \gamma^{t+1}:

o_t = \underbrace{\sum_{s\le t}\gamma^{t-s}(q_t\cdot k_s)v_s}_{\text{intra-chunk (parallel)}} \; + \; \underbrace{\gamma^{\,t+1}\,(q_t R)}_{\text{cross-chunk (recurrent)}}, \qquad R \leftarrow \gamma^{B} R + \sum_s \gamma^{\,B-1-s} k_s^\top v_s .

All three are the same function. linear_attention.py proves it numerically:

from linear_attention import (retention_parallel, retention_recurrent,
                              retention_chunkwise)

torch.manual_seed(0)
Qr = torch.randn(12, 6); Kr = torch.randn(12, 6); Vr = torch.randn(12, 6)
gamma = 0.9

y_par = retention_parallel(Qr, Kr, Vr, gamma)
y_rec = retention_recurrent(Qr, Kr, Vr, gamma)
y_chk = retention_chunkwise(Qr, Kr, Vr, gamma, chunk_size=4)

print("parallel == recurrent:", torch.allclose(y_par, y_rec, atol=1e-5))
print("parallel == chunkwise:", torch.allclose(y_par, y_chk, atol=1e-5))
parallel == recurrent: True
parallel == chunkwise: True

Real RetNet uses multi-scale retention: each head gets its own \gamma_h = 1 - 2^{-5-h}, so some heads keep long memory and others stay local — the decay analogue of multi-head attention’s different subspaces.

from linear_attention import multiscale_decays
print("per-head γ:", [round(v, 4) for v in multiscale_decays(4).tolist()])
per-head γ: [0.9688, 0.9844, 0.9922, 0.9961]

Interactive Exploration: The One Recurrence Behind Them All

Here is the idea this whole module builds toward. Linear attention, retention, and the diagonal SSM of m19 are the same recurrence

S_t = A_t\, S_{t-1} + k_t^\top v_t, \qquad y_t = q_t\, S_t ,

differing only in the gate A_t that scales the old state:

Model Gate A_t Behavior
Linear attention 1 (identity) Keep everything, equal weight
Retention (RetNet) \gamma (constant) Forget geometrically
SSM / gated linear attn / Mamba A_t = f(x_t) (input-dependent) Choose what to keep, per token

Here the gate A_t is a scalar — one number decaying the whole state. The next section lifts it to a per-dimension vector (Gated Linear Attention), so each feature can be kept or flushed independently; the same recurrence, a richer gate.

The unified_recurrence function is literally one loop with a swappable A_t. Below, you are the gate: check which tokens are “important”. Important tokens get a high gate (A_t \approx 0.95, the state holds); the rest get a low gate (A_t \approx 0.2, the state forgets). Watch the memory integrate only what you select — this is selectivity, the input-dependent gate that lifts a fixed decay into Mamba’s content-based reasoning.

Now compare the three regimes on one fixed input stream — linear attention’s state grows without bound (nothing forgets), retention’s saturates (a constant decay), and the gated state rises and falls as the gates open and close:

NoteKey Insight

This is the m19↔︎attention duality (formalized by Mamba-2) made runnable: the gate A_t here is exactly the SSM’s discrete decay \bar{A}. The difference between a “transformer” and a “state-space model” collapses to a single question — is the gate constant or input-dependent?

Why It Matters: Cost

The payoff, made quantitative. Softmax attention does two O(L^2 d) matmuls and stores a KV cache that grows linearly with the context. Linear attention does an O(d^2) state update per token over L tokens — O(L d^2), linear in L — and carries a fixed d \times d state that never grows. Drive the context length and watch the compute curves diverge:

NoteKey Insight

Linear attention trades a growing KV cache for a fixed matrix state. That state is larger than an SSM’s (d \times d vs d \times N with a small N), which is why pure linear attention has historically traded a little quality for that memory constant — and why the frontier (gated linear attention, Mamba, hybrids) works to recover the quality while keeping the constant-memory decode.

Gated Linear Attention: A Learned, Per-Dimension Gate

Retention’s single \gamma is a blunt instrument. It fades every feature at the same rate — the subject of a sentence decays as fast as the last comma. A good memory forgets selectively: hold the thread of the argument, flush the punctuation. Gated Linear Attention (GLA) does exactly that by making the gate two things at once: a vector — one decay per key dimension — and data-dependent — computed from the current token. The unifying recurrence we just wrote had a scalar A_t; GLA promotes it to \mathrm{Diag}(\alpha_t).

Drive it yourself. Below, four key dimensions each get a single input impulse and their own gate \alpha_i. Watch the state row light up and then fade — a high gate holds the memory for many steps, a low gate forgets it almost immediately. No single \gamma could give these rows different tails; that is the whole point.

NoteKey Insight

The gate is a vector, not a scalar. Retention decays the whole state matrix by one \gamma; GLA decays row i of the state by its own \alpha_{t,i}. Because \alpha_t is computed from the token, the model learns — per feature, per step — what to keep and what to flush. That is the expressiveness jump from a fixed decay to a selective one, and it is the additive-write cousin of the Gated DeltaNet you will meet two sections down.

The Math: A Diagonal Gate

The recurrence keeps the shape of everything before it and swaps in the vector gate:

S_t = \mathrm{Diag}(\alpha_t)\, S_{t-1} + k_t^\top v_t, \qquad o_t = q_t\, S_t ,

with \alpha_t \in (0, 1]^{d_k}. Writing it as the rank-one gating matrix G_t = \alpha_t \mathbf{1}^\top (each row of the d_k \times d_v state scaled by one entry of \alpha_t) makes the family relationships exact:

Gate \alpha_t Recovers Because
\alpha_t \equiv \mathbf{1} unnormalized linear attention nothing ever decays
\alpha_t \equiv \gamma\mathbf{1} (constant, all dims equal) retention (RetNet) one shared scalar decay
\alpha_t = \sigma(x_t W_1 W_2)^{1/\tau} GLA a learned, per-dimension decay

The gate itself is produced cheaply: a low-rank projection (W_1: d\to r, W_2: r\to d_k with r \ll d) into a sigmoid, optionally sharpened by a temperature \tau (16 in the paper) that biases the gate toward 1 — slow forgetting by default, fast only when the token asks for it.

Code: The Gate and the Recurrence

gla_gate and gla_recurrent in linear_attention.py are that math, verbatim. The two special cases above are not hand-waving — they hold to the bit:

import torch
from linear_attention import gla_gate, gla_recurrent, retention_recurrent

torch.manual_seed(0)
L, d, d_v = 8, 6, 6
Q, K, V = torch.randn(L, d), torch.randn(L, d), torch.randn(L, d_v)

# A data-dependent gate: low-rank map → sigmoid → per-dimension α_t.
X = torch.randn(L, d)                       # the token features driving the gate
W1, W2 = torch.randn(d, 2), torch.randn(2, d)
alpha = gla_gate(X, W1, W2)
print(f"gate shape {tuple(alpha.shape)}  ·  every entry in (0,1]: {bool((alpha > 0).all() and (alpha <= 1).all())}")

# Anchor 1: a constant gate α ≡ γ IS retention.
gamma = 0.9
same_as_retention = torch.allclose(
    gla_recurrent(Q, K, V, torch.full((L, d), gamma)),
    retention_recurrent(Q, K, V, gamma), atol=1e-5)

# Anchor 2: an all-ones gate keeps everything = linear attention (retention at γ=1).
same_as_linear = torch.allclose(
    gla_recurrent(Q, K, V, torch.ones(L, d)),
    retention_recurrent(Q, K, V, 1.0), atol=1e-5)

print(f"α ≡ γ  → retention:        {same_as_retention}")
print(f"α ≡ 1  → linear attention: {same_as_linear}")
gate shape (8, 6)  ·  every entry in (0,1]: True
α ≡ γ  → retention:        True
α ≡ 1  → linear attention: True

GLA doesn’t replace retention — it contains it, and unlocks the per-dimension gate retention can’t express.

Three Faces, One Function

Like every mixer in this module, GLA has a parallel training face and a chunkwise face, and all three compute the same thing. The trick is the cumulative gate b_t = \prod_{s\le t}\alpha_s: token m’s contribution reaches token n decayed by \prod_{m<s\le n}\alpha_s = b_n/b_m. So rescaling \tilde q_n = q_n \odot b_n and \tilde k_m = k_m / b_m turns the whole thing into one masked matmul \big(\tilde Q\,\tilde K^\top \odot \text{mask}\big)V — the parallel face.

from linear_attention import gla_parallel, gla_chunkwise

torch.manual_seed(1)
Q, K, V = torch.randn(12, 6), torch.randn(12, 6), torch.randn(12, 8)
alpha = torch.rand(12, 6) * 0.2 + 0.75              # gentle gates in (0.75, 0.95)

rec   = gla_recurrent(Q, K, V, alpha)
par   = gla_parallel(Q, K, V, alpha)
chunk = gla_chunkwise(Q, K, V, alpha, chunk_size=4)

print(f"|parallel  − recurrent| = {(par   - rec).abs().max():.2e}")
print(f"|chunkwise − recurrent| = {(chunk - rec).abs().max():.2e}")
|parallel  − recurrent| = 1.91e-06
|chunkwise − recurrent| = 1.91e-06

One recurrence, three schedules — a serial O(1)-memory decode, a fully parallel training pass, and the chunked form that makes long-sequence training practical. It is the exact per-dimension generalization of retention_chunkwise: its scalar \gamma^{\,t} simply becomes the vector cumulative product b_t.

Why Chunkwise: The Cumulative Product Underflows

The parallel face is elegant but fragile. b_t = \prod \alpha_s is a product of numbers below 1, so over a long sequence it collapses toward zero — and the k/b_t rescale it demands blows up as b_t^{-1}. In float32, once \alpha^{-L} passes the largest finite value (\approx 3.4\times10^{38}) the matmul overflows to infinity and the output is NaN. The chunkwise face fixes this the same way the gated-delta section did: reset the cumulative gate at every chunk boundary, so the product only ever spans a handful of tokens and stays well inside float’s range.

TipTry This

Drop the gate in demonstrate_gla_stability(gate=…) toward 0 and the overflow cliff marches left — a faster-forgetting gate underflows in fewer steps, so the parallel face dies sooner. Raise it toward 1 and the cliff slides right, past any length you would train on. This is why real GLA implementations never run the naive parallel form at length: they run the chunkwise one, which has no cliff.

A Different Kind of Memory: The Delta Rule

GLA learned how much to forget, per dimension — but its write is still the plain additive outer product every mixer in this module has used so far, S_t = A_t S_{t-1} + k_t v_t^\top. That write can only ever add. It is a Hebbian “write-and-never-erase” memory, and it has a sharp failure mode.

Suppose you store a value under a key, then later store a different value under the same key — an update. Write (k_A, v_1), then (k_A, v_2). The additive state becomes S = k_A v_1^\top + k_A v_2^\top, so reading key A back returns

k_A^\top S = v_1 + v_2 ,

the sum of both values — the old fact and the new one, blurred together. The memory cannot represent “A now means v_2, forget v_1”. This is exactly the associative-recall weakness that has kept pure linear attention a notch below softmax: real language needs to update what a name, variable, or entity refers to as the context unfolds, and an add-only memory smears every update into every prior value.

The delta rule fixes this with one idea: before writing, read what the memory already thinks, and write only the error. It is the classic learning rule of Widrow & Hoff (1960), recast as the memory update of a linear transformer — the “fast-weight” view (Schlag et al., 2021), and the recurrence behind DeltaNet (Yang et al., 2024). To store v_t under key k_t:

  1. Read the value the memory currently returns for k_t: \hat v_t = k_t^\top S_{t-1}.
  2. Take the error toward the target: \delta_t = v_t - \hat v_t.
  3. Nudge the memory by \beta_t times that error: S_t = S_{t-1} + \beta_t\, k_t\, \delta_t^\top.

If the memory already stored v_t for k_t, the error is zero and nothing changes. If it stored something else, the update erases that something and writes the new value in its place. Step through the read-modify-write:

NoteKey Insight

Substitute \hat v_t = k_t^\top S_{t-1} into the update and the additive gate becomes a matrix gate: S_t = \big(I - \beta_t\, k_t k_t^\top\big)\, S_{t-1} + \beta_t\, k_t v_t^\top . The rank-1 term \beta_t k_t k_t^\top subtracts the old value along the key direction before the new one is written. That is the whole difference from every other mixer in this module: linear attention and retention can only accumulate; the delta rule can overwrite.

The Math: Error-Correcting Writes

The delta rule keeps the fixed-size state and the two-faces structure of linear attention — it changes only the write. Two details make it well-behaved.

Write strength \beta_t. This is a per-token learning rate in (0, 1], usually produced as \beta_t = \sigma(\cdot) from the token. \beta_t = 1 replaces the stored value outright; \beta_t \to 0 leaves the memory untouched. It is the delta-rule analogue of retention’s decay, but local to the key being written rather than global to the whole state.

Unit keys. The transition I - \beta_t k_t k_t^\top is stable only when it does not stretch the state. With an L2-normalized key (\lVert k_t \rVert = 1) its eigenvalues are exactly 1 (perpendicular to k_t, untouched) and 1 - \beta_t (along k_t, shrunk) — all in [0, 1], so the state never expands. This is the same spectral-radius rule as the SSM’s \bar A \in (0,1) in m19; DeltaNet therefore normalizes keys (and usually queries) before the recurrence. linear_attention.py implements the read-modify-write directly:

import torch
from linear_attention import delta_rule_recurrent, l2_normalize

torch.manual_seed(0)
Q = torch.randn(6, 8)
K = torch.randn(6, 8)
V = torch.randn(6, 8)
beta = torch.full((6,), 0.8)          # per-token write strength in (0, 1]

y = delta_rule_recurrent(Q, K, V, beta)
print("keys normalized to unit length:", bool(torch.allclose(
    l2_normalize(K).norm(dim=-1), torch.ones(6), atol=1e-5)))
print("output shape:", tuple(y.shape))   # (length, d_v), fixed d×d state as before
keys normalized to unit length: True
output shape: (6, 8)

Code: Overwriting a Memory

Here is the payoff, made runnable. Store a value under key A, a value under key B, then overwrite A with a new value, and read A back. demonstrate_overwrite runs exactly that trace on orthonormal keys so the arithmetic is exact:

from linear_attention import demonstrate_overwrite

trace = demonstrate_overwrite()
print("additive memory returns:", trace["additive"])   # v1 + v2 — both values, blurred
print("delta rule returns:     ", trace["delta"])      # v2 — the update, clean
print("target (latest value):  ", trace["target"])
print("delta exact:", trace["delta_exact"])
additive memory returns: [1.0, 0.0, 1.0]
delta rule returns:      [0.0, 0.0, 1.0]
target (latest value):   [0.0, 0.0, 1.0]
delta exact: True

The additive memory returns [1, 0, 1] — it still carries the value it stored the first time. The delta rule returns [0, 0, 1], the current value of A, exactly. Bridge the readouts to a bar chart:

TipTry This
  1. Change \beta on the overwrite. In demonstrate_overwrite(beta=0.5), the overwrite is only half-applied — the readout becomes a blend of the old and new values. \beta is a dial between “keep the old memory” and “replace it”.
  2. Add a third write to key A. The delta rule tracks the latest value; the additive memory keeps summing. The gap only grows.

The Two Faces of the Delta Rule

Like linear attention and retention, the delta rule has a parallel training face — but the matrix transition I - \beta_t k_t k_t^\top means retention’s scalar decay-mask trick no longer applies. The fix is elegant. The state is still a sum of key outer products, S_t = \sum_{i \le t} k_i u_i^\top, for a set of effective values u_i. Matching the recurrence gives a triangular system for them:

u_t = \beta_t\Big(v_t - \sum_{i < t} (k_t \cdot k_i)\, u_i\Big) \quad\Longleftrightarrow\quad (I + T)\,U = \operatorname{diag}(\beta)\,V ,

where T_{ti} = \beta_t (k_t \cdot k_i) is strictly lower-triangular. Solve that unit-lower-triangular system once, then read out with a single causal matmul:

U = (I + T)^{-1}\operatorname{diag}(\beta)\,V , \qquad O = \operatorname{tril}(Q K^\top)\,U .

This is the UT transform — the I - \sum_i w_i k_i^\top WY representation that Yang et al. (2024) used to finally train DeltaNet in parallel over sequence length. No token-by-token loop; one solve and two matmuls. And it computes bit-for-bit the same function as the recurrence:

from linear_attention import delta_rule_parallel

torch.manual_seed(1)
Q = torch.randn(9, 6); K = torch.randn(9, 6); V = torch.randn(9, 6)
beta = torch.rand(9)

y_rec = delta_rule_recurrent(Q, K, V, beta)
y_par = delta_rule_parallel(Q, K, V, beta)
print("recurrent == parallel (UT transform):", torch.allclose(y_rec, y_par, atol=1e-5))
recurrent == parallel (UT transform): True
NoteKey Insight

The production kernels go one step further: they chunk this solve, computing the WY representation inside each chunk in parallel and carrying a single state matrix between chunks — the same parallel-in / recurrent-between structure as retention’s chunkwise form. But it is an optimization of the exact math above, not a different function. Recurrent for decoding, parallel (UT) for training, chunkwise for scale: three faces, one delta rule.

Interactive Exploration: Associative Recall at Scale

The overwrite demo used two keys. Does the advantage hold when the memory is full? demonstrate_recall stores n facts under orthonormal keys, updates all n with new values, then asks the memory to recall each one — sweeping n from 1 up to the state’s capacity. Watch the additive memory’s error stay stubbornly high (it always returns old + new) while the delta rule recalls the update essentially exactly, no matter how many facts you pack in:

TipTry This
  1. This is the associative-recall benchmark in miniature. The synthetic “store, update, recall” task here is exactly what the MAD and mechanistic evaluations that motivated DeltaNet measure on trained models — additive linear attention fails it, the delta rule passes.
  2. Orthonormal keys make it exact; realistic keys are harder. Here the keys are orthonormal, so the erase is perfect. With overlapping (learned) keys a single pass leaves some crosstalk — which is why real DeltaNet stacks many layers and learns the keys, and why gated DeltaNet adds a forget gate on top (see below).

Composing the Two Axes: Gated DeltaNet

This module has, quietly, laid out a small design space with two independent axes:

  • How the state forgets — the gate A_t. Linear attention keeps everything (A_t = 1), retention decays geometrically (A_t = \gamma), and the selective form lets the model choose per token (A_t = f(x_t)), all through the one unified_recurrence.
  • How the state writes — the update. Everything up to the delta rule wrote additively (+\,k_t v_t^\top); the delta rule made the write error-correcting (I - \beta_t k_t k_t^\top), so it can overwrite a slot instead of piling onto it.

Until now we have moved along one axis at a time. Gated DeltaNet (Yang, Kautz & Hatamizadeh, 2024) takes the obvious next step and turns both knobs at once: an input-dependent forget gate \alpha_t on top of the delta write.

Why bother? Because the two mechanisms erase different things. The delta rule’s I - \beta_t k_t k_t^\top is a targeted eraser: it clears the value stored under one key, the one being written right now. But it is helpless against memory it is not currently re-addressing — a fact stored under some other key just sits there forever. Real sequences need the other kind of erase too: a global “the topic just changed, let most of this go” — and, at the limit, a hard reset at a document boundary. That is exactly what a scalar forget gate \alpha_t \in (0,1) does: it shrinks the whole state every step, independent of content. Step through the combined write — decay first, then error-correct against the decayed memory:

The Math: A Forget Gate on the Delta Write

Multiply the delta-rule transition by the scalar \alpha_t and you have the whole model (paper Eq. 8, in this module’s S = (d \times d_v), o_t = q_t S_t form):

S_t = \alpha_t\big(I - \beta_t\, k_t k_t^\top\big)\, S_{t-1} + \beta_t\, k_t v_t^\top , \qquad \alpha_t \in (0,1),\ \beta_t \in (0,1).

It sits at the meeting point of two lines this book has drawn, and degenerates cleanly to each:

Set Recurrence You get
\alpha_t = 1 (I - \beta_t k_t k_t^\top)S_{t-1} + \beta_t k_t v_t^\top DeltaNet — forget gate off
drop -\beta k k^\top \alpha_t S_{t-1} + \beta_t k_t v_t^\top Mamba-2 / gated linear attention — additive gated write

So Gated DeltaNet is literally Mamba-2’s scalar-gated recurrence with its additive write swapped for the error-correcting one — the paper’s own framing, “improving Mamba2 with the delta rule.” The stability rules are unchanged and compose: keys are L2-normalized so I - \beta k k^\top never expands the state (eigenvalues in [1-\beta, 1]), and \alpha_t \in (0,1) only ever shrinks it — the same spectral rule as retention’s \gamma and the SSM’s \bar A. The \alpha_t=1 reduction is exact, and worth seeing run:

import torch
from linear_attention import gated_delta_rule_recurrent, delta_rule_recurrent

torch.manual_seed(0)
Q = torch.randn(6, 8); K = torch.randn(6, 8); V = torch.randn(6, 8)
beta = torch.full((6,), 0.7)

ones = torch.ones(6)                                  # forget gate fully open
gated_off = gated_delta_rule_recurrent(Q, K, V, ones, beta)
print("α≡1 recovers plain DeltaNet:",
      bool(torch.allclose(gated_off, delta_rule_recurrent(Q, K, V, beta), atol=1e-5)))
α≡1 recovers plain DeltaNet: True

Code: Gating, Parallelized for Free

The forget gate looks like it should demand a brand-new parallel algorithm — the delta rule already needed the UT transform, and now there is a per-token decay on top. It does not. A single change of variables folds the gate away.

Let b_t = \prod_{s \le t}\alpha_s be the running product of the gates, and rescale the state \tilde S_t = S_t / b_t. Dividing the gated recurrence by b_t cancels every \alpha:

\tilde S_t = \big(I - \beta_t k_t k_t^\top\big)\,\tilde S_{t-1} + \beta_t\, k_t\, \tilde v_t^\top , \qquad \tilde v_t = v_t / b_t .

That is the ungated delta rule on rescaled values \tilde v_t. Solve it with the UT transform we already built, then scale the output back up by b_t (since o_t = q_t S_t = b_t\,(q_t \tilde S_t)):

\texttt{gated}(Q,K,V,\alpha,\beta) \;=\; b \odot \texttt{delta\_rule\_parallel}(Q,\,K,\,V/b,\,\beta).

Gating adds no new parallel machinery — just a diagonal rescale around the delta solve we already have. This is the un-chunked heart of the paper’s chunkwise algorithm (which resets b per chunk and carries a state between chunks, so the /b never underflows). linear_attention.py implements both faces and they agree bit-for-bit:

from linear_attention import gated_delta_rule_parallel, GatedDeltaNetState

alpha = torch.rand(6) * 0.3 + 0.7                      # per-token forget gates in [0.7, 1.0)
y_rec = gated_delta_rule_recurrent(Q, K, V, alpha, beta)
y_par = gated_delta_rule_parallel(Q, K, V, alpha, beta)   # = b ⊙ delta_parallel(·, V/b)
print("recurrent == parallel (change of variables):",
      bool(torch.allclose(y_rec, y_par, atol=1e-4)))

state = GatedDeltaNetState(d=8, d_v=8)                 # O(1)-per-token decode
ys = torch.stack([state.step(Q[i], K[i], V[i], alpha[i], beta[i]) for i in range(6)])
print("incremental decode == batch recurrence:",
      bool(torch.allclose(ys, y_rec, atol=1e-5)))
recurrent == parallel (change of variables): True
incremental decode == batch recurrence: True

The recurrent dots land exactly on the parallel line — the gate did not cost us a new algorithm, only a reweighting of the same one:

What the Gate Buys: Global Forgetting

The same forget gate — a learned scalar that resets the running past — carries over to full softmax attention too: m09’s Forgetting Attention (FoX) adds \log\prod f_l as a decay bias on the scores, and a near-zero gate resets the context there just as \alpha\to0 does here.

Here is the capability the forget gate adds that the delta rule cannot provide. Store several facts under different keys, then hit a single reset token with \alpha = 0, then ask for the facts back. demonstrate_gated_reset runs exactly that on orthonormal keys, and compares Gated DeltaNet (with the reset) against plain DeltaNet (no gate at all):

from linear_attention import demonstrate_gated_reset

reset = demonstrate_gated_reset(n=5, d=16, seed=0)
print("gated: recall error after α=0 reset:", reset["gated_error"], "(≈ value norm — forgotten)")
print("delta: recall error (no forget gate):", reset["delta_error"], "(≈ 0 — still remembered)")
gated: recall error after α=0 reset: 3.62252 (≈ value norm — forgotten)
delta: recall error (no forget gate): 0.0 (≈ 0 — still remembered)

One \alpha = 0 step wipes the entire state, so every fact is gone (the readout is zero, and the recall error equals the values’ own norm). Plain DeltaNet has no such lever: it only ever overwrites the key it is currently writing, so all five facts survive untouched. This is the exact complement of the overwrite demo earlier — targeted erase (one key, via \beta) versus global erase (everything, via \alpha):

NoteKey Insight

The delta rule and the forget gate are orthogonal knobs on orthogonal problems. \beta (the delta write) answers “which one thing should I overwrite?” — precise, content-based, local to a key. \alpha (the forget gate) answers “how much of everything should I let go?” — content-free, global, and able to reset. Mamba-2 had \alpha but only an additive write; DeltaNet had the precise write but no \alpha. Gated DeltaNet is the corner of the design space where you get both.

Interactive Exploration: The Memory Horizon

A constant forget gate \alpha has a clean interpretation. Store a fact under key k_0, then write a stream of unrelated facts under other keys. Because those writes never re-address k_0, the delta rule leaves its slot alone — only the forget gate touches it, multiplying it by \alpha every step. So a fact you never overwrite decays geometrically:

\text{recall}(\text{age}) = \alpha^{\text{age}} , \qquad \text{memory horizon} \approx \frac{1}{1-\alpha}.

At \alpha = 1 the curve is flat — pure DeltaNet, an unbounded memory that forgets nothing. Drop \alpha and you set a tunable timescale for how long a fact survives if nothing refreshes it. Drive the gate and watch the horizon move (the dots are the Python-verified demonstrate_gated_horizon reference points; the line is \alpha^{\text{age}}):

TipTry This
  1. Push α to 1.0. The curve goes flat: pure DeltaNet, infinite horizon. This is why DeltaNet is superb at exact long-range recall but has no way to bound how much stale context it carries — the gate is precisely that missing knob.
  2. Push α to 0.5. The horizon collapses to ~2 tokens — the model becomes almost purely local, like a tiny sliding window, without any attention mask.
  3. This is one fixed α; the real model makes it input-dependent. A learned \alpha_t = \sigma(\cdot) lets the sequence hold α≈1 while a fact matters, then snap α→0 to forget at a boundary — a content-driven horizon, not a fixed one.

Training the Delta Rule: The Chunkwise Algorithm

Everything so far has two faces, and neither is what a GPU actually runs. The recurrent face steps token by token — O(L) serial work, a death sentence on hardware that wants to do thousands of things at once. The parallel UT transform solves one L\times L triangular system — beautifully parallel, but O(L^2) memory and compute, which blows up on long sequences (the very cost linear attention was meant to escape). RetNet already showed the way out for scalar decay — its chunkwise form is parallel within chunks and recurrent between them. The delta rule takes the same shape; it just has to carry its state through the UT transform instead of a plain outer-product sum.

Split the sequence into chunks of B tokens. Solve each chunk in parallel with a small B\times B UT transform, and carry one fixed-size state S \in \mathbb{R}^{d\times d_v} across the chunk boundaries. The cost lands in between: O(L\cdot B) work and only O(L/B) serial steps. B=1 is the recurrence; B\ge L is the one-shot solve; every B in between trades serial depth for parallel width — the knob real kernels tune to the hardware.

The Math: Carry a State Between Chunks

Inside a chunk, the delta rule’s only new ingredient is the incoming state. When it error-corrects token t, the memory it reads is not empty — it holds the carried S_{\text{in}} from every earlier chunk, plus this chunk’s own earlier writes:

S_{t-1} = S_{\text{in}} + \sum_{i<t} k_i u_i^\top , \qquad u_t = \beta_t\big(v_t - k_t^\top S_{t-1}\big).

Substituting S_{t-1} gives u_t = \beta_t\big(v_t - k_t^\top S_{\text{in}} - \sum_{i<t}(k_t\!\cdot\!k_i)\,u_i\big) — exactly the same unit-lower-triangular solve as delta_rule_parallel, but on values corrected by the carried state, V' = V - K S_{\text{in}}:

U = (I + T)^{-1}\operatorname{diag}(\beta)\,\underbrace{(V - K S_{\text{in}})}_{V'} , \qquad O = \underbrace{Q S_{\text{in}}}_{\text{read earlier chunks}} + \underbrace{\operatorname{tril}(Q K^\top)\,U}_{\text{read this chunk}} , \qquad S_{\text{in}} \leftarrow S_{\text{in}} + K^\top U .

Three pieces: correct the values by what the carried memory already answers for each key, run the same UT solve we built for one chunk, and read out a cross-chunk term (Q S_{\text{in}}, the query against the carried state) plus the intra-chunk term. Then fold this chunk’s writes K^\top U into S_{\text{in}} and move on. With one chunk, S_{\text{in}} = 0 and it collapses back to delta_rule_parallel.

Code: The Delta Rule, Chunked

delta_rule_chunkwise is the diagram above, ten lines of it. It reuses the exact UT solve from the parallel form and the carry pattern from retention_chunkwise — the only addition is the V - K S_{\text{in}} correction. It is bit-for-bit equal to the recurrence at every chunk size:

from linear_attention import delta_rule_chunkwise, delta_rule_parallel

torch.manual_seed(0)
Qc = torch.randn(12, 8); Kc = torch.randn(12, 8); Vc = torch.randn(12, 8)
beta_c = torch.rand(12)
y_rec = delta_rule_recurrent(Qc, Kc, Vc, beta_c)

for B in (1, 3, 5, 12):
    y_chunk = delta_rule_chunkwise(Qc, Kc, Vc, beta_c, chunk_size=B)
    print(f"chunk_size={B:2d}  ({-(-12 // B)} chunks)  max|chunk − recurrent| = "
          f"{float((y_chunk - y_rec).abs().max()):.1e}")
chunk_size= 1  (12 chunks)  max|chunk − recurrent| = 7.2e-07
chunk_size= 3  (4 chunks)  max|chunk − recurrent| = 6.0e-07
chunk_size= 5  (3 chunks)  max|chunk − recurrent| = 4.8e-07
chunk_size=12  (1 chunks)  max|chunk − recurrent| = 6.0e-07

Every chunk size computes the same function. The chunk size is a pure performance knob — it changes how the work is scheduled, never the answer.

Gating Forces the Chunk: A Local Decay

For the gated delta rule, chunking is not just about speed — it is the only way to compute the parallel form at all on a long sequence. Recall the change of variables that gave gating “for free”: \tilde v_t = v_t / b_t with b_t = \prod_{s\le t}\alpha_s. With \alpha < 1, that cumulative product decays geometrically — and in float32 it hits zero after only a few hundred tokens. Once b_t = 0, v_t / b_t = \infty, and the whole output is NaN. The book flagged this twice; here is the fix.

The chunkwise form makes the decay local: reset b to 1 at every chunk boundary. Inside a chunk of B tokens the local product P_t = \prod_{s\le t}\alpha_s divides by at most B gates, not L — bounded and safe. It rides on top of the same state carry, with two extra rescales: divide the chunk’s values by P going in, and multiply the read-out and the carried state by P coming out.

\tilde V = V / P ,\quad U = (I+T)^{-1}\operatorname{diag}(\beta)\,(\tilde V - K S_{\text{in}}) ,\quad O = P \odot\!\big(Q S_{\text{in}} + \operatorname{tril}(Q K^\top)U\big) ,\quad S_{\text{in}} \leftarrow P_B\,(S_{\text{in}} + K^\top U) .

Code: The Numerically Stable Form

gated_delta_rule_chunkwise closes both gaps. Two reductions pin it to what we already trust — \alpha \equiv 1 is the ungated chunkwise DeltaNet, and one big chunk is the global rescale — and it matches the recurrence bit-for-bit:

from linear_attention import gated_delta_rule_chunkwise

alpha_c = torch.rand(12) * 0.3 + 0.7                # per-token forget gates in [0.7, 1.0)
y_grec = gated_delta_rule_recurrent(Qc, Kc, Vc, alpha_c, beta_c)
y_gchunk = gated_delta_rule_chunkwise(Qc, Kc, Vc, alpha_c, beta_c, chunk_size=4)
print("chunkwise == recurrent:",
      bool(torch.allclose(y_gchunk, y_grec, atol=1e-4)))

ones = torch.ones(12)                               # α ≡ 1 ⇒ gate off ⇒ plain DeltaNet
print("α ≡ 1 == ungated chunkwise:",
      bool(torch.allclose(gated_delta_rule_chunkwise(Qc, Kc, Vc, ones, beta_c, chunk_size=4),
                          delta_rule_chunkwise(Qc, Kc, Vc, beta_c, chunk_size=4), atol=1e-5)))
chunkwise == recurrent: True
α ≡ 1 == ungated chunkwise: True

Now the payoff. On a long, strongly-decaying sequence in float32 — the training regime — the global rescale underflows to NaN while the chunkwise form sails through, still matching the recurrence to \sim\!10^{-6}:

torch.manual_seed(0)
L = 768
Ql = torch.randn(L, 8); Kl = torch.randn(L, 8); Vl = torch.randn(L, 8)
al = torch.full((L,), 0.8); bl = torch.rand(L)     # α^768 ≈ 4e-75, far below float32's 1.2e-38

ref = gated_delta_rule_recurrent(Ql, Kl, Vl, al, bl)
glob = gated_delta_rule_parallel(Ql, Kl, Vl, al, bl)          # divides by the GLOBAL b_t
chunk = gated_delta_rule_chunkwise(Ql, Kl, Vl, al, bl, chunk_size=64)  # local, per-chunk b

print(f"global rescale  → any NaN? {bool(torch.isnan(glob).any())}")
print(f"chunkwise (B=64)→ any NaN? {bool(torch.isnan(chunk).any())}  "
      f"max|chunk − recurrent| = {float((chunk - ref).abs().max()):.1e}")
global rescale  → any NaN? True
chunkwise (B=64)→ any NaN? False  max|chunk − recurrent| = 9.5e-07

The sweep below drives the sequence length and shows exactly where the cliff falls: the global form’s error stays tiny until \alpha^{L} slips under float32’s smallest normal (the dashed line), then vanishes into NaN; the chunkwise error never leaves the floor.

TipTry This
  1. Read the cliff. The green (chunkwise) line stays pinned near 10^{-6} across every length. The red (global) line tracks it — then simply stops: past that length its output is all NaN, so there is no error to plot.
  2. Shrink the chunk, push the cliff away. A smaller chunk_size multiplies fewer gates before dividing, so it tolerates stronger decay. This is the real trade-off a kernel tunes: small chunks are more stable and more serial, big chunks are faster and closer to the underflow edge.
  3. Turn the gate off. Set alpha = 1.0 everywhere. The cumulative product is always 1, nothing underflows, and both forms agree at all lengths — the instability is entirely a gated phenomenon.

Sharpening the Transition: DeltaProduct

Every mixer in this module — linear attention, retention, the SSM, DeltaNet, gated DeltaNet — writes its per-token state transition as an edit of the identity of rank at most one. Retention scales (A_t = \gamma I); DeltaNet subtracts a single outer product (A_t = I - \beta_t k_t k_t^\top). That rank-1 ceiling is not cosmetic — it is a hard limit on what the memory can track.

Here is the tell. With a unit key and the full write strength \beta = 2, the delta transition

I - 2\, k_t k_t^\top

is a Householder reflection: orthogonal, and it flips orientation (\det = -1). A reflection can send a vector to the other side of a mirror, but no single reflection is a rotation — and a rotation (\det = +1) is exactly what you need to carry a running angle, or a running permutation, forward through the sequence. A rank-1-per-token memory literally cannot hold a rotating state: it is stuck flipping.

NoteKey Insight

The delta rule’s per-token transition is one reflection. One reflection flips; it never rotates. But the product of two reflections is a rotation. So if you let a token take two delta steps instead of one, its transition becomes a product of two Householders — and the memory can, for the first time, rotate its state. That is the whole idea of DeltaProduct.

DeltaProduct (Siems et al., NeurIPS 2025) takes this literally. Reinterpret the delta rule as one step of online gradient descent on the associative-recall loss (it is: S \leftarrow S + \beta_t k_t(v_t - k_t^\top S) is a gradient step). Then take n_h steps per token instead of one. Each token i emits n_h sub-keys, sub-values, and sub-betas (k_{i,j}, v_{i,j}, \beta_{i,j}) from n_h separate projections, and the state runs a micro-recurrence before the next real token arrives:

H_{i,j} = \big(I - \beta_{i,j}\, k_{i,j} k_{i,j}^\top\big)\, H_{i,j-1} + \beta_{i,j}\, k_{i,j} v_{i,j}^\top, \qquad H_{i,0} = H_{i-1},\quad j = 1 \dots n_h,

reading out once, at the end: o_i = q_i H_{i, n_h}. Multiply the n_h transition factors together and the per-token transition is now a product of n_h generalized Householder transformations — a “diagonal plus rank-n_h” matrix:

A(x_i) = \prod_{j=1}^{n_h} \big(I - \beta_{i,j}\, k_{i,j} k_{i,j}^\top\big).

Two consequences fall straight out of the definition:

Set You get
n_h = 1 DeltaNet exactly — one Householder per token, the rank-1 transition.
n_h steps A rank-n_h transition. With \beta = 2 (reflections), \det A = (-1)^{n_h}: an even n_h can rotate.

And there is nothing new to implement: because each real token is just n_h ordinary delta steps run back-to-back, DeltaProduct is exactly DeltaNet run on a stretched micro-sequence of length n_h \cdot T, read out once per real token. The code you already have does all the work.

The Math: Why Two Reflections Rotate

The stability rule survives the change. Each factor I - \beta k k^\top (unit key) has eigenvalues 1 (on the hyperplane orthogonal to k) and 1 - \beta (along k). DeltaProduct lets \beta \in [0, 2] — the extended range — via \beta = 2\, \sigma(\cdot), so 1 - \beta \in [-1, 1]. The state is still non-expansive: the eigenvalue can reach -1 (a clean reflection) but never leaves the unit disk. You buy reflections without buying instability.

The reflection-to-rotation fact is elementary and exact. A Householder H(k) = I - 2kk^\top mirrors across the hyperplane orthogonal to k; in 2D, reflecting across a line then across another rotates by twice the angle between the lines. Pick the two mirror keys and you hit any target rotation R(\theta) on the nose:

One reflection:  det = -1.0   (flips orientation)
Two reflections: det = 1.0   (a rotation)
Reaches the exact target rotation? True (max error 6.0e-06)

The single Householder is orientation-reversing (\det = -1) and can never be a rotation; the product of two lands exactly on the 100° rotation matrix.

The same argument, made with integers, is the state-tracking headline. A transposition — swap two coordinates — is itself a Householder: with \hat u = (e_i - e_j)/\sqrt2 and \beta = 2,

I - 2\,\hat u \hat u^\top \;=\; I - (e_i - e_j)(e_i - e_j)^\top

is precisely the permutation matrix that swaps e_i \leftrightarrow e_j. A single transposition is an odd permutation (\det = -1) — so a one-step delta memory is stuck being odd. But a 3-cycle (0\!\to\!1\!\to\!2\!\to\!0) is an even permutation, and it is the product of two transpositions — reachable only when a token takes two delta steps:

One transposition (1 step):   det = -1.0   → odd permutation
3-cycle = two transpositions: det = 1.0   → even permutation
Product reproduces the exact 3-cycle matrix? True (order 3)

This is a group word problem: a single delta step per layer can only compose permutations of the same parity, so it cannot track membership in the alternating group; DeltaProduct with n_h steps reaches permutations of up to n_h + 1 elements in one layer. State tracking is the property that lets a model follow “who holds the ball now” across a long chain of swaps — the thing pure linear attention provably cannot do.

Code: Several Steps per Token

delta_product.py builds all of this on top of the delta rule you already have. The recurrence just loops the delta write n_h times per token; the transition helper multiplies the n_h Householders. Both agree with DeltaNet at n_h = 1, and with DeltaNet on the flattened micro-sequence for any n_h:

n_h = 1 matches DeltaNet exactly?           True (error 0.0e+00)
DeltaProduct == DeltaNet on the n_h·T seq?  True (error 0.0e+00)
Per-token transition det for n_h=1..4:      [-1.0, 1.0, -1.0, 1.0]
                                            (-1)^n_h: odd n_h flips, even n_h rotates

Watch the determinant alternate — -1, +1, -1, +1 — as n_h grows: every extra reflection flips orientation, and every pair buys a rotation the previous parity could not express.

TipTry This
  1. Set n_h = 2 and feed it a rotation task. Build the two mirror keys with delta_product_transition and confirm the per-token transition is a rotation (torch.det ≈ +1, orthogonal). A DeltaNet layer (n_h = 1) cannot produce it.
  2. Flatten and compare. Call flatten_micro_sequence, run delta_rule_recurrent on the result, read every n_h-th output — it is DeltaProduct, bit-for-bit. Convince yourself there is no new algorithm here.
  3. Clamp \beta to [0,1]. Now no factor can reach a reflection (1-\beta \ge 0), the transition determinants stay non-negative, and the rotation win vanishes — the reflection range is what buys the expressivity, not just the extra steps.

Interactive Exploration: Two Reflections, One Rotation

Drive the composition directly. A probe vector starts on the unit circle; step 1 reflects it across the first mirror, step 2 reflects the result across the second. The two flips compose into a single clean rotation — the net move a DeltaProduct token makes to its state, and the one a single delta step cannot.

Now trade the microscope for the dial. Each extra delta step costs one more Householder per token — the state work grows linearly in n_h — and buys the ability to track permutations of up to n_h + 1 elements. Slide n_h and watch the orientation flip and the reachable group grow against the rising cost:

NoteKey Insight

DeltaProduct adds no new machinery — it is the delta rule you built, run n_h times per token. What the extra steps buy is purely representational: a rank-n_h transition that, with reflections, can rotate and permute. It is the same lesson this whole module teaches, taken one turn further — a sequence mixer is defined by how its state forgets and how it writes, and DeltaProduct simply lets the write compose several times before the next token, trading a linear slice of compute for a transition the rank-1 memories cannot express.

RWKV: A Named RNN That Trains in Parallel

Every mechanism so far has been a building block. RWKV (Peng et al., 2023 — “Reinventing RNNs for the Transformer Era”) is a whole architecture built from these blocks that was trained at scale — dense RNNs up to 14B parameters, matching similarly-sized Transformers at O(L) cost and O(1) decode. It is the most widely deployed of the linear-attention-line models, so it is worth building its sequence mixer, the time-mixing block, end to end.

RWKV makes one radical simplification to everything above: it deletes the query. In every attention variant, a query q_t reads the state; the score from token t to token i is a q_t\cdot k_i interaction. RWKV’s WKV operator drops that. The weight token t gives token i depends only on their time gap and i’s key — never on t. It is attention whose entire “score matrix” is fixed by position and key, with no query in it at all.

So what decides how much each channel reads out? A separate receptance gate: the output is \sigma(r_t)\odot \text{wkv}_t, a per-channel sigmoid in (0,1) that plays the query’s old role as a cheap gate. “RWKV” is the four letters of the block: Receptance (the gate), Weight (the time-decay w), Key, Value.

One more idea appears here that the family sections didn’t need: a token shift. Before projecting, RWKV mixes each input with the one before it — \mu\odot x_t + (1-\mu)\odot x_{t-1}, a per-channel blend. It is the cheapest possible “look one token back” (a length-2 depthwise causal convolution, exactly the kind of local mixing m19’s Mamba block used), and it lets a single time-mix layer see a two-token window before the recurrence even starts.

The Math: The WKV Operator

Here is the whole mixer, the current-token part first. The output for token t is

\text{wkv}_t = \frac{\displaystyle\sum_{i<t} e^{-(t-1-i)\,w + k_i}\, v_i \;+\; e^{u + k_t}\, v_t} {\displaystyle\sum_{i<t} e^{-(t-1-i)\,w + k_i} \;+\; e^{u + k_t}} ,

a softmax over time. Each source i gets a log-weight k_i reduced by the decay w for every step of distance; the current token gets u + k_t. Two per-channel vectors control it:

  • w \ge 0 — the time decay. The weight of a token falls by a factor e^{-w} per step, so the memory horizon is \approx 1/w. Large w = short memory, small w = long memory. This is retention’s \gamma, written \gamma = e^{-w}.
  • u — the bonus. It is a separate weight for the current token so it does not have to compete on the same decay curve as the history. Push u up and the model “attends to now”; push it down and it leans on the past.

Two facts fall straight out of the formula. First, the immediately-preceding token (i = t-1) has decay exponent -(t-1-(t-1))w = 0 — it is undecayed, so u is what distinguishes the current token from the most recent one. Second, the very first token has no history at all, so

\text{wkv}_0 = \frac{e^{u+k_0} v_0}{e^{u+k_0}} = v_0

exactly, for any w, u, k. A token with nothing before it returns its own value.

Code: The Two Faces of WKV

Like every mechanism in this module, WKV has a parallel training form (the sum above, wkv_parallel) and an exactly-equal recurrence (wkv_recurrent) that decodes in O(1). The recurrence carries a decayed numerator and denominator:

\text{wkv}_t = \frac{\text{num} + e^{u+k_t} v_t}{\text{den} + e^{u+k_t}}, \qquad \text{num} \leftarrow e^{-w}\,\text{num} + e^{k_t} v_t, \quad \text{den} \leftarrow e^{-w}\,\text{den} + e^{k_t} ,

starting from \text{num} = \text{den} = 0. The read-out uses the bonus u; the state carries the token forward without it, so from the next step on the token decays like any other history. Both are in linear_attention.py:

from linear_attention import wkv_parallel, wkv_recurrent, demonstrate_wkv_faces

f = demonstrate_wkv_faces(length=12, d=8)
print("parallel WKV == recurrent WKV:", f["agree"], f"(max diff {f['max_abs_diff']:.1e})")
print("wkv_0 == v_0 exactly:          ", f["first_token_is_value"])
parallel WKV == recurrent WKV: True (max diff 2.4e-07)
wkv_0 == v_0 exactly:           True

The parallel form is the transformer face (train over a whole sequence at once); the recurrence is the RNN face (constant memory per token). Same function — that is RWKV’s entire promise, made runnable.

NoteKey Insight

WKV is retention with the query deleted and a normalizer kept. Retention’s state is S_n = \gamma S_{n-1} + k_n^\top v_n read by a query q_n; RWKV’s is \text{num}_t = e^{-w}\text{num}_{t-1} + e^{k_t} v_t (a scalar e^{k_t} key weight, per channel, no outer product) read by a gate \sigma(r_t) instead of q_t, and divided by a matching denominator so it stays a weighted average. On the module’s one recurrence S_t = A_t S_{t-1} + k_t^\top v_t, RWKV sits at A_t = e^{-w} (a fixed per-channel decay, like retention) but swaps the read from a query to a gate.

Token Shift: One Look Back, for Free

The other new piece is the input mix. token_shift blends each channel of the current token with the previous one — the identity at \mu = 1, the pure previous token at \mu = 0:

import torch
from linear_attention import token_shift

x = torch.tensor([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
print("μ = 1 (identity):\n", token_shift(x, torch.ones(2)))
print("μ = 0 (previous):\n", token_shift(x, torch.zeros(2)))   # zeros at t=0
μ = 1 (identity):
 tensor([[1., 2.],
        [3., 4.],
        [5., 6.]])
μ = 0 (previous):
 tensor([[0., 0.],
        [1., 2.],
        [3., 4.]])

Drive \mu and watch a channel slide from “current token” to “previous token”:

The Full Block, and Its O(1) Decode

RWKVTimeMix wires it all together: token-shift the input three ways, project to r, k, v, run WKV, gate with \sigma(r), and project out. Its .forward() is the parallel training pass; its .step() drives an RWKVState one token at a time. The headline is that the two agree and the decode state never grows:

from linear_attention import demonstrate_rwkv_decode

d = demonstrate_rwkv_decode(length=32, d=16)
print("incremental decode == parallel forward:", d["agree"],
      f"(max diff {d['max_abs_diff']:.1e})")
print("RWKV decode state:", d["state_bytes"], "bytes  (constant in length)")
print("softmax KV cache at L=32:", d["kv_cache_bytes_equiv"], "bytes  (grows with L)")
incremental decode == parallel forward: True (max diff 2.1e-07)
RWKV decode state: 192 bytes  (constant in length)
softmax KV cache at L=32: 2048 bytes  (grows with L)

The RWKV state is a fixed 3d numbers — prev_x, num, den — no matter how long the context. A softmax KV cache, by contrast, grows as 2Ld: at long context it is the memory wall RWKV was built to remove. This is “reinventing RNNs for the transformer era” in one line: train with .forward(), deploy with .step().

Interactive Exploration: WKV as Attention Over Time

Because there is no query, WKV’s weighting is easy to see whole. For the last token in a length-12 window (flat keys, so only distance and the bonus matter), the bar below is the weight it places on every earlier token plus itself. Turn up the decay w and the memory collapses onto the most recent tokens; turn up the bonus u and the current token spikes above the history — the two knobs, made visible (the dots are the Python-verified demonstrate_wkv_horizon reference points):

TipTry This
  1. Push w toward 2. The weight piles onto positions 10–11: a short memory, almost a sliding window of a couple of tokens — a fast-decaying channel.
  2. Push w toward 0. The bars flatten: every past token counts equally — a long-memory channel that averages the whole context.
  3. Raise u. The last bar (the current token) shoots up while the history shrinks proportionally. A real RWKV layer has one w and u per channel, so different channels are simultaneously short- and long-memory — a whole spectrum of horizons in a single layer.
NoteKey Insight

RWKV-4 built here keeps a scalar state per channel (num/den are vectors, not matrices). Its successors RWKV-5/6 (“Eagle”/“Finch”, 2024) promote the state to a matrix and make the decay data-dependent — which lands them squarely in the gated-linear-attention / DeltaNet design space this module built earlier. The lineage is one story: a query-free, time-decayed running state, made progressively more expressive.

Common Pitfalls

When building linear/recurrent attention, watch out for:

  1. A non-positive feature map. If \varphi can go negative, the normalizer \varphi(q_i)^\top Z_i can hit zero or flip sign and the output explodes. Keep \varphi non-negative (\text{elu}+1, or \text{ReLU}, or a softmax-feature map).
  2. Forgetting the normalizer in linear attention. Linear attention divides by \varphi(q_i)^\top Z_i; retention does not (it uses a GroupNorm downstream instead). Mixing them up gives the wrong scale.
  3. A decay \gamma \ge 1. Retention needs \gamma \in (0,1]; \gamma>1 makes \gamma^{n-m} grow with distance and the state diverges. (Same stability rule as the SSM’s \bar{A}\in(0,1) in m19.)
  4. Chunk-boundary bookkeeping. The cross-chunk term must be scaled by \gamma^{t+1} and the carried state updated as R \leftarrow \gamma^B R + \sum_s \gamma^{B-1-s}k_s^\top v_s. Off-by-one in these exponents silently breaks the equivalence — test chunkwise against parallel.
  5. Expecting linear attention to match softmax exactly. It is a different function, not an approximation of softmax. It trades some expressivity (the softmax’s sharp, data-dependent focus) for linear cost; the research since 2020 is largely about closing that quality gap.
  6. Confusing the state size with the SSM’s. Linear attention’s state is d \times d; an SSM’s is d \times N with N small (16). Both are constant in L, but linear attention’s is bigger — the price of using full key/value vectors as the “state coordinates”.
  7. Reading GLA’s gate as a scalar. The whole point of GLA over retention is that \alpha_t is a vector — one decay per key dimension — so the state can hold one feature while flushing another in the same step. Collapse it to a single number and you are back to retention; no scalar \gamma can reproduce a genuine per-dimension gate (test: sweep \gamma and watch the best match stay far from the GLA output).
  8. Running GLA’s naive parallel form at length. The parallel face rescales by the cumulative gate b_t = \prod\alpha_s; since each \alpha<1, b_t\to 0 and the k/b_t term overflows once \alpha^{-L} passes float32’s max (\approx3.4\times 10^{38}) — NaN after a few hundred tokens. It is correct only for short spans; train with gla_chunkwise, which resets b every chunk and has no cliff (the same necessity as gated DeltaNet’s chunked form, below).
  9. Un-normalized keys in the delta rule. The transition I - \beta k k^\top is non-expanding only when \lVert k \rVert = 1. Skip the key normalization and an eigenvalue can leave [0,1], so the state grows without bound over a long sequence. Normalize keys (the same spectral rule as the SSM’s \bar A \in (0,1)).
  10. Expecting one delta pass to give perfect recall on overlapping keys. With orthonormal keys the erase is exact; with realistic, overlapping keys a single layer leaves residual crosstalk. The delta rule reduces the associative-memory error, it does not zero it in one pass — real models stack many layers and learn the keys, and gated DeltaNet adds a forget gate to reclaim capacity over time.
  11. Dividing by the global b_t = \prod\alpha_s in gated DeltaNet. The change-of-variables parallel form (gated_delta_rule_parallel) rescales values by 1/b_t. With \alpha<1 over a long sequence b_t underflows toward 0 and v_t/b_t overflows — in float32 the output becomes all NaN after a few hundred tokens. Use it only for the short sequences it is shown on; for anything long reach for gated_delta_rule_chunkwise, which resets b per chunk and carries a state, staying finite and exact where the global form fails.
  12. Reading the forget gate as the delta rule’s job (or vice-versa). The gate \alpha forgets everything (global, content-free); the delta write \beta overwrites one key (local, content-based). They are orthogonal — a small \alpha will not sharpen recall, and a large \beta will not bound how much stale context you carry. Reach for the one that matches the problem.
  13. Expecting more DeltaProduct steps for free. Each of the n_h steps is a full delta write, so the per-token state work scales linearly in n_h — a rank-3 transition costs 3× a DeltaNet layer. The steps sharpen the transition; they are not a free lunch, and n_h = 2 or 3 is where the papers live.
  14. Clamping \beta to [0,1] and still expecting rotations. The reflection that makes two steps rotate needs \beta = 2 (eigenvalue 1-\beta = -1). Keep \beta \in [0,1] and every factor has non-negative determinant — the product can contract and overwrite but never rotate. DeltaProduct’s expressivity comes from the extended \beta \in [0,2] range (via 2\sigma), not from the extra steps alone.
  15. Overflowing WKV’s raw e^{k}. The clean WKV built here exponentiates keys directly, so large k (or a very long history) overflows the float32 numerator and denominator. Production RWKV kernels track a running max exponent and subtract it (a log-space softmax trick) to compute the same function safely. Keep key magnitudes modest in the from-scratch version, or fold in the running-max stabilization before feeding it real logits.
  16. Dropping the bonus u into the carried state. The current token reads out with the bonus e^{u+k_t}, but it must be carried forward without it (e^{k_t} only) — the bonus is a one-shot boost for “now”, not a permanent reweighting. Add u to the state update and every token stays over-weighted as it ages, breaking the parallel-vs-recurrent equality. Test the two faces against each other.

Exercises

Exercise 1: The two faces agree

from linear_attention import linear_attention_parallel, linear_attention_recurrent
import torch

# Generate random Q, K, V of your choice. Confirm the parallel and recurrent
# forms match, then explain in one sentence WHY they must: what property of the
# sum over j lets you regroup it into a running state?

# Your implementation here:

Exercise 2: Decay recovers linear attention

from linear_attention import retention_parallel
import torch

# Retention with gamma=1 (no decay) and the identity feature map is exactly
# unnormalized causal attention: (Q Kᵀ ⊙ tril) V. Build that expression by hand
# with torch.tril and confirm it equals retention_parallel(Q, K, V, gamma=1.0).
# Then set gamma=0.0 and describe, in words, what the output becomes.

# Your implementation here:

Exercise 3: You are the gate

from linear_attention import unified_recurrence
import torch

# Build Q, K, V (length 8). Using mode="gated", design a `gate` vector that keeps
# a HIGH gate (~0.95) only at positions 2 and 5 and a LOW gate (~0.1) elsewhere.
# Show the output differs from mode="retention" with a single fixed gamma — i.e.
# input-dependent selectivity buys something a constant decay cannot.

# Your implementation here:

Exercise 4: The gate is a vector

from linear_attention import gla_recurrent, gla_parallel, gla_chunkwise, retention_recurrent
import torch

# (a) Build Q, K, V with d_k = 2. Give dimension 0 a gate of 0.98 (holds) and
#     dimension 1 a gate of 0.2 (flushes), constant over time. Sweep a single
#     retention gamma over (0,1) and confirm NO gamma reproduces the GLA output —
#     the per-dimension gate is strictly more expressive than one scalar decay.
# (b) On a length-400 constant gate of 0.7, show gla_parallel returns NaN while
#     gla_chunkwise stays finite AND equals gla_recurrent. Explain in one sentence
#     which quantity overflowed.

# Your implementation here:

Exercise 5: Additive memory cannot overwrite

from linear_attention import delta_rule_recurrent, unified_recurrence
import torch

# Store value v1=[1,0] under key A=[1,0], then OVERWRITE with v2=[0,1] under the
# same key A, then probe key A (a third step with beta=0). Confirm the delta rule
# returns v2 while additive memory (unified_recurrence mode="linear") returns v1+v2.
# One line on WHY: what does the (I - beta k kᵀ) factor do that a plain outer
# product cannot?

# Your implementation here:

Exercise 6: The UT transform is the recurrence

from linear_attention import delta_rule_parallel, delta_rule_recurrent
import torch

# Generate random Q, K, V (length 10) and a random beta in [0,1]. Confirm
# delta_rule_parallel == delta_rule_recurrent. Then, from the definitions, explain
# in one sentence why the effective values u solve a *lower*-triangular system:
# which earlier tokens does u_t depend on, and which does it NOT?

# Your implementation here:

Exercise 7: Gating is free

from linear_attention import gated_delta_rule_recurrent, delta_rule_parallel
import torch

# Implement gated DeltaNet's PARALLEL face yourself from delta_rule_parallel, using
# ONLY the change of variables (no loop): b = cumprod(alpha); rescale the values by
# 1/b; call delta_rule_parallel; rescale the output by b. Confirm it matches
# gated_delta_rule_recurrent. Then set alpha to all-ones and explain, in one
# sentence, why your parallel form must collapse to the plain delta rule.

# Your implementation here:

Exercise 8: The chunk is where gating breaks

from linear_attention import (gated_delta_rule_recurrent, gated_delta_rule_parallel,
                              gated_delta_rule_chunkwise)
import torch

# On a LONG sequence with strong decay, watch the global rescale die and the
# chunkwise form survive. Build inputs of length 800 with alpha = 0.85 everywhere
# (float32). Compute all three: recurrent (ground truth), global-rescale parallel,
# and chunkwise (chunk_size=64). Show the global form contains NaN while the
# chunkwise form matches the recurrence to ~1e-6. Then find, by bisection, the
# smallest sequence length at which the global form first goes NaN, and check it
# against the prediction 0.85 ** L < 1.2e-38 (float32's smallest normal).

# Your experiment here:

Summary

Key takeaways:

  1. Linear attention is softmax attention minus the softmax. Replace \exp(q\cdot k) with a factored kernel \varphi(q)\cdot\varphi(k) and associativity turns the O(L^2) all-pairs sum into a running state S_i = \sum_{j\le i}\varphi(k_j)v_j^\top — a linear RNN hiding inside attention.
  2. It has two equal faces. A parallel masked-matmul form for training and a constant-memory recurrent form for decoding compute the same function.
  3. Retention adds a decay. S_n=\gamma S_{n-1}+k_n^\top v_n gives a \gamma^{n-m} decay mask (RetNet), which is the positional signal — no separate encoding — and comes in parallel, recurrent, and chunkwise forms, all provably identical.
  4. Chunkwise is the practical form. Parallel inside chunks, recurrent between them: GPU-parallel training that still scales linearly in sequence length.
  5. It is all one recurrence. S_t=A_tS_{t-1}+k_t^\top v_t specializes to linear attention (A_t=1), retention (A_t=\gamma), and a diagonal SSM / Mamba (A_t=f(x_t)). Transformer vs SSM collapses to is the gate input-dependent?
  6. The cost win is long-context. O(Ld^2) vs O(L^2d) and a fixed d\times d state vs a growing KV cache; the crossover is at L\approx d, so linear attention pays off exactly where softmax’s quadratic wall bites.
  7. Gated Linear Attention makes the forget gate learned and per-dimension. Replace retention’s single \gamma with a data-dependent vector \alpha_t=\sigma(x_tW_1W_2), so S_t=\operatorname{Diag}(\alpha_t)S_{t-1}+k_t^\top v_t decays each feature of the state at its own rate — hold one, flush another, in the same step. It keeps retention’s three faces (the scalar \gamma^{\,t} becomes the vector cumulative gate b_t=\prod\alpha), recovers retention at \alpha\equiv\gamma and linear attention at \alpha\equiv1, and needs the chunkwise reset: the parallel b_t underflows and its k/b_t rescale overflows float32 (\alpha^{-L}>3.4\times10^{38}) on a long sequence.
  8. The write is a second design axis. Every mixer here shares one gate axis (how the state forgets) and, until now, one write (an additive outer product). The delta rule changes the write to be error-correcting, turning the additive gate into a rank-1 matrix gate I - \beta_t k_t k_t^\top — a memory that can overwrite, not just accumulate. That single change is what lets DeltaNet solve associative recall that pure linear attention blurs.
  9. DeltaNet still has three faces. Recurrent (decode), the parallel UT transform U=(I+T)^{-1}\operatorname{diag}(\beta)V,\ O=\operatorname{tril}(QK^\top)U (train), and a chunked WY form (scale) — all the same function, exactly as retention did.
  10. Gated DeltaNet turns both knobs at once. Put an input-dependent forget gate on top of the delta write, S_t=\alpha_t(I-\beta_t k_t k_t^\top)S_{t-1}+\beta_t k_t v_t^\top, and you get Mamba-2’s gating with DeltaNet’s precise write. The forget gate \alpha erases globally (and resets at \alpha\to0); the delta write \beta overwrites one key — orthogonal erasers. And it costs no new parallel algorithm: a change of variables (\tilde v_t = v_t/\!\prod\alpha) folds the gate away and reuses the delta rule’s UT transform, so \alpha=1 recovers DeltaNet exactly.
  11. The chunkwise form is how it actually trains — and gating makes it mandatory. Split the sequence into chunks of B: a small B\times B UT solve inside each, one d\times d_v state carried between them (V'=V-KS_{\text{in}} in, S_{\text{in}}\!\leftarrow\!S_{\text{in}}+K^\top U out) — O(LB) work, O(L/B) serial, and bit-identical to the recurrence at every B. For gated DeltaNet it is not optional: the global rescale v_t/\!\prod\alpha underflows to NaN on a long decaying sequence in float32, so kernels reset the decay per chunk — a local b that divides by at most B gates keeps it finite.
  12. DeltaProduct sharpens the transition itself. Every mixer here edits the identity by rank \le 1 per token, and a rank-1 transition (one Householder) can only reflect, never rotate — so it cannot track a running rotation or permutation. Take n_h delta steps per token and the transition becomes a product of n_h Householders (A(x_i)=\prod_j(I-\beta_{i,j}k_{i,j}k_{i,j}^\top), “diagonal plus rank-n_h”); with the extended \beta\in[0,2], two reflections make a rotation and the memory can track permutations of up to n_h+1 elements. It is DeltaNet on a stretched n_h\!\cdot\!T sequence — no new algorithm, n_h\times the state work, and n_h=1 is DeltaNet.
  13. RWKV packages this into a deployed architecture. Its time-mixing block deletes the query: the WKV operator is a query-free softmax over time, \text{wkv}_t = \big(\sum_{i<t} e^{-(t-1-i)w+k_i}v_i + e^{u+k_t}v_t\big)/(\cdots), with a per-channel decay w (horizon \approx 1/w, i.e. retention’s \gamma=e^{-w}) and a current-token bonus u. A sigmoid receptance gate \sigma(r_t) does the query’s old job of deciding how much to read, and a token shift mixes in the previous input for free. Parallel form trains; the O(1) recurrence decodes — same function — so a 14B RNN trains like a transformer.

What’s Next

You have now seen both sides of the sub-quadratic coin — m19 approached it from control theory (SSMs), this module from attention (drop the softmax) — and watched them meet in one recurrence, then split along a second axis (how the state writes), and finally rejoin: gated DeltaNet turns both knobs at once, a global forget gate on top of the delta rule’s targeted overwrite, and gets it parallelized for free by folding the gate into the same UT transform.

The frontier keeps composing the same two axes. This module’s last section built DeltaProduct — several delta steps per token, a rank-n_h transition (a product of Householders) that can rotate and permute where a rank-1 memory only reflects; hybrid stacks (Jamba, Griffin, and the gated-DeltaNet-H2 recipes) interleave a few full-attention layers among many linear/SSM/delta layers to buy global exact recall at linear cost — because even a gated delta memory is still a fixed-size state, and some tasks need the full O(L) KV. The throughline is the one this module makes concrete: a sequence-mixing layer is a small design space — how does the state forget, and how does it write? — and the modern architectures are points in it.

Going Deeper

Core Papers:

Practical Resources: