Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

ArXiv: 2605.22791

🎯 Pitch

A single scalar gate can’t simultaneously decide what to erase on the key side and what to write on the value side of a compressed memory—yet that’s exactly how Gated DeltaNet and KDA work. Gated DeltaNet-2 splits this into independent channel-wise erase and write gates, yielding a 9.8-point gain in multi-key needle-in-a-haystack retrieval at 4K context over the next-best model and the strongest overall performance across a 1.3B-parameter sweep.

---

1. Executive Summary

This paper introduces Gated DeltaNet-2, a recurrent attention layer that decouples the active memory edit in delta-rule models by replacing the single scalar gate that jointly controls erasing and writing with a channel-wise erase gate $b_t$ on the key axis and a channel-wise write gate $w_t$ on the value axis. Evaluated at 1.3B parameters trained on 100B FineWeb-Edu tokens against Mamba-2, Gated DeltaNet, Kimi Delta Attention (KDA), and Mamba-3 variants, Gated DeltaNet-2 achieves the strongest overall results across language modeling, commonsense reasoning, and retrieval, with its advantage most pronounced on long-context RULER needle-in-a-haystack benchmarks — particularly multi-key retrieval (MK-NIAH-1), where it improves accuracy at 4K context by 37.8% in the recurrent setting versus the next-best 28.0% from KDA. Ablations confirm that both gates contribute, with the erase gate accounting for most of the gain, establishing that decoupling the two decisions directly targets the primary pressure point of fixed-state recurrence — interference among many compressed associations — only when the model can protect key channels from erasure independently of how strongly it commits new value content.

2. Context and Motivation

The Core Problem: A Single Scalar Cannot Simultaneously Control Erasing and Writing

The paper addresses a specific architectural bottleneck in recurrent linear attention models. When a model maintains a fixed-size key-value memory state $S_t \in \mathbb{R}^{d_k \times d_v}$ and must compress long sequences into that state, two distinct decisions arise at each token:

1. Which old associations should be removed from the memory on the key side — a selective erasure operation that determines which coordinates of the current read (what the state says about the current key) should be cleared to make room for new information.

2. Which new content should be committed to the memory on the value side — a selective write operation that determines which value channels of the incoming token should persist in the state.

The paper observes that in both Gated DeltaNet and Kimi Delta Attention (KDA), these two decisions are tied to a single scalar $\beta_t$. This is the central architectural limitation the paper identifies. The scalar $\beta_t$ simultaneously controls (a) how much of the old read $S_{t-1}^\top k_t$ is subtracted from the state during the delta-rule edit, and (b) how much of the candidate value $v_t$ is written into the state. These are operations on fundamentally different axes of the state matrix: the erase acts along the key dimension $d_k$, while the write acts along the value dimension $d_v$. Forcing them to share one scalar means the model cannot, for instance, heavily erase stale content along some key channels while cautiously writing new content along only a subset of value channels.

Why This Matters: The Interference Problem in Fixed-Size Memory

This limitation is not an abstract mathematical concern — it directly impacts the primary failure mode of recurrent linear models on long-context tasks. When many token associations must share a fixed-size memory state of dimension $d_k \times d_v$, interference becomes the bottleneck: writing a new key-value association can inadvertently overwrite or corrupt existing associations that share overlapping key subspaces.

The delta rule partially addresses this by subtracting the current read before writing, which prevents the new association from accumulating on top of the old one addressed by the same key. However, with a scalar gate $\beta_t$:

• Weak erase ($\beta_t$ small): The model cannot adequately clear stale associations, so old content persists and interferes with new retrieval. This is particularly damaging when the context shifts (e.g., the model encounters information about a new topic that contradicts earlier content).

• Strong erase ($\beta_t$ large): The model aggressively clears the old read, but it also aggressively writes new content. If the new value is noisy, irrelevant, or should only partially be committed, the aggressive write corrupts the state with information that doesn't deserve to persist.

There is no way, under a scalar $\beta_t$, to erase broadly while writing selectively, or vice versa. This matters most on tasks where the model must (a) clear competing associations to reduce interference while (b) discriminating which value channels of the incoming token are worth remembering. Multi-key needle-in-a-haystack (MK-NIAH) tasks exemplify this exact pressure: the state must store several key-value pairs, distinguish the requested key from distractors, and retrieve the correct value — all while managing interference among compressed associations.

The real-world significance is concrete. Recurrent linear models offer linear-time training and constant-memory decoding, making them attractive for long-context applications. But their retrieval performance degrades on tasks requiring precise associative recall under interference — exactly the tasks where softmax attention's explicit token-to-token comparisons excel. Improving the state update rule to better manage interference directly addresses the primary capability gap between recurrent and attention-based architectures, with practical implications for deploying efficient models that can handle long documents, multi-turn conversations, and retrieval-augmented generation without quadratic memory costs.

Prior Approaches and Where They Fall Short

The paper situates itself within a lineage of models that progressively improved the memory update in linear recurrent attention:

The Basic Linear Attention Recurrence (Katharopoulos et al., 2020). The simplest form is purely additive:

$S_t = S_{t-1} + k_t v_t^\top$

Every key-value outer product is added to the state, and nothing is ever explicitly removed. Old associations are overwritten only indirectly through later superpositions — but this is an unreliable mechanism for forgetting. The state becomes cluttered with stale information, making precise retrieval difficult as context length grows.

Mamba-2 (Dao and Gu, 2024). Introduces a data-dependent scalar decay $\alpha_t$ applied before each write:

$S_t = \alpha_t S_{t-1} + k_t v_t^\top, \quad \alpha_t \in (0,1]$

This gives the model a global forgetting operation: all existing associations are uniformly decayed before the new write. The key limitation is that forgetting is global and uniform — every coordinate of the state is decayed by the same factor. There is no mechanism to selectively preserve some associations while clearing others, and there is no active edit that targets specific key directions.

DeltaNet (Schlag et al., 2021; Yang et al., 2024). Introduces the delta rule, an active memory edit that reads the value currently associated with the key and subtracts it before writing:

$S_t = (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top$

When $\|k_t\|^2 = 1$, the matrix $k_t k_t^\top$ is a projector, so $\beta_t = 1$ completely overwrites the association at key $k_t$, and $\beta_t = 0$ leaves it unchanged. This is a significant improvement over purely additive writes because it prevents new information from accumulating on top of old content addressed by the same key direction. However, the model has no global forgetting mechanism — old content persists indefinitely unless it is actively overwritten by a matching key.

Gated DeltaNet (Yang et al., 2025). Combines the two operations:

$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top$

The scalar decay $\alpha_t$ provides global forgetting, while the scalar delta gate $\beta_t$ provides targeted editing. This is a useful division of labor: decay clears the state uniformly, and the delta rule edits specific associations. But both gates are scalar per head: every key channel decays at the same rate, and the erase-and-write strength is controlled by one number $\beta_t$.

Kimi Delta Attention (KDA; Kimi Team et al., 2025). Refines the decay side by making it channel-wise over the key dimension:

$S_t = (I - \beta_t k_t k_t^\top) D_t S_{t-1} + \beta_t k_t v_t^\top$

where $D_t = \text{Diag}(\alpha_t)$ and $\alpha_t \in (0,1]^{d_k}$. Each key channel can now decay at its own rate, giving the model fine-grained control over which information persists. Channels that encode stable, reusable information (e.g., syntactic patterns, factual knowledge) can decay slowly, while channels that encode transient, context-specific information can decay quickly. This is a substantial improvement over scalar decay.

But the remaining limitation is the scalar $\beta_t$. KDA still ties the erase and write decisions. The delta rule in KDA computes the residual $\beta_t(v_t - \bar{S}_t^\top k_t)$: the scalar $\beta_t$ multiplies both the read-subtraction term (erasing old content) and the value-writing term (committing new content). There is no mechanism to, for instance, strongly erase old content along key channels that encode outdated topic information while cautiously writing new content along only a subset of value channels that encode reliable, high-quality new information.

The paper explicitly positions Gated DeltaNet-2 as addressing this remaining tie — the scalar $\beta_t$ that couples two decisions that operate on different axes of the state and should, in principle, be independently controllable.

How This Paper Positions Itself Relative to Existing Work

The paper's contribution is best understood as a surgical intervention at the exact point where KDA and Gated DeltaNet leave a modeling restriction in place. It is not a fundamentally new class of model, nor a complete redesign of the recurrence. The positioning is precise:

What is preserved:

• The overall delta-rule structure: the state update still computes a residual (target value minus current read) and applies a rank-one edit.
• The channel-wise decay $\alpha_t$ from KDA, which gives fine-grained control over forgetting rates across key channels.
• The fast-weight update perspective, the WY-based chunkwise parallel form, and the efficient training path.

What changes:

• The single scalar $\beta_t$ that gates both the erase term and the write term is replaced by a channel-wise erase gate $b_t \in [0,1]^{d_k}$ and a channel-wise write gate $w_t \in [0,1]^{d_v}$.
• The residual becomes $w_t \odot v_t - (D_t S_{t-1})^\top (b_t \odot k_t)$, where the erase gate weights the key coordinates used to read old content, and the write gate weights the value coordinates being inserted.

Relationship to KDA and Gated DeltaNet as special cases. The paper explicitly shows that Gated DeltaNet-2 recovers KDA exactly when $b_t = \beta_t \mathbf{1}_{d_k}$ and $w_t = \beta_t \mathbf{1}_{d_v}$ (both gates collapse to the same scalar), and recovers Gated DeltaNet by further setting $\alpha_t = \alpha_t \mathbf{1}_{d_k}$ (decay also collapses to a scalar). This means the model preserves the known scalar-gated updates as a subspace of its parameterization — it can learn the tied behavior when that is optimal, but is not forced into it. The model is a strict generalization that increases expressive capacity only where the architectural restriction was binding.

Relationship to Mamba-3 (Lahoti et al., 2026). The paper also positions itself relative to the concurrent state-space model direction. Mamba-3 does not use a delta rule at all — it writes key-value correlations into a decayed state without subtracting the current read. Instead, it improves the state-space formulation through exponential-trapezoidal discretization, complex-valued state transitions implemented via data-dependent rotations, and a multi-input, multi-output (MIMO) formulation. The paper includes Mamba-3 in all comparisons and treats it as a representative of the SSM family rather than the delta-rule family. The finding that Gated DeltaNet-2 outperforms Mamba-3 on retrieval tasks (especially MK-NIAH-1) suggests that the delta rule's active read-subtraction mechanism provides a structural advantage for associative recall that the SSM path's improved discretization and rotations do not fully compensate for.

The conceptual contribution is not a new mechanism but a dimensional analysis of an existing one. The paper's key insight is that the scalar $\beta_t$ in Eqs. 6 and 7 conflates two operations on different axes: the erase factor $(I - \beta_t k_t k_t^\top)$ acts in the $d_k \times d_k$ key space, while the write term $\beta_t k_t v_t^\top$ acts in the $d_k \times d_v$ state space. There is no mathematical reason these must share a scalar — it is an artifact of the specific derivation of the delta rule from a local regression objective with a scalar step size. The paper decouples them by generalizing the objective itself (Section 3.2, Eq. 13) to allow different gating on the read direction and the write target, producing a model that is more flexible without changing the fundamental structure of the recurrence or the training algorithm.

3. Technical Approach

3.1 Reader Orientation

Gated DeltaNet-2 is a single recurrent attention layer that can replace self-attention in a Transformer-style language model architecture, enabling linear-time training and constant-memory decoding while maintaining competitive associative recall. The core problem it solves is that prior delta-rule models (Gated DeltaNet, KDA) use a single scalar gate to control both how much old content is erased from the compressed key-value memory and how much new content is written into it — two decisions that operate on entirely different axes of the state matrix. Gated DeltaNet-2 decouples these decisions by introducing a channel-wise erase gate on the key axis and a channel-wise write gate on the value axis, allowing the model to independently control what leaves memory and what enters it, while preserving the efficient chunkwise parallel training algorithm of its predecessors through a careful mathematical reformulation.

3.2 Big-Picture Architecture

The Gated DeltaNet-2 layer sits inside a standard Transformer-style residual block and consists of six major components:

1. Input Projections: Three linear projections plus short causal convolutions and SiLU activations produce the query $q_t \in \mathbb{R}^{d_k}$, key $k_t \in \mathbb{R}^{d_k}$, and value $v_t \in \mathbb{R}^{d_v}$ for each token. Query and key are L2-normalized per head for stability.

2. Decay Gate Branch: A separate linear projection produces a log-decay tensor $g_t \in \mathbb{R}^{d_k}$ (one value per key channel), which is activated through a parameterized softplus and exponentiated to produce channel-wise decay factors $\alpha_t \in (0,1]^{d_k}$. This is computed in fp32 to preserve numerical precision in long cumulative products.

3. Erase Gate Branch: An independent linear projection followed by sigmoid produces the erase gate $b_t \in [0,1]^{d_k}$, which determines which key channels of the old state should be read and removed.

4. Write Gate Branch: Another independent linear projection followed by sigmoid produces the write gate $w_t \in [0,1]^{d_v}$, which determines which value channels of the incoming token should be committed to memory.

5. Gated Delta Rule-2 State Update: The core recurrent operator, which first applies channel-wise decay to the previous state, then reads the decayed state's content along the gated key direction $b_t \odot k_t$, computes a residual against the gated value target $w_t \odot v_t$, and applies a rank-one edit $\bar{k}_t(\text{residual})^\top$ to the state. The output is produced by reading the updated state with the query.

6. Output Normalization and Gating: The recurrent output is RMS-normalized, multiplied elementwise by a SiLU-activated output gate (separate from the erase/write/decay gates), and projected back to the model dimension through a final linear layer.

Information flows sequentially: token representation → projection + convolution → query/key/value + three gate branches → Gated Delta Rule-2 state update → output RMSNorm + SiLU gate → output projection. For hybrid models, a sliding-window attention block is inserted after the Gated DeltaNet-2 mixer within the same residual cell.

3.3 Roadmap for the Deep Dive

I will explain the components in this order, which builds from the foundation up:

• First, the Gated Delta Rule-2 state update equation itself — the mathematical core of the layer — since every other component exists to produce its inputs or consume its output.

• Second, the gate parameterization (decay, erase, write) — how these are produced from token representations, their activation functions, and the precision choices that make long-context training stable.

• Third, the fast-weight update perspective — the online learning interpretation that motivates why this particular mathematical form is chosen and how it generalizes prior work.

• Fourth, the chunkwise parallel training algorithm — how the recurrence is reformulated into a WY-based parallel form that absorbs channel-wise decay into asymmetric erase factors, enabling efficient training on GPUs.

• Fifth, the gate-aware backward pass — the key implementation change required for training, explaining why the standard scalar post-scaling trick from KDA and Gated DeltaNet breaks and what replaces it.

• Sixth, the block design and hybrid architecture — how the Gated DeltaNet-2 mixer is integrated into a complete language model, including the output gating mechanism and the sliding-window attention augmentation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a model architecture paper whose core idea is that decoupling the erase and write decisions in the delta-rule state update — by replacing a scalar gate with separate channel-wise gates on the key and value axes — improves associative recall in fixed-size recurrent memory without sacrificing training efficiency.

---

The Gated Delta Rule-2 State Update

The central mathematical operation in Gated DeltaNet-2 is the state update equation (Eq. 10 in the paper), which defines how the recurrent memory state $S_t \in \mathbb{R}^{d_k \times d_v}$ evolves from token $t-1$ to token $t$.

Step 1: Channel-wise decay. The previous state is first decayed elementwise along the key dimension:

$\bar{S}_t = D_t S_{t-1}$

where $D_t = \text{Diag}(\alpha_t)$ and $\alpha_t \in (0,1]^{d_k}$ is a vector of per-channel decay factors. Each row of $S_{t-1}$ corresponds to one key channel and is multiplied by its own decay $\alpha_{t,i}$, so channel $i$ of the state is scaled by a cumulative product of all $\alpha_{1,i}, \alpha_{2,i}, \ldots, \alpha_{t,i}$ since the beginning of the sequence. Channels with small decay factors forget quickly; channels with decay factors near 1 retain information indefinitely.

Step 2: Gated read of old content. The decayed state is read using the gated erase direction:

$e_t = b_t \odot k_t$

$r_t = \bar{S}_t^\top e_t$

where $b_t \in [0,1]^{d_k}$ is the channel-wise erase gate and $k_t \in \mathbb{R}^{d_k}$ is the L2-normalized key. The vector $r_t \in \mathbb{R}^{d_v}$ is the value that the decayed state currently associates with the gated key direction — it is a weighted combination of all previously stored value vectors, where the weights are determined by how well each historical key direction aligns with $e_t$. The erase gate $b_t$ determines which key channels participate in this read: if $b_{t,i} = 0$, channel $i$ of the key is ignored, meaning the read does not capture content stored along that key subspace; if $b_{t,i} = 1$, channel $i$ participates fully.

Step 3: Gated write target. The incoming value is gated to produce the write target:

$z_t = w_t \odot v_t$

where $w_t \in [0,1]^{d_v}$ is the channel-wise write gate and $v_t \in \mathbb{R}^{d_v}$ is the value vector. The write gate $w_t$ determines which value channels are committed to memory: if $w_{t,j} = 0$, channel $j$ of $v_t$ is ignored and will not be stored; if $w_{t,j} = 1$, channel $j$ is stored fully.

Step 4: Delta-rule edit. The state is updated by subtracting the old read and adding the new target:

$S_t = \bar{S}_t + k_t(z_t - r_t)^\top$

The term $(z_t - r_t)^\top$ is the residual: the difference between what the memory currently says about the key direction and what it should say. This residual is written into the state as a rank-one outer product $k_t(\text{residual})^\top$, which modifies the state along the direction of $k_t$ in the key space to bring the read closer to $z_t$.

Step 5: Output read. The output for the current token is produced by reading the updated state with the query:

$o_t = S_t^\top q_t$

Combined form. Substituting the decay and read steps into the update yields the compact expression (Eq. 10):

$S_t = \left(I - k_t(b_t \odot k_t)^\top\right) D_t S_{t-1} + k_t(w_t \odot v_t)^\top$

What this equation computes: The state $S_t$ is the result of a rank-one edit on the decayed previous state. The left term $(I - k_t(b_t \odot k_t)^\top) D_t S_{t-1}$ takes the decayed state and subtracts from it the old content read along the gated key direction — this is the erase operation. The right term $k_t(w_t \odot v_t)^\top$ adds the gated value along the key direction — this is the write operation. The erase factor uses the gated key $b_t \odot k_t$ as the read direction, while the write factor uses the un-gated key $k_t$ as the write direction (with the gated value $w_t \odot v_t$ as the content). This asymmetry is deliberate: the erase reads along a channel-selective key subspace to decide what to remove, but the write always targets the full key direction to ensure the new association is addressable by the canonical key.

Why this form: The key innovation is the separation of $b_t$ and $w_t$. In KDA (Eq. 7), this equation would be:

$S_t = (I - \beta_t k_t k_t^\top) D_t S_{t-1} + \beta_t k_t v_t^\top$

The scalar $\beta_t$ multiplies both the erase term (controlling how much of the old read is subtracted) and the write term (controlling how much of $v_t$ is committed). Gated DeltaNet-2 replaces $\beta_t$ with $b_t$ inside the erase factor (controlling which key channels are read) and $w_t$ inside the write term (controlling which value channels are stored). The model can now, for example, set $b_t$ to be large (aggressively erase old content from all key channels) while setting $w_t$ to be small on some value channels (cautiously write only the most reliable dimensions of $v_t$). In the tied case $b_t = \beta_t \mathbf{1}_{d_k}$ and $w_t = \beta_t \mathbf{1}_{d_v}$, the equation exactly reduces to KDA; if additionally $\alpha_t = \alpha_t \mathbf{1}_{d_k}$, it reduces to Gated DeltaNet. The innovation is architectural, not algorithmic — the model has strictly more expressive capacity at the same recurrent state size.

---

Gate Parameterization

The three gates — decay $\alpha_t$, erase $b_t$, and write $w_t$ — are all produced from the token representation $x_t \in \mathbb{R}^{d_{\text{model}}}$ through independent linear projections with different activation functions. The parameterization choices reflect careful attention to numerical stability, training dynamics, and the semantic meaning of each gate's output range.

Log-decay gate (Eq. 12):

$g_t = -\exp(a) \odot \text{softplus}(W_f x_t + \delta)$

$\alpha_t = \exp(g_t)$

where $a \in \mathbb{R}^{d_k}$ is a per-key-channel learnable vector (broadcast within each head), $W_f \in \mathbb{R}^{d_{\text{model}} \times d_k}$ is the feed-forward projection weight, and $\delta \in \mathbb{R}^{d_k}$ is a per-key-channel bias term.

What it computes: The log-decay $g_t$ is a vector of $d_k$ values, each of which is guaranteed to be negative (since $\exp(a) > 0$ and softplus output is non-negative, their product is subtracted from zero). Exponentiating $g_t$ produces $\alpha_t \in (0,1]^{d_k}$, where each channel's decay is strictly in the unit interval. The bias $\delta$ and the learned magnitude $a$ together determine the baseline decay rate per channel: channels with smaller $a_i$ or more negative $\delta_i$ will have smaller $|g_{t,i}|$ after the softplus, producing $\alpha_{t,i}$ closer to 1 (slow decay, information persists longer).

Why this form: The log-space parameterization with explicit fp32 computation is critical for long-context stability. The cumulative decay over $T$ tokens is $\gamma_T = \prod_{t=1}^T \alpha_t = \exp(\sum_{t=1}^T g_t)$. If $\alpha_t$ were computed directly in low precision (e.g., bf16), the multiplicative accumulation would lose precision rapidly as $T$ grows — a product of thousands of bf16 numbers can drift significantly from the true value. By computing $g_t$ in fp32 and maintaining the cumulative sum $\sum g_t$ in fp32, the exponential can be applied exactly when needed. The softplus activation ensures $g_t$ is smooth and differentiable everywhere (unlike a hard clamp), while the $-\exp(a)$ scaling allows the model to learn how aggressively the decay responds to the token representation.

Erase gate (Eq. 11):

$b_t = \sigma(W_b x_t)$

where $\sigma$ is the sigmoid function and $W_b \in \mathbb{R}^{d_{\text{model}} \times d_k}$ is the erase projection weight.

What it computes: $b_t \in [0,1]^{d_k}$ is a per-key-channel value between 0 and 1. When $b_{t,i} = 0$, channel $i$ of the key is excluded from the read direction $e_t$, meaning the delta-rule edit will not erase information stored along this key subspace. When $b_{t,i} = 1$, channel $i$ participates fully in the read, and the edit will subtract the full amount of old content aligned with this key channel.

Why sigmoid: The range $[0,1]$ has a clear semantic interpretation: the fraction of the key channel used for reading old content. A sigmoid naturally saturates at 0 and 1, allowing the model to make crisp binary-like decisions about which key channels to erase from, while remaining differentiable during training. The paper also supports an expanded range $b_t \in [0,2]^{d_k}$ for the negative-eigenvalue variant (Grazzi et al., 2025), implemented by scaling the sigmoid output by 2. This allows the erase gate to amplify the key direction beyond 1, creating a negative-eigenvalue effect in the state transition matrix $(I - k_t e_t^\top)D_t$.

Write gate (Eq. 11):

$w_t = \sigma(W_w x_t)$

where $W_w \in \mathbb{R}^{d_{\text{model}} \times d_v}$ is the write projection weight.

What it computes: $w_t \in [0,1]^{d_v}$ is a per-value-channel value between 0 and 1. When $w_{t,j} = 0$, channel $j$ of the value $v_t$ is suppressed to zero in the write target $z_t$, meaning this value dimension is not committed to the state. When $w_{t,j} = 1$, channel $j$ is stored fully.

Why the write gate stays in $[0,1]^{d_v}$: The paper explicitly notes that the write gate does not need the expanded $[0,2]$ range because "the spectral effect concerns the state transition, not the value magnitude." The state transition matrix $I - k_t e_t^\top$ governs the dynamics of the recurrence — whether the state contracts, expands, or preserves its eigenvalues — and the erase gate influences this matrix through $e_t$. The write gate affects only the additive term $k_t z_t^\top$, which adds information to the state but does not change how existing information evolves. Therefore, the write gate's role is purely about content selection (which value dimensions to store), not about spectral dynamics, and the $[0,1]$ range is the appropriate constraint for a gating mechanism that selects a subset of value channels.

Key design choice — independent projections: Each gate uses its own weight matrix ($W_f$, $W_b$, $W_w$) rather than sharing parameters or deriving one gate from another. This is important because the three gates serve fundamentally different purposes: decay controls forgetting rates (a temporal property), erase controls which key subspaces to clean (a spatial property on the key axis), and write controls which value dimensions to store (a spatial property on the value axis). Forcing them to share parameters would create an artificial bottleneck — the model needs the freedom to express independent patterns for each decision.

---

Fast-Weight Update Perspective

The Gated Delta Rule-2 can be interpreted as an online learning update on a fast-weight memory, following the framework established by Schlag et al. (2021) and extended by Yang et al. (2024, 2025). This perspective provides the theoretical motivation for why the specific mathematical form in Eq. 10 is chosen and clarifies how it generalizes prior work.

At each time step, the model solves a local optimization problem:

$S_t = \arg\min_S \mathcal{L}_t(S)$

$\mathcal{L}_t(S) = \|S - \bar{S}_t\|_F^2 - 2\left\langle S^\top k_t, \, z_t - \bar{S}_t^\top e_t\right\rangle$

where $\| \cdot \|_F$ is the Frobenius norm and $\langle \cdot, \cdot \rangle$ is the Frobenius inner product.

What each term does: The first term $\|S - \bar{S}_t\|_F^2$ is a proximity regularizer: it penalizes the new state for deviating from the decayed previous state $\bar{S}_t = D_t S_{t-1}$. This ensures that the update is conservative — without a strong reason to change (from the second term), the state should stay close to its decayed previous value. The second term $-2\langle S^\top k_t, \, z_t - \bar{S}_t^\top e_t \rangle$ encodes the associative correction: it encourages the new state $S$ to satisfy $S^\top k_t \approx z_t$, i.e., reading from the new state with key $k_t$ should produce the gated write target $z_t$. The correction is measured relative to what the decayed state already reads: $(z_t - \bar{S}_t^\top e_t)$ is the residual — the gap between what we want the state to say and what it currently says along the erase-gated read direction.

Why this loss: The gradient of the loss with respect to $S$ is:

$\nabla_S \mathcal{L}_t(S) = 2(S - \bar{S}_t) - 2k_t\left(z_t - \bar{S}_t^\top e_t\right)^\top$

Setting this to zero gives exactly $S_t = \bar{S}_t + k_t(z_t - \bar{S}_t^\top e_t)^\top$, which is the Gated Delta Rule-2 update. The loss therefore formalizes what the update does: it makes the minimal Frobenius-norm change to the decayed state that corrects the read at key $k_t$ to produce the gated target $z_t$. This is a single step of online gradient descent with step size 1 on a regression problem where the state $S$ is being trained to map keys to values.

Comparison with prior fast-weight objectives (Table 1): Table 1 in the paper systematically compares the local objective for each model, and the differences are instructive.

• Mamba-2 uses $\mathcal{L}_t(S) = \|S - \alpha_t S_{t-1}\|_F^2 - 2\langle S^\top k_t, v_t \rangle$. The second term only encourages $S^\top k_t \approx v_t$ — there is no subtraction of the current read, so the update is purely additive. The state accumulates associations rather than editing them.

• DeltaNet uses $\mathcal{L}_t(S) = \|S - S_{t-1}\|_F^2 - 2\langle S^\top k_t, \beta_t(v_t - S_{t-1}^\top k_t) \rangle$. The second term now includes the residual $(v_t - S_{t-1}^\top k_t)$, but with a scalar $\beta_t$ gating both the erase and write. The decay is absent (no $\alpha_t$), so old content persists indefinitely unless actively overwritten.

• Gated DeltaNet adds the scalar decay back: $\mathcal{L}_t(S) = \|S - \alpha_t S_{t-1}\|_F^2 - 2\langle S^\top k_t, \beta_t(v_t - (\alpha_t S_{t-1})^\top k_t) \rangle$. Still scalar $\beta_t$, still scalar $\alpha_t$.

• KDA makes decay channel-wise: $\mathcal{L}_t(S) = \|S - D_t S_{t-1}\|_F^2 - 2\langle S^\top k_t, \beta_t(v_t - (D_t S_{t-1})^\top k_t) \rangle$. The decay is now $D_t = \text{Diag}(\alpha_t)$ with per-channel rates, but the correction term still uses $\beta_t$ as a scalar.

• Gated DeltaNet-2 makes the correction itself channel-aware: the residual is $z_t - \bar{S}_t^\top e_t = w_t \odot v_t - (D_t S_{t-1})^\top (b_t \odot k_t)$. The erase gate $b_t$ controls which key channels are used to compute the old read, and the write gate $w_t$ controls which value channels of the target are committed. The first term in the loss still uses channel-wise decay $D_t$.

• Mamba-3 does not fit neatly into this fast-weight framework because it does not subtract a current read from the state. Instead, it writes key-value correlations using a two-token exponential-trapezoidal input rule: $S_t = \alpha_t S_{t-1} + \eta_t \tilde{k}_{t-1} v_{t-1}^\top + \zeta_t \tilde{k}_t v_t^\top$, where $\tilde{k}$ includes a data-dependent rotation and $\eta_t, \zeta_t$ are step-size coefficients. This is in the Mamba-2 family (correlation writes, no read-subtraction), not the delta-rule family.

What the fast-weight perspective reveals about the contribution: The generalization from KDA to Gated DeltaNet-2 is entirely in the interior dot product of the correction term. KDA computes $\beta_t(v_t - (D_t S_{t-1})^\top k_t)$ — the read uses the full key $k_t$, the target uses the full value $v_t$, and a single scalar $\beta_t$ gates both. Gated DeltaNet-2 computes $w_t \odot v_t - (D_t S_{t-1})^\top (b_t \odot k_t)$ — the read uses a gated key subspace, the target uses a gated value subspace, and the proximity regularizer remains unchanged. The theoretical insight is that the scalar tie in KDA is not a necessary consequence of the fast-weight formulation; it is an arbitrary restriction that can be lifted by allowing the key read and value target to be gated independently. The model's increased expressive capacity comes from the fact that $(b_t \odot k_t)$ and $(w_t \odot v_t)$ can express different channel patterns, enabling the state update to, for instance, erase broadly (many key channels have high $b_{t,i}$) while writing selectively (few value channels have high $w_{t,j}$).

---

Chunkwise Parallel Training via WY Representation

Training a recurrent model token by token is prohibitively slow on GPUs, which achieve their throughput through parallel matrix operations. The standard solution for linear recurrent models is a chunkwise algorithm: split the sequence into fixed-size chunks, compute token interactions within each chunk using dense matrix multiplication, and propagate the recurrent state across chunks sequentially. The key challenge for Gated DeltaNet-2 is that the channel-wise decay $\alpha_t$ makes the transition matrix non-identical across timesteps, and the channel-wise erase gate $b_t$ makes the erase factor asymmetric ($k_t e_t^\top$ rather than $k_t k_t^\top$). The paper shows that both complications can be absorbed into a single normalized recurrence that preserves the efficient WY (Householder) parallel form used by KDA and DeltaNet.

Step 1: Decay-normalized state. Define the cumulative log-decay $G_r = \sum_{i=1}^r g_i$ where $g_i$ is the log-decay vector at token $i$, and let $\gamma_r = \exp(G_r) \in \mathbb{R}^{d_k}_{\geq 0}$ with $\gamma_0 = \mathbf{1}_{d_k}$ (all ones). Define a normalized state $\hat{S}_r$ such that:

$S_r = \text{Diag}(\gamma_r) \hat{S}_r$

Since $\gamma_0 = \mathbf{1}_{d_k}$, the normalized initial state is $\hat{S}_0 = S_0$ (the state at the start of the chunk). Substituting $S_r = \text{Diag}(\gamma_r) \hat{S}_r$ into the Gated Delta Rule-2 recurrence (Eq. 29 in the appendix, equivalent to Eq. 10) and using $\alpha_r = \gamma_r / \gamma_{r-1}$ (elementwise division), the channel-wise decay cancels entirely:

$\hat{S}_r = \left(I - \bar{k}_r \bar{e}_r^\top\right) \hat{S}_{r-1} + \bar{k}_r z_r^\top$

where the decay-normalized key and decay-normalized erase direction are:

$\bar{k}_r = \gamma_r^{-1} \odot k_r$

$\bar{e}_r = \gamma_r \odot e_r = \gamma_r \odot (b_r \odot k_r)$

and $z_r = w_r \odot v_r$ is unchanged (no decay normalization on the value side).

What the normalization achieves: The channel-wise decay $\alpha_t$ has been completely absorbed into the left and right factors of each rank-one erase matrix. The normalized recurrence is now a pure asymmetric delta recurrence with no decay term — exactly the same mathematical structure as the original DeltaNet, but with an asymmetric erase factor $\bar{e}_r \neq \bar{k}_r$ (in DeltaNet, $\bar{e}_r = \beta_r \bar{k}_r$, which is symmetric up to a scalar). The decay is now carried implicitly by the fact that early tokens have large $\gamma_r$ (since they have accumulated many small decays, $\gamma_r$ shrinks over time), so $\bar{k}_r = \gamma_r^{-1} \odot k_r$ amplifies early keys while $\bar{e}_r = \gamma_r \odot e_r$ suppresses early erase directions. This is precisely the mechanism that makes the WY representation work: the recurrence is now a product of matrices of the form $I - \bar{k}_r \bar{e}_r^\top$, which can be inverted efficiently using the Woodbury / Householder WY identity.

Step 2: WY representation. Inside a chunk of size $C$, let $\bar{K} \in \mathbb{R}^{C \times d_k}$, $\bar{E} \in \mathbb{R}^{C \times d_k}$, and $Z \in \mathbb{R}^{C \times d_v}$ contain rows $\bar{k}_r^\top$, $\bar{e}_r^\top$, and $z_r^\top$, respectively. In matrix form, using Eq. 20:

$\bar{E} = \gamma \odot (B \odot K)$

$Z = W \odot V$

where $\gamma \in \mathbb{R}^{C \times d_k}$ contains rows $\gamma_r^\top$, $B$ and $W$ contain the erase and write gate rows, and $K$ and $V$ contain the key and value rows.

Define the strictly lower triangular matrix:

$T = \text{tril}(\bar{E} \bar{K}^\top, -1) \in \mathbb{R}^{C \times C}$

The element $T_{rs} = \bar{e}_r^\top \bar{k}_s$ for $s < r$ (and zero otherwise) captures the interaction between the erase direction at position $r$ and the key at position $s$. Since $T$ is strictly lower triangular, it is nilpotent: $T^C = 0$.

Define the WY inverse:

$A = (I + T)^{-1} \in \mathbb{R}^{C \times C}$

Because $T$ is strictly lower triangular with zero diagonal, $I + T$ is lower triangular with unit diagonal and is trivially invertible by forward substitution — no expensive matrix inversion is needed. The WY auxiliaries are:

$Y = A \bar{E} \in \mathbb{R}^{C \times d_k}$

$U = A Z \in \mathbb{R}^{C \times d_v}$

These are the "compressed" representations that encode all token interactions within the chunk.

What $A$, $Y$, and $U$ represent: The inverse matrix $A$ satisfies $(I+T)A = I$, which means $A = I - TA$ or equivalently $A = I - \text{tril}(\bar{E}\bar{K}^\top A, -1)$. This is the standard Householder WY representation: the product of many rank-one corrections $(I - \bar{k}_r \bar{e}_r^\top)$ can be compactly expressed as $I - \bar{K}^\top Y$ (or variations thereof). The auxiliary $Y$ encodes the cumulative effect of all erase operations on the state, while $U$ encodes the cumulative effect of all write operations. Both are computed by forward substitution through the same triangular system $(I+T)$, which is efficient because it avoids materializing the full $C \times C \times d_k$ interaction tensor.

Step 3: State and output equations. The end-of-chunk state and the per-chunk output are expressed directly in terms of $Y$, $U$, and the initial state $S_0$ (Eqs. 23 and 24):

$S_{C} = \text{Diag}(\gamma_C) S_0 + K_{\text{tail}}^\top \left(U - Y S_0\right)$

$O = Q_\gamma S_0 + A_{qk} \left(U - Y S_0\right)$

where:

• $K_{\text{tail}}$ has row $r$ equal to $((\gamma_C / \gamma_r) \odot k_r)^\top$ — the decay-compensated keys that undo the normalization to produce the correct end-of-chunk state.
• $Q_\gamma$ has row $r$ equal to $(\gamma_r \odot q_r)^\top$ — the decay-normalized queries.
• $A_{qk}$ is the causal score matrix with element $(A_{qk})_{rs} = \mathbf{1}_{r \ge s} \, q_r^\top \text{Diag}(\gamma_r / \gamma_s) k_s$ — the decay-aware attention scores between query $r$ and key $s$.

What these equations compute: The term $R = U - Y S_0$ is the chunk-local residual: it captures how much each token's write target differs from what the initial state $S_0$ would predict, after accounting for all the erase operations within the chunk. The end-of-chunk state is $S_0$ scaled by the cumulative decay $\gamma_C$, plus the contributions of all chunk-local writes projected through $K_{\text{tail}}$. The output at each position $r$ is the initial state read plus the chunk-local read, where the score matrix $A_{qk}$ determines how much each past token's residual contributes to the current token's output.

Why this form preserves efficiency: The computational cost is dominated by three operations: computing $T = \bar{E}\bar{K}^\top$ (a $C$-by-$C$ matrix multiply over $d_k$), the forward substitution to compute $A$ (triangular solve, $O(C^2)$), and the auxiliary constructions $Y = A\bar{E}$ and $U = AZ$ (two more $C$-by-$C$ times $d_k$/$d_v$ multiplies). With a fixed chunk size $C = 64$, all these operations are small constant-size matrix multiplies that map well to tensor cores. The inter-chunk recurrence involves only the state $S_0$ (size $d_k \times d_v$) and the residuals $R$ (size $C \times d_v$), both of which are linear in the number of chunks. The total cost is $O(LC d_k)$ for a sequence of length $L$ with chunk size $C$, which is linear in sequence length and compares favorably to the $O(L^2 d_k)$ cost of softmax attention.

The only difference from KDA is how $\bar{E}$ and $Z$ are formed. In KDA, $\bar{E}$ would be $\gamma \odot (\beta_r k_r)$ — the erase direction is a scalar multiple of the key. In Gated DeltaNet-2, $\bar{E}$ is $\gamma \odot (b_r \odot k_r)$ — the erase direction is the elementwise product of the erase gate and the key, so each key channel can be independently gated. Similarly, $Z$ in KDA would be $\beta_r v_r$, while in Gated DeltaNet-2 it is $w_r \odot v_r$. The rest of the computation — the $T$ matrix construction, the $A$ inverse, the $Y$ and $U$ auxiliaries, and the state/output equations — is structurally identical. This is the key engineering insight of the paper: the WY framework is agnostic to how the erase and write factors are parameterized, as long as they are rank-one per token. By absorbing the gates into $\bar{E}$ and $Z$, the entire computational machinery of efficient delta-rule training is reused without modification.

Row recurrences (Appendix A.4). The auxiliaries can also be computed row by row to clarify the forward substitution:

$y_r^\top = \bar{e}_r^\top - \sum_{s < r} \bar{e}_r^\top \bar{k}_s \, y_s^\top$

$u_r^\top = z_r^\top - \sum_{s < r} \bar{e}_r^\top \bar{k}_s \, u_s^\top$

Each row $y_r$ is the erase direction $\bar{e}_r$ minus the cumulative effect of all past keys ($\bar{k}_s$) on the current erase direction, weighted by the past auxiliaries $y_s$. Similarly, each $u_r$ is the write target $z_r$ minus the cumulative effect of past keys on the current write. This row-wise view makes clear that $y_r$ and $u_r$ solve the same triangular system with different right-hand sides ($\bar{e}_r$ vs. $z_r$), which is why the WY inverse $A$ can be shared between both auxiliaries.

Chunk size and implementation details (Appendix C.2). The chunk size is fixed to $C = 64$. The forward kernels are implemented as fused Triton kernels. The intra-chunk product kernel computes $A_{qk}$ and $T$, with the Gated DeltaNet-2 specific computation being the tow factor of $T$:

$T_{rs} = \bar{e}_r^\top \bar{k}_s = (\gamma_r \odot b_r \odot k_r)^\top (\gamma_s^{-1} \odot k_s), \quad s < r$

The erase gate $b_r$ is multiplied into the key tile before the dot product, which is the only place where Gated DeltaNet-2 differs from KDA in the forward pass. The WY solve kernel performs the forward substitution to compute $A$. The auxiliary construction kernel builds $U = A(W \odot V)$ and $Y = A\bar{E}$. The state and output kernels are structurally identical to KDA because they consume $Y$ and $U$ as already-formed tensors.

---

Gate-Aware Backward Pass

The backward pass for training must compute gradients with respect to all parameters, including the erase gate projection $W_b$, the write gate projection $W_w$, the decay projection $W_f$, and the query/key/value projections. The paper emphasizes that the backward pass requires a specific modification — gate-aware accumulation — because the standard scalar post-scaling trick used in KDA and Gated DeltaNet is invalid when the gates are channel-wise.

What breaks in the scalar post-scaling trick. In KDA, the erase factor is $\beta_r k_r$ and the write factor is $\beta_r v_r$. When computing the gradient of the loss $\mathcal{L}$ with respect to the WY inverse $A$, the chain rule requires accumulating contributions from the products $U = AZ$ and $Y = A\bar{E}$:

$\frac{\partial \mathcal{L}}{\partial A} = \frac{\partial \mathcal{L}}{\partial U} Z^\top + \frac{\partial \mathcal{L}}{\partial Y} \bar{E}^\top$

In KDA, $Z = \beta_r V$ and $\bar{E} = \beta_r (\gamma \odot K)$. Because $\beta_r$ is a scalar for each row, it can be factored out of the dot products:

$\frac{\partial \mathcal{L}}{\partial A} = \sum_r \beta_r \frac{\partial \mathcal{L}}{\partial u_r} v_r^\top + \sum_r \beta_r \frac{\partial \mathcal{L}}{\partial y_r} (\gamma_r \odot k_r)^\top$

The scalar post-scaling trick computes the dot products without the $\beta_r$ factors, then multiplies by $\beta_r$ afterward. This is valid because $\beta_r$ is a single scalar that can be moved outside the vector-vector outer product.

In Gated DeltaNet-2, $Z = W \odot V$ and $\bar{E} = \gamma \odot (B \odot K)$. The write gate $w_r \in \mathbb{R}^{d_v}$ is a different vector for every row, and the erase gate $b_r \in \mathbb{R}^{d_k}$ is a different vector for every row. There is no scalar or vector that can be factored out to recover the correct gradient from a post-scaling:

$\frac{\partial \mathcal{L}}{\partial A} = \sum_r \frac{\partial \mathcal{L}}{\partial u_r} (w_r \odot v_r)^\top + \sum_r \frac{\partial \mathcal{L}}{\partial y_r} (\gamma_r \odot b_r \odot k_r)^\top$

The Hadamard products $\odot$ with $w_r$ and $b_r$ must be present inside the accumulation, because $w_r$ is different for every row and interacts with $v_r$ elementwise before the outer product with $\partial \mathcal{L}/\partial u_r$. A post-scaling would need a different scaling per value channel per token, which defeats the purpose of factored computation.

The gate-aware backward equations (Appendix B.2, Eqs. 64–65):

$\text{d}A \mathrel{+}= \text{d}U \, Z^\top, \quad Z = W \odot V$

$\text{d}A \mathrel{+}= \text{d}Y \, \bar{E}^\top, \quad \bar{E} = \gamma \odot (B \odot K)$

where $\text{d}A$ is the gradient with respect to $A$, $\text{d}U$ is the upstream gradient from the state and output paths, and $\text{d}Y$ is the upstream gradient from the state path. The $+=$ notation indicates accumulation (since both terms contribute to $\text{d}A$).

What these equations compute: The first term says: the gradient flowing from $U$ into $A$ is $\text{d}U$ multiplied by $Z^\top$, where $Z$ includes the write gate $W$ elementwise-multiplied with $V$. The second term says: the gradient flowing from $Y$ into $A$ is $\text{d}Y$ multiplied by $\bar{E}^\top$, where $\bar{E}$ includes the erase gate $B$ elementwise-multiplied with $K$ and scaled by $\gamma$. Both gates are baked into the accumulation products — there is no separate post-scaling step.

After accumulating $\text{d}A$, the WY inverse gradient follows the standard triangular vector-Jacobian product:

$\text{d}T = -\text{tril}\left(A^\top \, \text{d}A \, A^\top, -1\right)$

which propagates the gradient through the inverse $(I+T)^{-1}$ to the strictly lower triangular matrix $T$. From $\text{d}T$, the gradients to the elementwise components follow:

$\text{d}\bar{E} \mathrel{+}= \text{d}T \, \bar{K}, \quad \text{d}\bar{K} \mathrel{+}= \text{d}T^\top \bar{E}$

These are standard operations that decompose the gradient through the outer product $T = \bar{E}\bar{K}^\top$ (strictly lower triangular). New in Gated DeltaNet-2 are the gradients through the gates:

$\text{d}B \mathrel{+}= \text{d}\bar{E} \odot \gamma \odot K$

$\text{d}W \mathrel{+}= \text{d}Z \odot V$

$\text{d}V \mathrel{+}= \text{d}Z \odot W$

where $\text{d}Z = A^\top \text{d}U$ is the gradient through the write auxiliary (Eq. 64). The erase gate gradient $\text{d}B$ involves the elementwise product of three tensors: $\text{d}\bar{E}$, $\gamma$, and $K$. The write gate gradient $\text{d}W$ involves the elementwise product with $V$, which is natural because $W$ gates $V$ in the forward pass.

The complete backward flow (Appendix B):

1. Output and state paths: Gradients flow from the chunk output $O$ and end-of-chunk state $S_C$ into the residuals $R = U - Y S_0$ and into the initial state $S_0$. These operations are structurally identical to KDA because they only consume $Y$ and $U$ as already-formed tensors (Eqs. 53–63 in the appendix).

2. Gate-aware WY inverse path: Gradients from the residuals propagate into $U$, $Y$, and $A$ through Eq. 64 and Eq. 65, with the gates included in the accumulation products as described above.

3. Intra-chunk products: Gradients from $\text{d}T$ propagate into $\text{d}\bar{E}$ and $\text{d}\bar{K}$, and then into $\text{d}K$, $\text{d}Q$, $\text{d}B$, and $\text{d}g$ (the log-decay) through elementwise products and reverse cumulative sums (Eqs. 69–82 in the appendix).

4. Cumulative decay gradients: Since $\gamma_r = \exp(\sum_{i \le r} g_i)$, the gradient of $\gamma_r$ with respect to $g_i$ is $\gamma_r$ for all $i \le r$ (and zero otherwise). Therefore $\text{d}g_i = \sum_{r \ge i} \text{d}\gamma_r \odot \gamma_r$ — a reverse cumulative sum over the chunk. This is why the log-decay must be available in fp32: the cumulative sum of gradients over potentially thousands of tokens in a long sequence would lose precision in bf16.

Why this backward pass is efficient: The only mathematical change from KDA is in the accumulation of $\text{d}A$ (Eqs. 64–65) and the gate gradient emissions (Eqs. 74–75, 73 in the appendix). The rest of the backward computation — the triangular inverse vector-Jacobian product, the state and output vector-Jacobian products, the intra-chunk score and tail-key gradients — retains exactly the same matrix shapes and operation types as KDA. The additional cost is a constant factor per chunk: the gate elementwise products add $O(C d_k + C d_v)$ operations, which is negligible compared to the $O(C^2 d_k + C^2 d_v)$ matrix multiplies.

Kernel-level details (Appendix C.3): The gate-aware WY vector-Jacobian product is implemented as a dedicated Triton kernel that accumulates $\text{d}A$ with $Z^\top = (W \odot V)^\top$ and $\bar{E}^\top = (\gamma \odot B \odot K)^\top$, and simultaneously emits the direct gradients $\text{d}W = \text{d}Z \odot V$, $\text{d}V = \text{d}Z \odot W$, and $\text{d}B += \text{d}\bar{E} \odot \gamma \odot K$. The kernel has shape $B \times T \times H \times d_k$ for the erase gate gradient and $B \times T \times H_v \times d_v$ for the write gate gradient (where $H_v$ is the number of value heads for grouped-value attention). On Hopper GPUs, the warp search is restricted to two and four warps to avoid a Triton WGMMA layout assertion for the $64 \times 64$ accumulator.

---

Block Design and Hybrid Architecture

Gated DeltaNet-2 is used as the recurrent token mixer in a standard Transformer-style residual block, with several design choices that affect training stability and model capacity.

Gated DeltaNet-2 token mixer block (Figure 1, right):

The block processes a token representation $x_t \in \mathbb{R}^{d_{\text{model}}}$ through the following pipeline:

1. Query/Key/Value projections: Three linear projections produce $q_t$, $k_t$, and $v_t$, each with their own weight matrix. Each projection is followed by a short causal convolution (kernel width not explicitly specified in the paper, but typical values for this model family are 3 or 4) and a SiLU activation function. The convolution provides local temporal smoothing, which is particularly important for recurrent models because the state update operates on a per-token basis and lacks the local receptive field that attention heads naturally have.

2. L2 normalization: The query $q_t$ and key $k_t$ are L2-normalized per head. This is critical for the delta-rule update because when $\|k_t\|^2 = 1$, the matrix $k_t k_t^\top$ is a projector (its eigenvalues are 0 and 1). In Gated DeltaNet-2, the erase factor is asymmetric ($k_t e_t^\top$ where $e_t = b_t \odot k_t$), so the matrix is not strictly a projector, but normalization still stabilizes the scale of both the attention scores $A_{qk}$ and the interaction matrix $T$. Without normalization, keys with varying magnitudes would cause the state transition matrix $I - k_t e_t^\top$ to have unpredictable spectral radii, potentially leading to exploding or vanishing state dynamics.

3. Gate branches: Three separate linear projections (no convolution, no activation before the gate-specific activation) produce:

   • The log-decay $g_t$ via Eq. 12 (softplus activation, computed in fp32).
   • The erase gate $b_t$ via sigmoid of a linear projection.
   • The write gate $w_t$ via sigmoid of a linear projection.

4. Gated Delta Rule-2: The core recurrence (Eq. 10) updates the state $S_{t-1}$ to $S_t$ and produces the output $o_t = S_t^\top q_t$.

5. Output normalization and gating: The output $o_t$ is first passed through RMSNorm, then multiplied elementwise by a SiLU-activated output gate. This output gate is separate from the erase, write, and decay gates — it is a standard gating mechanism used in many recurrent architectures (e.g., Mamba, Gated DeltaNet) to control the magnitude of the mixer output before it is added to the residual stream.

6. Output projection: A final linear projection maps the gated output back to dimension $d_{\text{model}}$.

Why SiLU for Q/K/V but sigmoid for gates: The SiLU (Sigmoid Linear Unit, also called Swish) is used for the query, key, and value activations because these are continuous-valued representations that benefit from the smooth, unbounded-above nature of SiLU — values can be positive or negative, and the self-gating property ($\text{SiLU}(x) = x \cdot \sigma(x)$) provides a form of soft thresholding. The gates $b_t$ and $w_t$ use sigmoid because their outputs are semantically gates: values between 0 and 1 that represent the fraction of a channel to include or exclude. The sigmoid naturally saturates at 0 and 1, which allows the model to make crisp decisions about which channels to erase or write, while remaining differentiable. The decay gate uses softplus instead of sigmoid because the decay factors $\alpha_t = \exp(-\exp(a) \odot \text{softplus}(\cdot))$ must be in $(0,1]$, and softplus provides a smooth, positive-valued activation that can be scaled and exponentiated.

Headed architecture and grouped-value attention: The paper uses $H = 16$ heads with $d_k = 128$ and $d_v = 128$ for the recurrent models, giving a per-layer state size of $H d_k d_v = 16 \times 128 \times 128 = 262,144$ floats per batch element. This equals $128 d_{\text{model}}$ since $d_{\text{model}} = 2048$. When grouped-value attention is used (with $H_v > H$ value heads), the key-side tensors $q$, $k$, $g$, and $b$ are repeated across the value-head group, while $v$ and $w$ already live on the value-head axis. This design reduces parameters by sharing the key-side computation across multiple value heads, which is important because the per-head state already costs $d_k d_v = 16,384$ floats per head — scaling the number of heads multiplicatively would blow up the state size.

Hybrid architecture (Figure 1, left): The hybrid model interleaves Gated DeltaNet-2 token mixers with sliding-window attention (SWA) under the same residual block structure. A repeated cell contains:

• Gated DeltaNet-2 token mixer
• MLP (standard feed-forward network)
• Sliding-window attention (SWA) with a window size of 2K tokens
• MLP (second feed-forward network)

The design rationale (Section 3.5): "Gated DeltaNet-2 compresses long histories into constant-size memory, while SWA handles exact local interactions such as short shifts, comparisons, and local retrieval." The recurrent mixer provides global context through the fixed-size state, which can in principle attend to tokens arbitrarily far in the past (albeit through a compressed representation). The sliding-window attention provides perfect local recall — within the window, every token can directly attend to every other token with softmax attention, which is superior to compressed memory for tasks requiring exact token-level comparisons (e.g., copying, local pattern matching, short-range syntactic dependencies).

With a fixed window size, the hybrid model retains linear sequence scaling because the SWA cost is $O(L \cdot W)$ for sequence length $L$ and window size $W = 2048$, which is linear in $L$ with a constant factor proportional to $W$. This follows the design pattern established by Griffin (De et al., 2024) and Samba (Ren et al., 2025), which also combine recurrent layers with local attention.

Throughput characteristics (Figure 2): The hybrid model's training throughput on an H100 GPU is measured at approximately 38.0 Kt/s (thousands of tokens per second) at sequence length 2K×8 and drops only mildly to 36.1 Kt/s at 16K×1. This near-flat scaling contrasts sharply with the Transformer, which degrades from approximately 44 Kt/s at 2K×8 to approximately 28 Kt/s at 16K×1. Relative to KDA, the throughput gap is small — the added channel-wise gates introduce a modest constant overhead per token (additional linear projections and elementwise products) but do not change the asymptotic scaling. The efficient WY-based chunkwise algorithm ensures that training speed remains dominated by the $O(C^2 d_k)$ intra-chunk matrix multiplies, which are the same cost as KDA.

Initialization (Appendix D.5): All linear layers use Xavier uniform weight initialization with a gain of $2^{-2.5}$. Biases are initialized to zero when present. This initialization scheme is inherited from the Gated DeltaNet family and is designed to control the early recurrent state magnitudes — the low gain prevents the initial state updates from being too large, which could cause instability in the recurrent dynamics before the gates learn appropriate values.

Recurrent decoding kernel (Appendix C.5): For autoregressive decoding at short sequence lengths, a separate forward-only recurrent kernel applies Eq. 10 token by token. The kernel keeps the state in fp32, multiplies it by $\exp(g_t)$, reads the decayed state along $b_t \odot k_t$, writes $w_t \odot v_t$ along $k_t$, and returns $S_t^\top q_t$. This is more efficient than chunkwise processing for single-token generation because the chunk overhead ($O(C^2)$ per chunk) is wasted when only one new token is being processed. Training always uses the chunk kernel for parallelism across the sequence dimension.

Variable-length sequences (Appendix C.6): Packed variable-length batches are represented with cumulative sequence lengths. The chunk index construction resets the recurrent state at every sequence boundary, ensuring that information from one sequence does not leak into the next. Padding is removed before the layer and restored after the output projection to avoid wasting computation on padding tokens.

4. Key Insights and Innovations

Innovation 1: The Dimensional Diagnosis — Erase and Write Operate on Different Axes and Should Not Share a Gate

The paper's most distinctive conceptual contribution is not the introduction of a new mechanism, but a diagnostic reframing of why prior delta-rule models leave performance on the table. The field's dominant assumption — carried forward from DeltaNet through Gated DeltaNet to KDA — was that a single scalar step size $\beta_t$ was the natural parameterization of the delta rule's "strength": it controls how aggressively the memory edit is applied. The scalar $\beta_t$ traces back to the original Widrow-Hoff delta rule (Widrow and Hoff, 1960) and the online gradient descent interpretation where $\beta_t$ is the learning rate for a local regression problem on the recurrent state.

Gated DeltaNet-2 identifies a dimensional mismatch that this assumption conceals. The erase operation $(I - \beta_t k_t k_t^\top)$ acts in the $d_k \times d_k$ key space — it subtracts old content along the key direction $k_t$. The write operation $\beta_t k_t v_t^\top$ acts in the $d_k \times d_v$ state space — it adds new content along key direction $k_t$ with content $v_t$. These are operations on different axes of the state matrix. The key insight is that there is no mathematical or semantic reason they must share a scalar. The scalar $\beta_t$ is an artifact of the specific online learning objective (Eq. 13 with tied gates), not a requirement of the delta rule's structure. The paper's conceptual move is to ask: what if the model needs to erase broadly from many key channels while writing selectively to only a few value channels, or vice versa? Under scalar $\beta_t$, this is impossible — a single number cannot express independent channel-level decisions on two different axes. Under channel-wise $b_t$ and $w_t$, it becomes possible, because the gates operate on their respective axes independently.

This reframes the contribution from "we added more gates" (which would be an incremental parameter-count increase) to "we identified that the existing parameterization conflated two semantically distinct operations, and separating them addresses the primary failure mode of fixed-size recurrent memory." The significance is that it opens a design space for delta-rule models: the erase and write operations can be independently parameterized, regularized, and analyzed. Future work could explore different activation functions, sparsity patterns, or auxiliary losses specific to each gate, precisely because the paper establishes them as distinct architectural primitives.

The evidence that this is not merely a parameter-count expansion comes from the ablation in Table 5: keeping channel structure only in $b_t$ (scalar $w_t$) recovers most of the full model's performance on language modeling and retrieval, whereas keeping channel structure only in $w_t$ (scalar $b_t$) recovers significantly less. This asymmetry confirms that the two gates serve functionally different roles in the state update — the erase gate's channel-wise control over which key subspaces to clean is more critical than the write gate's channel-wise control over value storage. If the gates were merely providing redundant degrees of freedom, the ablation would show symmetric contributions.

Innovation 2: The WY Absorption Trick — Channel-Wise Decay and Asymmetric Erase Factors Can Be Absorbed into a Standard Delta Recurrence

Prior work established the WY (Householder) representation as the key to efficient parallel training of delta-rule models. DeltaNet (Yang et al., 2024) showed that the product $I - \beta_r k_r k_r^\top$ (symmetric, scalar-gated) could be parallelized using the Woodbury identity. KDA (Kimi Team et al., 2025) extended this to channel-wise decay by absorbing the cumulative decay $\gamma_r$ into normalized keys and erase factors, recovering a pure delta recurrence with symmetric erase factors $\bar{e}_r = \beta_r (\gamma_r \odot \gamma_r) \odot \bar{k}_r$.

The non-obvious question for Gated DeltaNet-2 is: does the WY absorption still work when the erase factor is asymmetric ($\bar{e}_r \neq \bar{k}_r$ up to a scalar) and the write term is gate-weighted ($z_r = w_r \odot v_r$)? The paper's answer is yes, and the mathematical demonstration is itself a contribution because it shows that the WY framework is agnostic to how the erase and write factors are parameterized — it only requires that each token contributes a rank-one update $I - \bar{k}_r \bar{e}_r^\top$ to the normalized state, regardless of how $\bar{e}_r$ and $z_r$ are constructed.

The absorption trick works by defining $\bar{e}_r = \gamma_r \odot (b_r \odot k_r)$ and $z_r = w_r \odot v_r$, then proceeding with exactly the same WY machinery: form the strictly lower triangular interaction matrix $T_{rs} = \bar{e}_r^\top \bar{k}_s$, invert $I+T$ by forward substitution, compute auxiliaries $Y = A\bar{E}$ and $U = AZ$, and plug into the state and output equations. The channel-wise decay is absorbed into the asymmetric factors $\bar{k}_r = \gamma_r^{-1} \odot k_r$ and $\bar{e}_r = \gamma_r \odot e_r$ in the same way as KDA. The erase gate $b_r$ enters through $\bar{e}_r$; the write gate $w_r$ enters through $Z$. The computational cost of the WY solve — dominated by the $O(C^2 d_k)$ matrix multiplications for chunk size $C$ — remains unchanged.

This is significant because it establishes a modular interface between the gate parameterization and the training algorithm. The WY layer takes $\bar{K}$, $\bar{E}$, and $Z$ as inputs and produces the chunk-level state and output without knowing how those tensors were constructed. This means future work can experiment with arbitrary gate structures, sparsity patterns, or even learned functions of the token representation that produce $\bar{e}_r$ and $z_r$ — as long as they produce rank-one factors per token, the entire efficient training stack remains valid. The absorption trick converts what could have been an implementation barrier (asymmetric erase factors breaking the WY form) into a design principle: the WY representation is a general-purpose parallelizer for rank-one recurrent edits, not a special property of symmetric delta rules.

The evidence is in Appendix A: the derivation shows that Eq. 29 (the asymmetric recurrence with channel-wise decay) reduces to the normalized recurrence Eq. 32, which is structurally identical to the KDA normalized recurrence except that $\bar{e}_r$ is $\gamma_r \odot (b_r \odot k_r)$ rather than $\beta_r (\gamma_r \odot k_r)$. The state and output equations (Eqs. 40 and 44) retain exactly the same form as KDA. The throughput comparison in Figure 2 confirms this experimentally: the near-flat scaling from 38.0 Kt/s at 2K×8 to 36.1 Kt/s at 16K×1 for the hybrid model is competitive with KDA, confirming that the gate overhead is a small constant factor.

Innovation 3: The Gate-Aware Backward Pass Diagnostic — Why Scalar Post-Scaling Breaks and What Replaces It

The paper makes a methodological contribution to the implementation of delta-rule training by identifying precisely where and why the standard gradient computation from KDA and Gated DeltaNet fails when gates become channel-wise. This is not a mathematical discovery (the chain rule is unambiguous), but an engineering diagnostic that the paper treats as a first-class contribution because it determines whether the architecture is practically trainable.

In KDA, the scalar $\beta_t$ can be factored out of the gradient accumulation for the WY inverse $A$:

$\frac{\partial \mathcal{L}}{\partial A} = \sum_r \beta_r \frac{\partial \mathcal{L}}{\partial u_r} v_r^\top + \sum_r \beta_r \frac{\partial \mathcal{L}}{\partial y_r} (\gamma_r \odot k_r)^\top$

The scalar post-scaling trick computes the unweighted outer products $\frac{\partial \mathcal{L}}{\partial u_r} v_r^\top$ and $\frac{\partial \mathcal{L}}{\partial y_r} (\gamma_r \odot k_r)^\top$ first, then multiplies by $\beta_r$ afterward. This is valid because $\beta_r$ is a scalar that can be moved outside the outer product.

Gated DeltaNet-2's gradients are:

$\frac{\partial \mathcal{L}}{\partial A} = \sum_r \frac{\partial \mathcal{L}}{\partial u_r} (w_r \odot v_r)^\top + \sum_r \frac{\partial \mathcal{L}}{\partial y_r} (\gamma_r \odot b_r \odot k_r)^\top$

The Hadamard products $\odot$ with $w_r$ and $b_r$ occur inside the outer products. Since $w_r$ is a different vector for every row, and $b_r$ is a different vector for every row, there is no single scalar or vector that can be factored out. The gates must be baked into the dot products during accumulation, which the paper terms gate-aware accumulation. The diagnostic distinguishes between a correct implementation (Eqs. 64-65 in Appendix B.2) and an incorrect one (scalar post-scaling), and the paper explicitly flags that an incorrect backward pass would silently produce wrong gradients without runtime errors.

This matters because the efficiency of WY-based delta-rule training depends on fusing these gradient accumulations into the same kernel that computes the WY inverse vector-Jacobian product. The paper's gate-aware backward kernel (Appendix C.3) simultaneously accumulates $\text{d}A$, computes the inverse gradient $\text{d}T = -\text{tril}(A^\top \text{d}A A^\top, -1)$, and emits direct gradients $\text{d}W$, $\text{d}V$, $\text{d}B$, and $\text{d}K$ — all within a single fused Triton kernel. The practical significance is that the backward pass remains efficient: the gate-aware accumulation adds only $O(C d_k + C d_v)$ elementwise operations per chunk, which is negligible compared to the $O(C^2 d_k + C^2 d_v)$ matrix multiplies that dominate the WY inverse and auxiliary constructions. The paper's verification (Appendix D.6) that fp64 reference gradients match the chunkwise implementation to machine precision confirms correctness, and the bf16 error analysis confirms that the precision loss is at the expected mantissa level.

This contribution is incremental but practically essential. It doesn't change the mathematical form of the backward pass (it's still the chain rule), but it identifies a critical implementation detail that would break training if done naively, and it provides the fused kernel design to handle it efficiently. For practitioners implementing Gated DeltaNet-2 or similar channel-wise gated recurrences, this diagnostic is the difference between a working implementation and silently incorrect gradients.

Innovation 4: Asymmetric Channel Importance — The Erase Gate Matters More Than the Write Gate

The gate structure ablation in Table 5 produces a finding that is conceptually revealing beyond its raw performance numbers: the erase gate $b_t$ accounts for most of the model's gain, while the write gate $w_t$ contributes modestly. The "b-only" variant (channel $b_t$, scalar $w_t$) achieves 52.79 common-sense average and 35.2% on MK-NIAH-1 at 4K, compared to full Gated DeltaNet-2's 53.11 and 37.8% — recovering most of the gain. The "w-only" variant (scalar $b_t$, channel $w_t$) achieves only 52.45 and 30.6% — significantly trailing the full model on retrieval.

This asymmetry is not obvious a priori. One might expect that selectively writing value channels (e.g., suppressing noisy or irrelevant value dimensions) would be equally or more important than selectively erasing key channels. The results suggest the opposite: controlling which key subspaces are erased is the higher-leverage operation for fixed-size recurrent memory.

The paper explains this through the structure of the state update (Eq. 10). The erase gate $b_t$ changes the erase factor $k_t(b_t \odot k_t)^\top$, which directly shapes the spectrum of the state transition matrix $I - k_t e_t^\top$. This matrix governs how quickly existing information decays in different key subspaces — channels where $b_{t,i}$ is high experience strong subtraction of old content, while channels where $b_{t,i}$ is low preserve their content. The erase gate therefore controls the temporal dynamics of memory: which information persists and which is cleared.

The write gate $w_t$, by contrast, only affects the additive term $k_t(w_t \odot v_t)^\top$, which adds new information but does not change how existing information evolves. Its role is content selection: which value dimensions of the incoming token are worth storing. While this is useful (the "w-only" variant does outperform the pure scalar-gated baselines on some metrics), it is secondary to controlling retention dynamics.

This finding has design implications for future delta-rule models. If erase is the higher-leverage operation, future work could (a) allocate more parameters to the erase gate projection (e.g., a deeper MLP), (b) explore more expressive erase gate structures (e.g., low-rank rather than diagonal), (c) regularize the erase gate to encourage structured sparsity, or (d) simplify the write gate to reduce parameter count without significant performance loss. The paper's expanded erase range experiment ($b_t \in [0,2]^{d_k}$) tests one such direction — allowing negative eigenvalues — but finds no consistent gain at 1.3B scale. This negative result (Table 5, bottom row) is itself informative: it suggests that the $[0,1]$ range is sufficient for the erase operation at this scale, and that the main benefit comes from channel-wise control rather than expanded dynamic range.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All models are trained on 100B tokens sampled from FineWeb-Edu (Penedo et al., 2024). Language modeling quality is evaluated on WikiText (Merity et al., 2017) and LAMBADA (Paperno et al., 2016) using perplexity. Zero-shot transfer is assessed on LAMBADA accuracy and a commonsense reasoning suite spanning PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2020), ARC-Easy and ARC-Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), Social IQa (Sap et al., 2019), and BoolQ (Clark et al., 2019). In-context retrieval is evaluated on synthetic tasks from RULER (Hsieh et al., 2024) — specifically S-NIAH-1 (passkey retrieval), S-NIAH-2 (numerical needle), S-NIAH-3 (word-based needle), and MK-NIAH-1 (multi-key with distractors) — and on real-world recall tasks from Arora et al. (2024) including SWDE (structured web extraction), FDA (PDF key-value retrieval), SQuAD, TriviaQA, DROP, and Natural Questions.

• Base model(s). All models are trained at 1.3B parameters from scratch on the 100B-token FineWeb-Edu corpus. The paper evaluates its own Gated DeltaNet-2 architecture in both recurrent-only and hybrid (recurrent + sliding-window attention) configurations, and compares against equivalently-sized Transformer, Mamba-2 (Dao and Gu, 2024), Gated DeltaNet (Yang et al., 2025), KDA (Kimi Team et al., 2025), and Mamba-3 variants. The 1.3B scale is chosen as a representative mid-range size that allows systematic comparison across architectures under controlled training conditions. To ensure fair comparison, all recurrent models are matched on both total parameter count and main recurrent state size: Gated DeltaNet, KDA, and Gated DeltaNet-2 use 16 heads with $d_k = 128$ and $d_v = 128$ for a per-layer state of $16 \times 128 \times 128 = 262,144$ floats; Mamba-2 and Mamba-3 use expansion factor 2 and head dimension 64 with $d_{\text{state}} = 64$ for the same $4096 \times 64 = 262,144$ float state size.

• Metrics. Perplexity is reported for language modeling on WikiText and LAMBADA. Accuracy is reported as percentage correct for LAMBADA zero-shot and all commonsense reasoning tasks, with an aggregate "Avg." computed over LAMBADA accuracy and the nine reasoning accuracies. Retrieval tasks are evaluated as accuracy (percentage of correctly retrieved needles or answers). Training throughput is measured in thousands of tokens per second (Kt/s) on a single H100 GPU under a fixed token budget.

• Baselines. The paper compares against six baseline architectures. Transformer: standard softmax self-attention with the same 1.3B parameter count, trained identically. Mamba-2 (Dao and Gu, 2024): data-dependent scalar decay with additive key-value writes, no delta rule. Gated DeltaNet (Yang et al., 2025): scalar decay and scalar delta gate combining global forgetting with delta-rule editing. KDA (Kimi Team et al., 2025): channel-wise decay with scalar delta gate, representing the immediate predecessor. Mamba-3 SISO (Lahoti et al., 2026): exponential-trapezoidal discretization with complex-valued state transitions in single-input single-output form. Mamba-3 MIMO (Lahoti et al., 2026): same as SISO but with multi-input multi-output formulation at rank $R = 4$. Each baseline is evaluated in both recurrent-only and hybrid configurations (paired with sliding-window attention where applicable).

• Generation budget / compute accounting. Compute is equated by matching total model parameters (1.3B for all models) and training on exactly the same 100B tokens from FineWeb-Edu under identical training hyperparameters. All models use the same recipe: AdamW optimizer with peak learning rate $4 \times 10^{-4}$, weight decay 0.1, gradient clipping at 1.0, cosine annealing, 1B-token warm-up, global batch size of 0.5M tokens, and training sequence length of 4K tokens. Hybrid models use a 2K sliding-window attention size. Throughput comparisons (Figure 2) use a fixed token budget and measure Kt/s on an H100 GPU at different sequence length × batch size configurations.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. Results are reported as single-run evaluations at the end of training. All models are trained once under the fixed recipe, with no mention of multiple seeds, standard deviations, or confidence intervals. This is a limitation: the differences between methods — particularly in the commonsense reasoning averages where the spread is narrow (e.g., 52.07 for Gated DeltaNet vs. 53.11 for Gated DeltaNet-2 in the recurrent setting) — may not be statistically robust. The ablation studies (Table 5, Appendix K) similarly report single-run results. The throughput measurements in Figure 2 are presumably averaged over multiple runs, but the paper does not state this explicitly.

Main Quantitative Results

Language Modeling and Commonsense Reasoning

Table 2 reports the core language modeling and zero-shot transfer results across all architectures in both recurrent and hybrid settings. Gated DeltaNet-2 achieves the best average accuracy in both model families. In the recurrent-only setting, Gated DeltaNet-2 reaches 53.11% average (computed over LAMBADA accuracy and nine reasoning benchmarks), compared to 52.39% for Mamba-3 MIMO, 52.28% for KDA, 52.07% for Gated DeltaNet, and 51.82% for Mamba-2. The margin over KDA (0.83 percentage points) is the most direct comparison since KDA is the immediate predecessor that shares channel-wise decay but uses a scalar delta gate.

The perplexity advantage follows the same pattern. On WikiText, Gated DeltaNet-2 achieves 15.90 in the recurrent setting, versus 16.81 for KDA, 16.40 for Gated DeltaNet, and 16.45 for Mamba-3 MIMO — an improvement of 0.91 perplexity points over KDA. On LAMBADA perplexity, Gated DeltaNet-2 reaches 11.41 versus 11.68 for KDA and 11.66 for Mamba-3 MIMO. These perplexity gains are consistent across both evaluation sets, suggesting a genuine improvement in next-token prediction quality rather than a benchmark-specific artifact.

The LAMBADA accuracy improvement is notable: Gated DeltaNet-2 achieves 48.09% in the recurrent setting versus 48.13% for KDA and 49.62% for Gated DeltaNet. Here Gated DeltaNet actually leads, but the differences are small (within 1.5 percentage points). The commonsense reasoning suite shows a clearer pattern: Gated DeltaNet-2 leads on PIQA (72.80%), HellaSwag (56.84%), WinoGrande (57.85%), and ARC-Easy (72.43%), while trailing Gated DeltaNet slightly on ARC-Challenge (38.23% vs. 35.15% for Gated DeltaNet and 38.07% for Mamba-3 MIMO). The Social IQa and BoolQ results are roughly middle-of-the-pack.

In the hybrid setting, the pattern is similar but the margins shift. Gated DeltaNet-2 achieves 53.97% average, versus 52.72% for Mamba-3 MIMO, 52.69% for Mamba-3 SISO, 52.68% for KDA, 52.25% for Gated DeltaNet, and 51.99% for Mamba-2. The gain over KDA is 1.29 percentage points — slightly larger than in the recurrent setting. The hybrid Transformer achieves 50.86%, trailing all recurrent-based hybrids, which is expected given that the Transformer at 1.3B parameters trained on 100B tokens is under-trained relative to the data scale at which Transformers typically excel.

Key takeaway from Table 2: Gated DeltaNet-2 is the strongest overall architecture across both recurrent and hybrid settings, but the margins are modest — typically 0.5–1.3 percentage points on the average over Gated DeltaNet and KDA. The gains are most consistent on perplexity (lower is better) and on reasoning tasks requiring broader context (PIQA, HellaSwag, WinoGrande), while LAMBADA accuracy and some individual reasoning benchmarks show mixed results. Since the recurrent state size is matched across all delta-rule and SSM models, the improvement must come from the update rule itself (decoupling erase and write) rather than from a larger memory capacity.

In-Context Retrieval on Synthetic Data (RULER)

Table 3 reports accuracy on the RULER needle-in-a-haystack benchmarks across context lengths from 1K to 8K tokens (with 4K for S-NIAH-3 and MK-NIAH-1 limited to 4K). This is the centerpiece evaluation for the paper's central claim that decoupling erase and write directly targets interference in fixed-state memory.

S-NIAH-1 (passkey retrieval): At 8K context, the recurrent-only setting shows a dramatic divergence. Gated DeltaNet-2 achieves 97.8%, KDA achieves 70.6%, Gated DeltaNet reaches 97.6%, Mamba-3 MIMO achieves 35.6%, and Mamba-2 achieves 55.8%. The near-perfect scores at 1K and 2K across all methods confirm that the difficulty is not the retrieval task itself but the interference from long context. Gated DeltaNet-2, Gated DeltaNet, and KDA all maintain strong performance at 4K, but at 8K, KDA degrades substantially (70.6%) while Gated DeltaNet-2 remains near-perfect (97.8%). This is the paper's strongest single-result evidence that the scalar $\beta_t$ in KDA becomes a bottleneck at long context lengths, and that channel-wise erase control alleviates it. In the hybrid setting, all methods perform well (97–100% at all lengths), likely because the sliding-window attention can directly attend to the needle without relying on the compressed state.

S-NIAH-2 (numerical needle): At 8K context in the recurrent setting, Gated DeltaNet-2 achieves 39.2%, versus 30.6% for KDA, 32.0% for Gated DeltaNet, 27.2% for Mamba-3 MIMO, and 21.0% for Mamba-2. At 4K, the ordering is similar: 93.0% (Gated DeltaNet-2), 89.0% (KDA), 87.2% (Gated DeltaNet), 64.2% (Mamba-3 MIMO), 62.6% (Mamba-2). The S-NIAH-2 task is harder than S-NIAH-1 because the needle is a numerical value rather than a distinctive passkey, so the model must more precisely retrieve the correct value from among similar-looking numbers. The gap between Gated DeltaNet-2 and KDA widens from 4.0 points at 4K to 8.6 points at 8K, consistent with the paper's claim that the erase-write decoupling is most beneficial as context length — and thus interference — grows.

S-NIAH-3 (word-based needle): At 2K context (the longest evaluated), Gated DeltaNet-2 achieves 89.8% in the recurrent setting versus 63.2% for KDA and 54.2% for Gated DeltaNet — a substantial 26.6-point margin over KDA. At 4K, all methods degrade significantly: 31.8% (Gated DeltaNet-2), 26.2% (KDA), 60.6% (Gated DeltaNet). The unusually high score for Gated DeltaNet at 4K (60.6% vs. 31.8% for Gated DeltaNet-2) is an anomaly in the trend and may reflect variance from the small evaluation set. In the hybrid setting, Gated DeltaNet-2 leads at both 2K (99.0%) and 4K (55.6%), with Mamba-3 MIMO second at 4K (54.2%).

MK-NIAH-1 (multi-key with distractors): This is the most challenging and most diagnostic task. The model must store several key-value pairs, distinguish the requested key from distractor keys, and retrieve the correct value — all while managing interference from the distractors and the long context. In the recurrent setting at 4K, Gated DeltaNet-2 achieves 37.8%, versus 28.0% for KDA, 27.8% for Gated DeltaNet, 21.4% for Mamba-2, and 36.2% averaged across the three Mamba-3 variants (which range from 18.0% to 20.2% at the lower end to 27.2–29.2% for the remaining). The 9.8-point gap over KDA is the largest relative improvement on any task in Table 3. At 2K, the margin is 7.2 points (51.4% vs. 44.2%), and at 1K it is 18.6 points (72.6% vs. 54.0%). In the hybrid setting, Gated DeltaNet-2 leads at all lengths: 93.0% at 1K, 84.6% at 2K, and 48.0% at 4K. Mamba-3 MIMO is second at 4K with 46.6%, and KDA is third at 40.4%.

The MK-NIAH-1 results are the paper's strongest empirical evidence for its central mechanism claim. The task requires precisely the capability that decoupled erase and write provides: the model must erase competing distractor associations to reduce interference (using the erase gate $b_t$ to select which key subspaces to clean) while discriminating which value channels of the correct answer to commit (using the write gate $w_t$). The paper does not provide mechanistic interpretability analysis to confirm that this is how the gates are actually being used — the evidence is correlational (performance on the diagnostic task) rather than causal (verified internal behavior). The ablation in Table 5 supports the interpretation (the erase gate accounts for most of the gain), but does not directly demonstrate that the gates are performing interference management as hypothesized.

In-Context Retrieval on Real-World Tasks

Table 4 reports accuracy on six real-world retrieval tasks with input length truncated to 2K tokens. These tasks are less controlled than the synthetic NIAH benchmarks but better reflect fixed-state memory performance under realistic query patterns.

In the recurrent setting, Gated DeltaNet-2 achieves the highest average at 29.88%, compared to 28.67% for KDA, 28.35% for Mamba-3 MIMO, 28.09% for Gated DeltaNet, and 26.84% for Mamba-2. The margin over KDA (1.21 points) is smaller than on the synthetic tasks, which is expected because real-world retrieval tasks involve recall of genuinely informative content (not randomly placed distractor needles), and the memory pressure from a 2K context is moderate.

The per-task breakdown reveals an uneven pattern. Gated DeltaNet-2 leads strongly on SWDE (23.65% vs. 22.49% for KDA and 17.24% for Mamba-2) — SWDE involves structured extraction from HTML, which requires precise key-value association recovery, exactly the delta rule's strength. SQuAD shows a small advantage (36.75% vs. 36.65% for Mamba-3 MIMO and 35.10% for KDA). TriviaQA shows a larger gap (61.37% vs. 58.89% for Mamba-3 SISO and 58.12% for KDA). However, the model trails on DROP (17.87% vs. 21.80% for KDA and 21.32% for Mamba-3 SISO) — DROP requires discrete reasoning over paragraphs, and the gap suggests that stronger memory editing does not necessarily translate to better multi-step reasoning when the base model's reasoning capability is the bottleneck. Natural Questions similarly shows a deficit (19.64% vs. 20.16% for Gated DeltaNet).

In the hybrid setting, Gated DeltaNet-2 achieves 42.28% average, versus 41.01% for Mamba-3 SISO, 40.14% for KDA, 40.11% for Mamba-3 MIMO, and 39.11% for Gated DeltaNet. The gains are again modest (1.27 points over Mamba-3 SISO, 2.14 over KDA). The SWDE advantage is substantial (41.96% vs. 39.83% for KDA and 32.21% for Transformer), and TriviaQA is strong (62.38% vs. 60.60% for Gated DeltaNet). The remaining NQ and DROP gaps narrow in the hybrid setting — SWA presumably supplies the local evidence aggregation that the recurrent memory compressor struggles to preserve.

The paper notes that "the remaining NQ and DROP gaps point to formats that also need local evidence aggregation, which SWA supplies in the hybrid model." This interpretation aligns with the hybrid setting results where NQ reaches 26.31% (matching the leader) and DROP reaches 23.67% (versus 24.68% for Mamba-2 hybrid). The recurrent setting weakness on these tasks is consistent with the known limitation of delta-rule models: the compressed state is excellent at associative recall of discrete key-value pairs but loses the token-level detail needed for extractive question answering where the answer must be precisely located in the source text.

Training Throughput

Figure 2 reports single-H100 training throughput for the hybrid 1.3B models. At sequence length 2K×8, Gated DeltaNet-2 achieves approximately 38.0 Kt/s, slightly below Mamba-2 (approximately 39.5 Kt/s), Mamba-3 SISO (approximately 39.0 Kt/s), and Gated DeltaNet (approximately 38.5 Kt/s), and roughly comparable to KDA (approximately 38.0 Kt/s). At 16K×1, Gated DeltaNet-2 achieves approximately 36.1 Kt/s — a drop of only ~4.8% from the 2K×8 configuration. The Transformer degrades from approximately 44 Kt/s at 2K×8 to approximately 28 Kt/s at 16K×1 — a ~36% drop.

The near-flat scaling profile of all recurrent models (Mamba-2 through Gated DeltaNet-2) confirms that the chunkwise WY algorithm with fixed chunk size $C = 64$ keeps the cost linear in sequence length, and that the gate overhead in Gated DeltaNet-2 adds only a small constant factor. The small gap between Gated DeltaNet-2 and KDA — less than 1 Kt/s at most configurations — reflects the cost of the additional erase and write gate projections and the gate-aware backward accumulation, but does not change the asymptotic complexity. This is consistent with the paper's claim that "Gated DeltaNet-2 retains practical training efficiency while paying a modest constant cost for finer memory control."

Ablation Studies and Robustness Checks

Channel structure of erase and write gates (Table 5): The "w-only" variant (scalar $b_t$, channel-wise $w_t$) achieves a common-sense average of 52.45% versus 53.11% for full Gated DeltaNet-2 — a drop of 0.66 points. The "b-only" variant (channel-wise $b_t$, scalar $w_t$) achieves 52.79% — a drop of only 0.32 points and recovering most of the full model's performance. On MK-NIAH-1 at 4K, the difference is starker: "w-only" achieves 30.6% (a 7.2-point drop from the full model's 37.8%), while "b-only" achieves 35.2% (a 2.6-point drop). This confirms that the erase gate $b_t$ accounts for the majority of the retrieval gain, while the write gate $w_t$ contributes meaningfully but secondarily. The asymmetry is consistent with the paper's theoretical framing: the erase gate changes the state transition matrix (controlling information retention dynamics), while the write gate only affects the additive write term (controlling content selection at insertion time). Controlling retention is the higher-leverage operation for managing interference in fixed-size memory.

Erase gate range (Table 5): Expanding the erase gate range from $b_t \in [0,1]^{d_k}$ to $b_t \in [0,2]^{d_k}$ (enabling negative eigenvalues in the state transition matrix, following Grazzi et al., 2025) produces no consistent gain: WikiText perplexity 15.95 vs. 15.90, common-sense average 53.04 vs. 53.11, S-NIAH-2 at 4K 93.1% vs. 93.0%, S-NIAH-3 at 2K 89.4% vs. 89.8%, MK-NIAH-1 at 4K 37.6% vs. 37.8%. All differences are within 0.2 points or less, and the direction is inconsistent (some metrics slightly favor the expanded range, some the restricted range). This is a negative result that suggests the $[0,1]$ range is sufficient for the erase operation at the 1.3B scale. It does not rule out benefits at larger scales or on different tasks, but at this configuration, the channel-wise control matters more than the expanded dynamic range.

Chunk size and numerical precision choices (Appendix D): The paper verifies the chunkwise forward pass against a tokenwise recurrent reference for random configurations (Appendix D.6). In fp64, gradients for Q, K, V, B, W, the log-decay, and the initial state "agree to machine precision." In production fp32, differences are "at the expected tensor-core accumulation noise level." In bf16, the error "follows the bf16 mantissa." These checks confirm correctness of the WY formulation and the gate-aware backward pass, but do not constitute an ablation — they are verification, not sensitivity analysis.

Decay precision (Appendix D.1): The log-decay is computed in explicit fp32 before entering the kernels because "a low precision mantissa can perturb long products of decays even when each tokenwise gate is small." The paper does not provide an ablation comparing fp32 vs. bf16 decay computation, but the justification is reasonable: the cumulative decay over thousands of tokens is path-length-dependent, and precision loss in the log-sum would compound multiplicatively after exponentiation.

Recurrent state size matching (Appendix E.1): The experimental design explicitly matches recurrent state size across architectures: delta-rule models use $H d_k d_v = 16 \times 128 \times 128 = 262,144$ floats, while Mamba-2 and Mamba-3 use $(2 \cdot d_{\text{model}}) \cdot d_{\text{state}} = 4096 \times 64 = 262,144$ floats. This is an important control that ensures performance differences are attributable to the update rule rather than raw memory capacity. However, it does not control for the effective memory utilization — a state with $d_k = 128, d_v = 128$ may have different information-theoretic capacity than one with $d_k = 64, d_v = 64$, even if the total number of floats is identical. The paper does not discuss this subtlety.

Mamba-3 MIMO rank (Appendix E.1): The MIMO variant uses rank $R = 4$ following the original paper's recommendation. No ablation over different MIMO ranks is provided. This means the Mamba-3 results represent a single configuration rather than the best possible Mamba-3 performance.

ReST$^{EM}$ revision model ablation (Appendix K, Figure 16): The paper briefly mentions an experiment with a ReST$^{EM}$-optimized revision model (a different training pipeline for iterative revisions, not the Gated DeltaNet-2 architecture) and reports that "additional sequential revisions substantially hurt performance" and that "the on-policy data collection in ReST$^{EM}$ exacerbates spurious correlations in revision data." This is a negative result for the revision training recipe, but is included presumably because it was attempted as an alternative direction. It is not directly relevant to the Gated DeltaNet-2 architecture evaluation and is not in the main text.

Missing ablations: Several experiments that would have strengthened the paper are absent. There is no ablation over the key dimension $d_k$ or value dimension $d_v$ to test whether the benefit of channel-wise gates scales with head dimension (i.e., does a larger $d_k$ at fixed state size amplify the advantage?). There is no ablation over the number of heads at fixed total state size. There is no ablation over the chunk size $C = 64$ — one could test whether larger chunks (which change the ratio of intra-chunk parallelism to inter-chunk recurrence) favor or disadvantage the gated architecture. There is no ablation over training data scale — it would be informative to see whether the gap between Gated DeltaNet-2 and KDA grows or shrinks with more training tokens. There is no ablation over model scale — all experiments are at 1.3B parameters, and the paper does not test whether the erase-write decoupling advantage persists or changes at smaller (e.g., 300M) or larger (e.g., 7B) scales.

Critical Assessment

The paper makes one central empirical claim and several supporting ones. The central claim, as stated in the abstract and Section 1, is that decoupling erase and write in the delta rule produces the strongest overall results across language modeling, commonsense reasoning, and retrieval at the 1.3B scale, with the advantage most pronounced on long-context RULER benchmarks. I evaluate each claim against the reported experiments.

The retrieval advantage is genuine and well-supported, but the effect size varies dramatically across tasks. The strongest evidence comes from the RULER tasks in Table 3, particularly MK-NIAH-1 where Gated DeltaNet-2 achieves 37.8% at 4K versus KDA's 28.0% — a relative improvement of 35%. S-NIAH-3 at 2K shows 89.8% versus 63.2%, a 42% relative improvement. These are substantial margins that are unlikely to be noise, even with single-run evaluations. The real-world retrieval results in Table 4 are weaker — the average improvement over KDA is 1.21 points in the recurrent setting (29.88% vs. 28.67%), and three of six tasks show gains while two show deficits (DROP, NQ). The retrieval claim holds conditionally: it is strongest on synthetic multi-key interference tasks and weaker on real-world extractive QA where local token-level evidence aggregation matters more than associative memory editing. The paper acknowledges the DROP/NQ deficits and attributes them to a need for local evidence aggregation, which the hybrid model's sliding-window attention partially addresses (Table 4, hybrid setting).

The language modeling and commonsense reasoning advantage is modest and its statistical reliability is uncertain. Table 2 shows Gated DeltaNet-2 at 53.11% average versus KDA at 52.28% — a 0.83-point difference. The individual benchmark scores are a mix of wins and losses: Gated DeltaNet-2 wins on PIQA, HellaSwag, WinoGrande, ARC-Easy, and BoolQ, but loses on LAMBADA accuracy (48.09% vs. 48.13% for KDA and 49.62% for Gated DeltaNet) and OpenBookQA (31.60% vs. 30.40% for KDA — a 1.2-point gain — but trailing Gated DeltaNet's 30.20% by an even larger margin). With a 500-question test set for LAMBADA and similar small sets for the reasoning benchmarks, single-percentage-point differences across 10 benchmarks can easily arise from variance. The paper reports no standard deviations, no multiple seeds, and no significance tests. The perplexity improvements (15.90 vs. 16.81 on WikiText, 11.41 vs. 11.68 on LAMBADA) are more consistent and less likely to be noise given that perplexity aggregates over many tokens, but the paper does not report whether these differences are computed over the full test set or a subset, and whether the differences are stable across evaluation runs.

The paper does not demonstrate that the gates are actually being used for their hypothesized purpose. The claim that decoupling erase and write "directly targets the primary pressure point of fixed-state recurrence — interference among many compressed associations" is supported only by the correlational evidence from retrieval benchmarks. There is no mechanistic interpretability analysis: no visualization of gate activation patterns, no analysis of whether the erase gate learns to selectively protect certain key channels during context shifts, no demonstration that the write gate suppresses value channels that encode distractor information. The ablation in Table 5 shows that the erase gate matters more than the write gate, which is consistent with the hypothesis, but it could also be explained by the erase gate simply providing more useful degrees of freedom for the state transition dynamics independent of any "interference management." This is a significant gap between the paper's motivating narrative and its empirical evidence. The strong synthetic retrieval results are suggestive but not conclusive.

The throughput measurements confirm efficiency but the comparison is incomplete. Figure 2 shows that Gated DeltaNet-2's throughput is competitive with KDA and Mamba-3 (within ~1 Kt/s at most configurations). This supports the claim that the gate overhead is modest. However, the throughput comparison is at a single model scale (1.3B) and single GPU (H100). There is no analysis of how the overhead scales with model size — do the additional gate projections become a smaller fraction of total compute at larger scales (favorable) or do they interact poorly with memory bandwidth constraints (unfavorable)? The paper also does not report inference latency for autoregressive decoding, which is arguably more important for deployment than training throughput. The recurrent decoding kernel is described in Appendix C.5 but no latency numbers are provided.

The matched state size control is necessary but may be misleading. The paper matches recurrent state size at 262,144 floats across all models, which is correct for isolating the effect of the update rule. However, the state structure differs: delta-rule models store a $d_k \times d_v$ matrix (128 × 128), while SSMs store a vector of length $2 \cdot d_{\text{model}} \cdot d_{\text{state}}$ (4096 × 64). These are different tensor shapes with potentially different information-theoretic capacities for the same total number of floats. The paper also matches total parameter count at 1.3B, but does not provide a breakdown of parameter allocation across components. It's possible that Gated DeltaNet-2's gate projections ($W_b$ and $W_w$) consume parameters that could otherwise go to deeper layers or wider FFNs in the baselines, and the paper's matching protocol may inadvertently advantage Gated DeltaNet-2 if the gates are highly parameter-efficient relative to the baseline architectures' parameter usage.

The single-scale, single-seed evaluation limits generalizability. All experiments are at 1.3B parameters trained on 100B tokens from a single dataset (FineWeb-Edu) for exactly one training run each. There is no evidence that the ranking of architectures would persist at 7B, 13B, or 70B parameters; at 1T tokens; on different data distributions (code, multilingual, conversational); or with different training hyperparameters. The paper includes no scaling law analysis or even a single smaller-scale replication. The finding that the expanded erase range $[0,2]$ produces no gain at 1.3B (Table 5) could reverse at larger scales where the model has more capacity to exploit negative eigenvalues. Similarly, the write gate's modest contribution at 1.3B could grow at larger scales where the value dimension carries more semantic structure.

Key experiments that would have strengthened the paper but were not run:

• Multiple training seeds with standard deviations to assess whether the reported differences are statistically significant, particularly for the commonsense reasoning benchmarks where margins are under one percentage point.
• Scaling across at least one additional model size (e.g., 300M or 7B) to test whether the erase-write decoupling advantage scales with model capacity.
• Mechanistic interpretability analysis of gate behavior: what patterns do $b_t$ and $w_t$ learn? Do they correlate with context boundaries? Do they specialize across heads?
• An ablation where $b_t$ and $w_t$ share a projection layer (fewer added parameters) to test whether the gate parameter count rather than the gate structure drives the improvement.
• Inference latency benchmarks for autoregressive decoding with the recurrent kernel.
• Evaluation on additional long-context benchmarks beyond RULER (e.g., LongBench, SCROLLS) to test generalizability of the retrieval advantage.
• An experiment replacing the channel-wise gates with a low-rank structure (e.g., $b_t = U_b \sigma(W_b x_t)$ where $U_b$ has rank $r < d_k$) to test whether full channel-wise freedom is necessary or whether a compressed gate parameterization suffices.

Summary of what the experiments do and do not establish: The experiments establish that at 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 (a) achieves modest improvements in language modeling perplexity and commonsense reasoning over KDA and Gated DeltaNet, (b) substantially outperforms all baselines on the interference-heavy MK-NIAH-1 and S-NIAH-3 tasks from RULER, particularly as context length grows, (c) shows smaller but still positive gains on real-world retrieval, (d) maintains training throughput competitive with KDA and Mamba-3, and (e) demonstrates through ablation that the erase gate contributes more than the write gate. What they do not establish is whether the performance advantage is statistically significant, whether it generalizes to other scales or data distributions, whether the gates actually perform the interference-management role hypothesized by the paper's motivation, or whether the advantage persists under stricter parameter-count controls (e.g., giving baselines equivalent additional parameters). The retrieval results are the strongest part of the empirical case; the commonsense reasoning results are suggestive but inconclusive as presented.

6. Limitations and Trade-offs

Single Scale, Single Training Run, No Statistical Replication

The assumption or constraint. The entire empirical case for Gated DeltaNet-2 rests on one model size (1.3B parameters), one training data scale (100B FineWeb-Edu tokens), and one training run per architecture. The paper reports no standard deviations, confidence intervals, or multiple random seeds for any result in Tables 2–5 or Figure 2. This is not acknowledged as a limitation by the authors — the paper presents results as point estimates without qualification.

The consequence. The modest margins on language modeling and commonsense reasoning become difficult to interpret. In the recurrent setting (Table 2), Gated DeltaNet-2 achieves 53.11% average versus KDA's 52.28% — a difference of 0.83 percentage points across 10 benchmarks, several of which have test sets of only a few hundred examples (e.g., LAMBADA accuracy is evaluated on 500 questions; ARC-Challenge has even fewer). With a single training run per model, it is impossible to distinguish a genuine architectural improvement from training variance. A practitioner deciding between KDA and Gated DeltaNet-2 based on these numbers cannot assess whether the 0.83-point advantage would persist under retraining with a different random seed, different data order, or different initialization. The retrieval results on RULER (Table 3) show larger margins — MK-NIAH-1 at 4K is 37.8% vs. 28.0% for KDA — which are less likely to be noise, but even here, a second training run could shift absolute scores by several percentage points. The ablation study (Table 5) reports single-run differences of 0.3–0.7 points on the commonsense average, which are well within plausible seed-to-seed variation.

What evidence exists in the paper. None. The paper reports no multiple-seed experiments, no error bars, no statistical tests. The training recipe is described as a single run per model (Section 4, Appendix E.1). The absence of any replication is a methodological gap that affects the strength of every quantitative claim.

Mitigation status. Not addressed. The paper does not mention this as a limitation and proposes no future work to address it. For a model architecture paper at this scale, 1.3B parameters with 100B tokens is a non-trivial training budget — running multiple seeds for all six architectures would be expensive. However, running even 2–3 seeds for the top two competitors (Gated DeltaNet-2 vs. KDA) on the key metrics would substantially strengthen the claims.

---

No Demonstration That the Gates Perform the Hypothesized Mechanism

The assumption or constraint. The paper's central motivation — stated in the abstract, introduction, and throughout Section 3 — is that a scalar $\beta_t$ cannot simultaneously control erasing and writing because these operations "act on different axes of the state," and that decoupling them through $b_t$ and $w_t$ "directly targets the primary pressure point of fixed-state recurrence — interference among many compressed associations." This is an architectural hypothesis about why the gates should help, which implies specific gate behaviors: the erase gate $b_t$ should selectively clear key channels when the context shifts or when old associations become stale, while the write gate $w_t$ should suppress value channels that encode noisy or distractor information.

The consequence. Without evidence that the gates learn these behaviors, the performance gains could be explained by other factors. The channel-wise gates add parameters (two additional linear projections $W_b$ and $W_w$) that could simply provide more model capacity — the improvement might come from having more parameters rather than from the specific erase-write decoupling structure. Alternatively, the gates might learn patterns unrelated to interference management (e.g., the erase gate might act as an additional key modulation for the state transition dynamics, and the write gate might act as an additional value modulation for content filtering — both useful, but neither necessarily performing the "targeted interference reduction" the paper claims). The gate structure ablation (Table 5) shows that the erase gate matters more than the write gate, which is consistent with the hypothesis but could equally be explained by (a) the erase projection adding more useful parameters on the key axis than the write projection adds on the value axis, (b) the erase gate affecting the spectral radius of the state transition matrix (a dynamic property) while the write gate only affects the additive term (a static property), making the erase gate inherently higher-leverage regardless of interference.

What evidence exists in the paper. The paper provides only correlational evidence. MK-NIAH-1 performance improves substantially with the decoupled gates (Table 3), and the ablation shows this benefit is driven primarily by the erase gate (Table 5). These results are consistent with the interference-management hypothesis but do not distinguish it from the alternative hypothesis that channel-wise key modulation is simply more expressive than scalar key modulation, independent of interference. The paper contains no analysis of gate activation patterns — no visualization of how $b_t$ and $w_t$ vary across tokens, across context positions, or across heads. There is no experiment where distractor content is systematically varied and gate behavior is tracked. There is no analysis of whether the erase gate activates differently at context boundaries versus mid-context, or whether the write gate suppresses value channels that encode distractor-key information.

Mitigation status. Not addressed. The paper does not acknowledge this gap between mechanistic claim and mechanistic evidence. Section 8 (Conclusion) reiterates the interference-management interpretation without caveat. A practitioner considering whether the architecture is worth adopting for their application — particularly one where interference patterns differ from the RULER benchmarks — has no information about whether the gates have learned generalizable interference-management behavior or have simply provided additional capacity that happens to help on these specific tasks. This is the most significant gap between the paper's narrative and its evidence.

---

Difficulty Estimation Cost Is Not Factored into the Efficiency Gains

The assumption or constraint. This limitation is specific to the broader context of test-time compute allocation papers, and is explicitly flagged in the reference example as a pattern to identify. In Gated DeltaNet-2, the analogous concern is that the channel-wise gates add parameters and computation that are not accounted for in the matched comparisons, but this is a minor point. The more consequential version of this limitation pattern for Gated DeltaNet-2 is: the paper does not report inference latency for autoregressive decoding, which is the deployment mode where recurrent models' constant-memory property is most valuable.

The consequence. Training throughput (Figure 2) shows that Gated DeltaNet-2 is competitive with KDA and Mamba-3 during parallel chunkwise training. However, during autoregressive decoding — where tokens are generated one at a time and the recurrent state is updated sequentially — the overhead picture may differ. The recurrent decoding kernel (Appendix C.5) applies the Gated Delta Rule-2 update token by token, which includes: computing $g_t$, $b_t$, and $w_t$ from three additional linear projections; multiplying the state by $\exp(g_t)$; reading along $b_t \odot k_t$; writing $w_t \odot v_t$; and returning $S_t^\top q_t$. These additional elementwise operations and projections may be a larger fraction of the per-token cost during decoding than during training, because decoding is memory-bandwidth-bound rather than compute-bound — the state read/write dominates, and the additional gate projections compete for memory bandwidth. For deployment scenarios where latency matters (interactive assistants, real-time applications), a small constant-factor increase in per-token decoding time could be significant, especially if it does not come with a correspondingly large quality improvement for the specific task. The paper provides no data to assess this tradeoff.

What evidence exists in the paper. None. The paper reports only training throughput (Figure 2) and describes the recurrent decoding kernel (Appendix C.5) without latency numbers. There is no comparison of tokens-per-second during autoregressive generation across architectures, no measurement of time-to-first-token or per-token latency, and no discussion of how the gate overhead interacts with memory bandwidth constraints during decoding.

Mitigation status. Not addressed. The paper does not acknowledge decoding latency as a factor worth reporting. For a recurrent architecture whose primary deployment advantage is constant-memory, low-latency decoding (compared to the KV-cache growth of Transformers), the absence of any decoding benchmarks is a significant gap. A practitioner choosing between Gated DeltaNet-2 and KDA for a latency-sensitive application cannot assess whether the quality gains justify the additional decoding overhead without this data.

---

Evaluation Limited to a Single Data Distribution and Task Family

The assumption or constraint. All models are trained on 100B tokens from FineWeb-Edu, a web-text dataset filtered for educational quality. All evaluations use English-language benchmarks: WikiText and LAMBADA (language modeling), a standard commonsense reasoning suite (PIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, Social IQa, BoolQ), English QA datasets (SQuAD, TriviaQA, NQ, DROP), and synthetic English retrieval tasks (RULER). There is no evaluation on code generation, mathematical reasoning, multilingual text, or long-form generation tasks.

The consequence. The paper's claims about Gated DeltaNet-2 being "the strongest overall" architecture are conditioned on this specific evaluation suite. It is possible that the erase-write decoupling provides benefits primarily for English-language associative recall (as measured by the retrieval benchmarks) and commonsense reasoning, while offering no advantage — or even a disadvantage — on tasks requiring different memory patterns. Code generation, for instance, requires the model to track variable names, function signatures, and control flow across long contexts — a form of exact symbolic retrieval that the delta rule might handle well, but also one where local syntactic precision matters, and where the erase gate's tendency to aggressively clear old content could inadvertently delete still-relevant variable bindings. Conversely, mathematical reasoning requires step-by-step deduction where the state must preserve intermediate results precisely, and the channel-wise gates' additional modulation of value content could introduce subtle distortions. Without evaluation on any non-English, non-web-text domain, a practitioner cannot assess whether the architecture generalizes beyond the training distribution's retrieval patterns.

What evidence exists in the paper. The real-world retrieval suite (Table 4) does include structured extraction (SWDE) and PDF key-value retrieval (FDA), which are modestly out-of-domain relative to FineWeb-Edu, but still English-language and text-based. All other evaluations are standard English benchmarks. The paper does note that the remaining NQ and DROP gaps "point to formats that also need local evidence aggregation," which indirectly acknowledges that the architecture has format-specific weaknesses, but does not extend this acknowledgment to domain generalization. The paper includes no code, math, multilingual, or long-form generation evaluation.

Mitigation status. Not addressed. The paper does not discuss generalizability to other domains as a limitation or propose evaluation on code or math benchmarks. This is a standard limitation for architecture papers at the 1.3B/100B-token scale — evaluating on additional domains requires additional training runs — but it means the paper's conclusions are domain-conditional in a way that is not explicitly stated. A practitioner working on code generation or multilingual applications cannot extrapolate from these results.

---

The Write Gate Contributes Modestly, but Its Parameter Cost Is Not Analyzed

The assumption or constraint. The gate structure ablation (Table 5) reveals a specific asymmetry: the "b-only" variant (channel-wise $b_t$, scalar $w_t$) achieves a commonsense average of 52.79% and MK-NIAH-1 at 4K of 35.2%, while the full model achieves 53.11% and 37.8%. The write gate contributes only 0.32 points to the commonsense average and 2.6 points to MK-NIAH-1. However, the write gate requires an additional linear projection $W_w \in \mathbb{R}^{d_{\text{model}} \times d_v}$ — a non-trivial parameter cost. For the 1.3B model with $d_{\text{model}} = 2048$, $H_v$ value heads, and $d_v = 128$ per head, the write gate projection adds $2048 \times (H_v \cdot 128)$ parameters per layer. Across 24 layers (typical for a 1.3B model), this could be tens of millions of parameters.

The consequence. The paper's claim that Gated DeltaNet-2 generalizes KDA and Gated DeltaNet is technically true but practically incomplete: the model achieves its best results with both gates, but the write gate's marginal contribution is small relative to its parameter cost. A practitioner with a fixed parameter budget might be better off removing the write gate projection and reallocating those parameters to, for example, a wider FFN or an additional layer. The paper provides no analysis of this tradeoff — there is no experiment that gives the "b-only" variant the same total parameter count as the full model by expanding other components, which would test whether the write gate's benefit survives a fair parameter-matched comparison. If the "b-only" variant with additional FFN width or depth matches or exceeds the full model, the architectural contribution would be more precisely described as "channel-wise erase gate with optional write gate, the latter providing diminishing returns."

What evidence exists in the paper. Table 5 provides the raw performance of the "w-only," "b-only," and full variants, but does not control for parameter count. The paper states that "both gates use their channel degrees of freedom" and that the asymmetry confirms different functional roles, but does not address whether the write gate's degrees of freedom are cost-effective. The total parameter count is matched across all architectures (1.3B), but within Gated DeltaNet-2, the write gate parameters are not ablated against alternative uses of those parameters.

Mitigation status. Partially addressed by implication. The MK-NIAH-1 results in Table 5 show a 7.2-point gap between "w-only" (30.6%) and full (37.8%), and a 2.6-point gap between "b-only" (35.2%) and full. The write gate provides a meaningful retrieval boost on the most diagnostic task, even if its contribution to commonsense reasoning is small. This suggests the write gate is not redundant, but the paper does not quantify whether an alternative use of those parameters — such as a wider value projection or deeper FFN — would provide equivalent or greater benefit. The paper does not acknowledge this as a limitation or propose a parameter-matched ablation as future work.

---

The Expanded Erase Range Experiment Produces a Null Result Without Interpretation

The assumption or constraint. The paper tests an expanded erase gate range $b_t \in [0,2]^{d_k}$ (allowing negative eigenvalues in the state transition matrix, following Grazzi et al., 2025) and finds "no consistent gain at this scale" (Table 5). The results are essentially identical to the standard $[0,1]^{d_k}$ range across all metrics: WikiText perplexity 15.95 vs. 15.90, commonsense average 53.04 vs. 53.11, MK-NIAH-1 at 4K 37.6% vs. 37.8%. The paper interprets this as evidence that "the $[0,1]$ range is sufficient for the erase operation at this scale."

The consequence. This null result is informative but incomplete. It leaves open several important questions: Does the expanded range fail because (a) the model does not benefit from negative eigenvalues at any scale, or (b) the 1.3B/100B-token scale is too small for the model to learn when negative eigenvalues are useful, or (c) the training recipe (learning rate, warmup, etc.) is not tuned for the expanded range, or (d) the negative eigenvalues would help on tasks not included in the evaluation suite (e.g., state tracking, where Grazzi et al. showed benefits)? The paper's interpretation — "sufficient at this scale" — implies answer (b), but this is not tested (no larger-scale experiment). A practitioner considering the expanded range for their application cannot determine from these results whether the null finding is fundamental or scale-dependent.

What evidence exists in the paper. Only the single-row ablation in Table 5. There is no scaling analysis, no task-specific breakdown of the expanded range results (only aggregate metrics are reported), and no analysis of whether the model actually uses the expanded range when it is available (e.g., what fraction of gate values fall in $(1,2]$ during inference?). The Grazzi et al. (2025) paper that motivated this experiment found benefits for state-tracking tasks specifically — none of which are included in the evaluation suite.

Mitigation status. Partially addressed by acknowledgment. The paper includes the expanded range experiment and reports the null result transparently, which is better than omitting it. However, it draws a stronger conclusion ("sufficient at this scale") than the experiment supports, and does not propose future work to test the null result at larger scales or on state-tracking tasks. The experiment is presented as a robustness check rather than an open question, which may mislead practitioners into concluding that the expanded range is definitively unnecessary.

7. Implications and Future Directions

How This Work Changes the Landscape

Gated DeltaNet-2 shifts the conversation around delta-rule recurrent architectures from what operations the state update should support to whether those operations should be independently parameterizable. The dominant assumption in the lineage from DeltaNet through Gated DeltaNet to KDA was that a single scalar step size $\beta_t$ was the natural and sufficient parameterization of the delta rule's strength — it controls "how aggressively" the memory edit is applied, as a unified quantity. Gated DeltaNet-2 reframes this as a dimensional conflation: the erase operation modifies the state transition matrix $I - k_t e_t^\top$ in the $d_k \times d_k$ key space to determine what old content is removed, while the write operation inserts new content $k_t z_t^\top$ into the $d_k \times d_v$ state space to determine what new content is committed. These are operations on fundamentally different axes of the state. The contribution is not "we added more gates" — it is the diagnostic that tying these two decisions to one scalar is an arbitrary architectural restriction inherited from the online gradient descent step-size interpretation, not a necessary property of the delta rule.

The magnitude of this shift is incremental but strategically important. It does not introduce a new class of models or a new training paradigm. What it does is open a design space within the delta-rule family: the erase and write operations can now be independently parameterized, regularized, analyzed, and potentially simplified or elaborated in future work. The ablation finding that the erase gate accounts for most of the gain (Table 5: the "b-only" variant recovers 35.2% on MK-NIAH-1 at 4K vs. the full model's 37.8%, while "w-only" drops to 30.6%) provides immediate guidance for where to invest future parameter budget — the key-side erase operation is the higher-leverage target for improving fixed-state memory, while the write gate offers diminishing returns relative to its parameter cost.

The paper also reconciles a latent tension between the SSM and delta-rule lines of recurrent architecture research. Mamba-3 advances the SSM path with exponential-trapezoidal discretization, complex-valued rotations, and MIMO — all improvements to the correlation-write family (Mamba-2 lineage) that adds key-value products without read-subtraction. Gated DeltaNet-2 demonstrates that on the most interference-sensitive tasks (MK-NIAH-1), the delta-rule's active read-subtraction mechanism provides a structural advantage that the SSM path's improved discretization does not fully compensate for: at 4K context, Gated DeltaNet-2 achieves 37.8% vs. Mamba-3 MIMO's 18.0–20.2% (Table 3, recurrent setting). This does not mean delta-rule models are universally superior — Mamba-3 is competitive or better on commonsense reasoning and some real-world retrieval tasks — but it clarifies that the two families have complementary strengths that map to different memory pressure patterns. The SSM path excels when the primary requirement is retaining information over long durations with smooth decay; the delta-rule path excels when the primary requirement is precisely editing competing associations to reduce interference. The paper's hybrid architecture — recurrent delta-rule mixer plus sliding-window attention — can be seen as combining both strengths: compressed associative recall from the delta rule and exact local token-level operations from attention. This framework makes the architecture choice task-conditional rather than absolute, which is a more mature picture of the recurrent model landscape than the winner-take-all comparisons common in earlier work.

A specific research direction that becomes less attractive after this paper is the pursuit of ever-more-complex scalar-gating schemes for the delta rule. KDA represents the endpoint of that line: channel-wise decay with scalar delta gate is the most expressive possible parameterization under the tied-erase-and-write assumption. Gated DeltaNet-2 demonstrates that further progress on that path requires breaking the scalar tie — simply making the scalar $\beta_t$ more expressive (e.g., deeper MLP, learned per-head rather than per-layer) would not address the dimensional mismatch that the paper identifies as the bottleneck. Similarly, the expanded erase range experiment (Table 5, $b_t \in [0,2]^{d_k}$ producing no gain at 1.3B scale) suggests that the $[0,1]$ range is not the binding constraint for the erase operation at this scale — channel-wise control matters more than dynamic range expansion. This redirects research effort away from spectral-tuning tricks and toward channel-level specialization of the state update.

Follow-Up Research This Work Enables

Mechanistic interpretability of gate behavior under controlled interference. The paper's central mechanistic claim — that $b_t$ and $w_t$ learn to manage interference among compressed associations — has no direct evidence beyond the correlational improvement on MK-NIAH-1. A strong follow-up would construct a synthetic dataset where interference patterns are systematically varied: sequences containing multiple key-value pairs with controlled key similarity (orthogonal vs. overlapping key subspaces), controlled distractor density, and controlled context shifts. By analyzing how $b_t$ activates across heads and tokens as a function of key overlap and distractor presence, one could test whether the erase gate selectively clears key channels that encode stale distractor information when a new, similar key arrives, and whether the write gate suppresses value channels when the incoming value is a known distractor. The prediction from the paper's hypothesis is that $b_t$ should show elevated activation on key channels that strongly correlate with distractor keys, and this effect should be most pronounced in heads that specialize for the multi-key retrieval subtask. A negative result — gates showing no interpretable interference-management pattern — would reframe the paper's contribution as a parameter-count expansion rather than a mechanistic intervention, which is a critical distinction for the architecture's adoption.

Parameter-matched ablation of the write gate against alternative uses of those parameters. The gate structure ablation (Table 5) shows the write gate provides a modest gain: +2.6 points on MK-NIAH-1 at 4K and +0.32 points on the commonsense average. However, the write gate projection $W_w$ of shape $d_{\text{model}} \times (H_v d_v)$ consumes a non-trivial parameter budget — potentially tens of millions of parameters across all layers. A rigorous follow-up would test three configurations at matched total parameter count: (a) Gated DeltaNet-2 with both gates, (b) "b-only" Gated DeltaNet-2 with the write gate parameters reallocated to a wider FFN or an additional layer, and (c) "b-only" with the write gate parameters reallocated to a wider value projection $W_v$ or additional value heads. The specific question is whether the write gate's benefit survives a fair parameter-matched comparison — if the reallocated models match or exceed the full model, the architectural contribution would be more accurately described as "channel-wise erase gate with write gate providing optional, diminishing-return capacity." The experiment should be run at the 1.3B scale (to replicate) and ideally at a larger scale (e.g., 7B) to test whether the write gate's marginal value grows with model capacity.

Scaling analysis of the erase-write decoupling advantage with model size and training tokens. All results are at 1.3B parameters and 100B tokens. It is unknown whether the gap between Gated DeltaNet-2 and KDA on retrieval tasks grows, shrinks, or plateaus as model scale increases. A scaling law experiment would train both architectures at 300M, 700M, 1.3B, and 2.7B parameters (or the largest feasible given computational constraints) on the same data distribution, measuring the MK-NIAH-1 accuracy at each scale. If the gap widens with scale — e.g., Gated DeltaNet-2 shows a steeper scaling slope on retrieval — that would indicate the erase-write decoupling addresses a capability bottleneck that becomes more severe as the model has more capacity to store competing associations. If the gap shrinks or plateaus, the benefit may be specific to the 1.3B regime. Similarly, training on 300B or 1T tokens would test whether the advantage persists as the models become better at utilizing their fixed-size memory through more training. This experiment also tests whether the expanded erase range $[0,2]^{d_k}$ (which showed no gain at 1.3B) becomes beneficial at larger scales where the model has more capacity to exploit negative eigenvalues for state tracking.

Integration of Gated DeltaNet-2 with the Mamba-3 state-space improvements. Gated DeltaNet-2 and Mamba-3 excel on different axes: Gated DeltaNet-2 on interference-heavy associative recall (MK-NIAH-1), Mamba-3 on smooth long-range retention (S-NIAH-1, commonsense reasoning). A natural follow-up would combine the Gated Delta Rule-2 erase-write decoupling with Mamba-3's exponential-trapezoidal discretization and data-dependent rotations. Specifically, the decay side could use Mamba-3's complex-valued state transitions (which provide a richer temporal parameterization than real-valued channel-wise decay), while the active edit side uses the decoupled erase and write gates. This would test whether the two families' improvements are additive or whether they interact in unexpected ways — for instance, the exponential-trapezoidal input rule might introduce a two-token window that conflicts with the per-token delta-rule edit, or the data-dependent rotations might provide enough key-space modulation to make the explicit erase gate redundant. The hybrid architecture in this paper already combines Gated DeltaNet-2 with sliding-window attention; combining it with Mamba-3's state-space innovations at the recurrence level would test the ceiling of what a recurrent token mixer can achieve.

Real-world long-context evaluation beyond RULER. The paper's strongest results are on synthetic NIAH tasks, with only a small real-world retrieval suite truncated to 2K tokens (Table 4). A definitive follow-up would evaluate Gated DeltaNet-2 on standard long-context benchmarks at 8K, 16K, and 32K context lengths: LongBench (which includes multi-document QA, summarization, and few-shot learning tasks), SCROLLS (narrative QA and summarization), and the ZeroSCROLLS variant. The specific hypothesis is that Gated DeltaNet-2's advantage should be most pronounced on tasks requiring associative recall from long contexts with high distractor density — multi-document QA where the answer appears in one document and distractors appear in others, or few-shot learning where the model must retrieve the correct in-context example among many. Tasks requiring verbatim copying or exact token-level precision (e.g., long-form summarization) might show smaller advantages, since the delta-rule state compresses information and loses token-level detail that sliding-window attention or full softmax attention preserve. This evaluation would map the architecture's strengths onto realistic deployment scenarios and identify where the hybrid configuration (recurrent + SWA) is necessary vs. where the recurrent-only model suffices.

Practical Applications and Downstream Use Cases

Long-context retrieval-augmented generation with bounded memory. Retrieval-augmented generation (RAG) systems typically retrieve documents and prepend them to the LLM's context window, which grows linearly with the number of retrieved documents. For high-volume RAG pipelines — legal document review, multi-document scientific literature synthesis, customer support with large knowledge bases — the context can easily exceed 16K or 32K tokens, at which point a Transformer's KV cache becomes a memory bottleneck. A hybrid Gated DeltaNet-2 model with sliding-window attention (window size 2K) would keep the recurrent state size fixed regardless of how many documents are retrieved, while the SWA window provides exact local attention within each document. The paper's MK-NIAH-1 result — 48.0% accuracy at 4K context in the hybrid setting (Table 3) — suggests the model can discriminate the correct key-value pair among distractors, which is the core retrieval operation in RAG. The constant-memory decoding (Appendix C.5) means the system can scale to arbitrarily large retrieval sets without growing latency per token, making it suitable for high-throughput, latency-sensitive RAG deployments.

On-device language models with long context support. Deploying LLMs on mobile or edge devices is severely constrained by memory: storing a large Transformer's KV cache for long contexts can exceed the available RAM. A recurrent model like Gated DeltaNet-2 stores a fixed-size state of 262,144 floats per layer (for the 1.3B configuration), which for a 24-layer model is approximately 25 MB in fp32 — constant regardless of context length. For applications like on-device email summarization, document Q&A, or personal assistant conversations that may reference long message histories, the constant-memory property is a hard requirement. The paper's training throughput near-flatness (Figure 2: 38.0 Kt/s at 2K×8, 36.1 Kt/s at 16K×1) suggests that the architecture also supports efficient on-device training or fine-tuning, where memory bandwidth is the primary constraint. The modest perplexity improvement over KDA (15.90 vs. 16.81 WikiText perplexity, Table 2) would translate to incrementally better generation quality, and the retrieval improvement would directly benefit applications where the model must reference specific facts from the user's conversation history.

Training-data generation for self-improvement with associative recall requirements. When LLMs are used to generate training data for themselves — as in self-play, iterative refinement, or distillation pipelines — the quality of retrieved information from long contexts (e.g., previous outputs, reference documents, multi-turn trajectories) directly affects the quality of generated data. A hybrid Gated DeltaNet-2 model could serve as the generator in such pipelines, using its recurrent state to maintain compressed representations of the full context history and its SWA window for exact local attention to the most recent content. The paper's strong MK-NIAH-1 performance suggests the model would be less likely to confuse similar-but-distinct pieces of information (e.g., confusing the hyperparameters from a failed experiment with those from a successful one in a multi-turn self-improvement trajectory). The linear training cost (chunkwise WY algorithm, Section 3.3) means that generating data at scale — potentially millions of tokens across thousands of trajectories — remains computationally tractable, avoiding the quadratic attention cost that would make a Transformer-based generator prohibitively expensive for long trajectories.

When to Prefer This Method

The paper does not articulate an explicit tradeoff decision rule against named alternatives. It presents Gated DeltaNet-2 as a strict generalization of KDA and Gated DeltaNet — the specializations $b_t = \beta_t \mathbf{1}_{d_k}, w_t = \beta_t \mathbf{1}_{d_v}$ recover KDA, and further $\alpha_t = \alpha_t \mathbf{1}_{d_k}$ recovers Gated DeltaNet — implying that Gated DeltaNet-2 is always at least as expressive, with the only cost being additional parameters and a small constant-factor throughput overhead. The ablation in Table 5 quantifies this overhead as small (the "b-only" variant recovers most of the gain, the full model provides a modest additional boost). The paper does not describe scenarios where a practitioner should prefer KDA or Gated DeltaNet over Gated DeltaNet-2, nor does it identify a regime where the scalar-gated simpler model outperforms the decoupled version.

The comparison with Mamba-3 is similarly non-prescriptive. The results in Tables 2–4 show Gated DeltaNet-2 leading overall but Mamba-3 competitive or better on specific tasks (e.g., Mamba-3 MIMO achieves higher S-NIAH-3 at 4K in the hybrid setting: 55.6% vs. 54.2% for Gated DeltaNet-2 — though Gated DeltaNet-2 leads at 2K). The paper does not interpret these cross-task differences as a systematic tradeoff and does not propose a decision rule for choosing between the delta-rule and SSM families. The experimental design — matching recurrent state size and total parameters while testing both recurrent and hybrid variants — treats all architectures as competing on equal footing, with the conclusion being that Gated DeltaNet-2 is strongest overall at the tested scale, not that there is a principled regime where a practitioner should prefer one over another.

A speculative tradeoff could be inferred post-hoc from the results, but it is not articulated in the paper. If forced to extrapolate: the delta-rule family (including Gated DeltaNet-2) provides a structural advantage for tasks requiring associative recall under interference (MK-NIAH patterns), while the SSM family (Mamba-3) may be preferable when the primary requirement is smooth long-range information retention and the application benefits from the exponential-trapezoidal discretization's temporal precision. But these are my inferences, not the paper's explicit claims, and a "Prefer A when... Prefer B when..." matrix would be speculative rather than grounded in the paper's stated tradeoff analysis. The paper's implicit position is that Gated DeltaNet-2 represents the new state of the art within the delta-rule family, and the choice between delta-rule and SSM architectures is a broader research question that the paper's results inform but do not resolve.