A Systematic Analysis of Hybrid Linear Attention

ArXiv: 2507.06457

🎯 Pitch

Superior standalone linear models do not reliably produce the best hybrid architectures: HGRN-2, a third-tier variant in isolation, outperforms the standalone leader GatedDeltaNet once combined with just a few full-attention layers, matching Transformer recall at a fraction of the KV-cache memory. The key ingredients are selective gating, hierarchical recurrence, and controlled forgetting, and the optimal linear-to-full ratio lands between 3:1 and 6:1.

---

1. Executive Summary

This paper conducts a systematic empirical analysis of how different linear attention architectures perform when hybridized with full attention layers, training and open-sourcing 72 models—36 at 340M parameters (20B tokens) and 36 at 1.3B parameters (100B tokens)—spanning three generations of linear attention mechanisms (vector recurrence, outer-product state with decay, and delta-rule controlled forgetting) across five hybridization ratios. The central finding is that standalone linear model quality does not predict hybrid performance: while GatedDeltaNet leads in purely linear form, HGRN-2 achieves the best results once a minority of full-attention layers are interleaved, surpassing the Transformer baseline by 1.2 percentage points at a 6:1 linear-to-full ratio at 340M scale, and matching it at 1.3B parameters. Language modeling performance proves largely insensitive to the linear-to-full ratio, while recall capacity—as measured by RULER—improves sharply as the proportion of full-attention layers increases, nearly doubling from 24:1 to 3:1 configurations and saturating around 3:1, establishing that three architectural properties—selective gating, hierarchical recurrence, and controlled forgetting—are jointly necessary for hybrid models to reach Transformer-level recall while cutting KV-cache memory by a factor of 4–7×.

2. Context and Motivation

The Core Gap: We Don't Know Which Linear Attention to Use in Hybrid Models

The paper addresses a specific, pragmatic gap in the design of efficient large language models. The Transformer architecture's core computational engine—softmax self-attention—has quadratic complexity $\mathcal{O}(L^2)$ in sequence length $L$, meaning that processing a sequence twice as long costs four times as many operations. Simultaneously, the Key-Value (KV) cache—which stores intermediate representations to avoid recomputation during autoregressive decoding—grows linearly with sequence length. For long-context applications, these costs become prohibitive, motivating a broad research effort into linear-complexity alternatives that compress the per-token KV cache into a fixed-size hidden state, achieving $\mathcal{O}(L)$ overall complexity.

The sublinear scaling community has coalesced around a practical compromise: hybrid architectures that interleave a small number of full-attention layers with a majority of linear-attention layers. This gives you much of the KV-cache memory savings while preserving the recall capabilities that purely linear models struggle with. Leading production models have converged on roughly 6–7:1 linear-to-full ratios—Jamba (Lieber et al., 2024), Character.AI's deployment (2024), and MiniMax-01 (2025) all independently arrived at similar recipes. The approach demonstrably works at scale.

The gap this paper identifies is subtle but critical: while the choice of full-attention mechanism is nearly always standard softmax attention, the choice of which linear attention variant to plug into the hybrid stack appears essentially arbitrary. As the authors note in Section 1:

"The selection of the linear attention component appears relatively arbitrary. Existing research predominantly focuses on ablation studies concerning the ratio of full to linear attention layers, rather than on the specific architectural choices within the linear models themselves."

Prior hybrid-model papers treat the linear component as interchangeable. They justify their choices implicitly through standalone benchmarks: if model A beats model B when evaluated in pure form on standard language modeling benchmarks, it's assumed that A will also make a better hybrid backbone. The field has been operating under an implicit assumption—call it the transferability assumption—that standalone linear model quality translates directly into hybrid model quality. This paper asks a deceptively simple question: is that assumption valid?

Why This Matters: Practical and Scientific Stakes

The gap is not merely academic. Three concrete consequences make it urgent to resolve:

1. Architectural decisions in hybrid models are locked in early but persist through scale. The choice of linear backbone for a hybrid model is a foundational architectural decision that propagates through pretraining, fine-tuning, and deployment. If the transferability assumption is wrong—if the best standalone model does not produce the best hybrid—then large-scale hybrid projects may be systematically choosing suboptimal linear components. Given that training runs for billion-parameter models cost millions of dollars, an incorrect assumption at the design stage wastes enormous resources. The paper's 72-model study is large enough to detect systematic patterns but small enough (340M–1.3B parameters) to be feasible, serving as a pilot study whose findings can inform architectural decisions at production scale.

2. The ratio between linear and full attention controls a fundamentally asymmetric trade-off. Section 2 of the paper notes that the linear-to-full ratio is typically tuned by minimizing language modeling loss on short-text benchmarks. But as the paper demonstrates, language modeling loss is nearly flat across ratios—varying by less than one percentage point from 24:1 to 3:1—while recall capacity doubles over the same range. If you tune your hybrid ratio by looking at perplexity, you'll see no signal and likely settle on whatever ratio is convenient, potentially leaving enormous recall performance on the table. The paper reframes the optimization problem: the linear-to-full ratio is primarily a recall knob, not a perplexity knob, and should be tuned accordingly. This is a non-obvious insight with direct practical implications for anyone designing hybrid models.

3. Understanding why some linear models hybridize well while others don't reveals fundamental principles about memory in recurrent architectures. The paper is not merely an empirical bake-off. By selecting one representative model from each "generation" of linear attention—HGRN (Gen-1: gated vector recurrence), RetNet and GLA (Gen-2: outer-product state with decay at different gating levels), HGRN-2 (Gen-2: hierarchical gating), and GatedDeltaNet (Gen-3: delta-rule controlled forgetting)—the authors construct a controlled comparison where each model differs along a specific design axis. This allows them to go beyond ranking models and instead identify the architectural properties that enable effective hybridization: selective gating, hierarchical recurrence, and controlled forgetting. These findings constitute a vocabulary for reasoning about what makes a linear backbone suitable for hybrid deployment, which the field had previously lacked.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in prior work, which I'll walk through in detail.

Missing comparative analysis of linear backbones within hybrid stacks. The original Jamba paper (Lieber et al., 2024) hybridized Mamba with Transformers at a 7:1 ratio and demonstrated strong results, but didn't ask whether a different state-space model or gated RNN would perform better. StripedHyena (Poli et al., 2023) used a 1:1 ratio with Hyena operators but didn't compare against alternatives. RecurrentGemma (Botev et al., 2024) chose a specific gated linear recurrence (Griffin) with sliding window attention at 2:1 but provided no head-to-head comparison with other linear mechanisms at the same ratio. Mamba-2-Hybrid (Waleffe et al., 2024) hybridized Mamba-2 at 4:1, again without exploring alternatives. In each case, the choice of linear component was made a priori—based on the standalone reputation of the linear model—and then fixed while the ratio was swept. The field had no systematic data on whether a different choice would have produced better results.

The transferability assumption is untested. The paper cites explicit evidence that this assumption may be flawed. Shen et al. (2024) published scaling laws for linear complexity language models showing that different linear architectures have different scaling exponents—their relative performance at small scale may not predict their ranking at large scale. More fundamentally, hybridization introduces a new factor: the interaction between the linear layers' hidden state management and the sparse full-attention layers' global context. A linear model that excels at maintaining long-range dependencies in pure form may rely on mechanisms that are redundant in the presence of full-attention layers, while a simpler model that's weak on recall may benefit disproportionately from the full-attention "boost." The paper's central empirical finding—that GatedDeltaNet leads in standalone but HGRN-2 wins in hybrid form—confirms that this interaction is real and non-trivial.

Ratio selection is guided by the wrong metric. Section 2 and the related work discussion identify that existing hybrid model papers typically ablate the linear-to-full ratio by measuring perplexity or language modeling loss—metrics that are dominated by local token prediction accuracy. The paper cites Ref. (Poli et al., 2024), which demonstrated that optimal hybrid architectures leverage specialized layers through hybrid topology, suggesting that different capabilities (language modeling vs. recall) may respond differently to architectural choices. This paper operationalizes that insight by measuring both dimensions and showing they have qualitatively different relationships with the ratio, with profound implications for how hybrid models should be designed and evaluated.

Recall limitations of pure linear models are well-documented but not systematically studied in hybrids. The paper notes that prior work (Jelassi et al., 2024; Sieber et al., 2024) has established that purely linear/recurrent models underperform Transformers on copying, retrieval, and in-context learning tasks. What was missing is a systematic characterization of how much full attention is needed to close this gap, and which linear backbones close it most efficiently. The paper's RULER benchmark results (Figure 3, right panel) provide the first such characterization: recall performance rises steadily with full-attention proportion, saturates around 3:1, and critically, only certain linear architectures (those with gating and controlled forgetting) ever reach the Transformer baseline even at the most attention-heavy ratio.

Scaling state size has diminishing returns. The paper references Liu et al. (2025), who showed that simply increasing the state size of RNN-based LLMs yields diminishing returns for long-context performance. This finding suggests that the bottleneck in recall tasks is not raw memory capacity but rather the ability to effectively manage stored information—what to keep, what to discard, and how to retrieve. The paper builds on this insight by hypothesizing that the architectural properties governing state management (gating, hierarchical organization, controlled forgetting) are what distinguish effective hybrid backbones from ineffective ones, and tests this hypothesis through comparison of models that differ specifically along these dimensions.

How This Paper Positions Itself

The paper positions itself at the intersection of three research threads: linear attention mechanisms (which it traces through three generations of evolution), hybrid architectures (which it treats as the deployment target), and principled architecture design (which it aspires to contribute to). Its contribution is explicitly empirical and systematic—it does not propose a new linear attention mechanism, a new hybridization scheme, or a new training objective. Instead, it provides the first controlled study where the linear backbone is varied while all other factors (training data, token budget, model scale, evaluation protocol, and hybridization strategy) are held constant.

This is methodologically important because prior work varied multiple factors simultaneously: Jamba used Mamba with a particular tokenizer, training recipe, and data mixture; MiniMax-01 used Lightning Attention with a different recipe; and so on. The resulting models are not comparable—you cannot tell whether Jamba's performance relative to MiniMax-01 is due to the linear backbone, the training data, the ratio, the tokenizer, or the scale. By training all models in this study from scratch under identical conditions, the paper isolates the effect of the linear attention architecture. The 72-model matrix is the mechanism for achieving this control: 6 linear attention variants × 5 ratios (plus pure linear) = 36 configurations × 2 model sizes = 72 models.

The paper also positions itself as providing actionable design guidelines rather than merely benchmarking. The practical recommendations—use a gated, hierarchically recurrent model with controlled forgetting (HGRN-2 or GatedDeltaNet), aim for a 3:1 to 6:1 linear-to-full ratio, expect roughly flat language modeling loss across ratios but tune for recall—are concrete enough that a practitioner building a hybrid model today could adopt them immediately. This distinguishes the work from purely analytical studies and aligns it with the engineering-oriented hybrid architecture literature (Jamba, Hymba, Zamba).

A subtle but important positioning choice: the paper does not claim to have identified a universally optimal linear backbone or ratio. It explicitly frames its findings as bounded by the experimental conditions—models up to 1.3B parameters, 2,048-token context windows, block-wise mixing, pretraining on fineweb-edu, no instruction tuning. The conclusion section flags these as limitations and calls for verification at larger scales, longer contexts, and under different training paradigms. This restraint strengthens the contribution: the paper's claims are precise and falsifiable, which makes them useful for the next round of research.

Finally, the paper connects to the emerging literature on principled hybrid design—specifically the STAR framework (Thomas et al., 2025) for automated architecture synthesis and the mechanistic design approach of Poli et al. (2024). By identifying specific architectural properties (gating, hierarchy, controlled forgetting) that correlate with hybrid effectiveness, the paper provides a vocabulary and set of hypotheses that could inform future architecture search algorithms, moving the field beyond empirical ratio sweeps toward optimization grounded in mechanistic understanding.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a controlled experimental framework for systematically evaluating how different linear attention mechanisms perform when hybridized—that is, interleaved with standard softmax attention layers—in a language model pretraining setting. The system under study is not a single model but rather a 36-configuration matrix (6 linear attention variants × 5 hybridization ratios plus pure linear baselines) at two parameter scales (340M and 1.3B), all trained from scratch under identical conditions, enabling isolation of the linear backbone's contribution to hybrid performance and direct falsification of the assumption that stronger standalone linear models necessarily produce stronger hybrids.

3.2 Big-Picture Architecture

The evaluation framework has five major components, organized as a pipeline from architectural selection through training to measurement:

1. Linear Attention Backbone Selection — Six linear attention variants spanning three "generations" are chosen as representatives of distinct design axes: HGRN (Gen-1 vector recurrence), RetNet and GLA (Gen-2 outer-product with fixed vs. data-dependent decay), HGRN-2 (Gen-2 hierarchical gating), and GatedDeltaNet (Gen-3 delta-rule controlled forgetting). Each serves as a "spokesperson" for a cluster of related mechanisms.

2. Hybrid Stack Construction — For each backbone, a hybrid architecture is assembled by interleaving linear-attention layers with full self-attention layers in a repeating block pattern at five different ratios: 24:1, 12:1, 6:1, 3:1 (linear:full), plus a pure linear variant. The full-attention layers maintain a standard KV cache; linear layers use a fixed-size hidden state.

3. Controlled Pretraining — All 36 configurations (plus baselines) are trained from scratch on the fineweb-edu dataset using identical optimization hyperparameters, token budgets (20B for 340M, 100B for 1.3B), and the flash-linear-attention library for efficient implementation. This eliminates confounding variables like data mixture, training recipe, or tokenizer differences.

4. Two-Dimensional Evaluation — Each trained model is evaluated on two distinct capability axes: language modeling (six benchmarks: ARC-Challenge, ARC-Easy, HellaSwag, LAMBADA, OpenBookQA, PIQA) measured via zero-shot accuracy, and recall (the RULER suite: retrieval, multi-hop tracing, aggregation, question answering) measured by RULER score.

5. Architectural Property Analysis — By comparing models that differ only along specific design axes (gating type, recurrence hierarchy, forgetting mechanism), the framework identifies which architectural properties correlate with strong hybrid performance, going beyond simple model rankings.

Information flows linearly: an architecture is selected → models at all ratios are trained → each model is evaluated on both LM and recall benchmarks → results are analyzed across the ratio sweep and across architectural families → design principles are extracted.

3.3 Roadmap for the Deep Dive

The detailed technical breakdown follows this order:

• First, the three generations of linear attention mechanisms (Sections 3.1–3.2 of the paper), because understanding what distinguishes HGRN from HGRN-2 from GatedDeltaNet is prerequisite to interpreting why they perform differently in hybrids. I'll explain the state update equations for each generation, what each gate controls, and what property each generation adds over its predecessor.

• Second, the rationale for the specific representatives chosen (Section 3.3), including which closely-related models each "spokesperson" stands in for—this matters because it establishes the scope of the findings beyond the six tested models.

• Third, the hybridization strategy (Section 3.4), covering how linear and full-attention layers are interleaved, the ratio sweep, and the implications for KV-cache memory and computational cost during training versus inference.

• Fourth, the benchmarking framework and training protocol (Section 3.5), including exact dataset, token budgets, optimizer settings, model sizes, and evaluation benchmarks—these details are essential for assessing the validity and limitations of the empirical results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical analysis paper whose core idea is that the transferability assumption—"better standalone linear model → better hybrid model"—is false, and that identifying the architectural properties that do predict hybrid effectiveness requires a controlled comparison across generations of linear attention mechanisms with all confounding factors held constant.

---

Generation 1: Gated Vector Recurrence (HGRN)

The first generation of linear attention mechanisms collapses the per-token key-value cache into a single $d$-dimensional vector $\boldsymbol{h}_t \in \mathbb{R}^d$ that is updated additively and modulated by a learned element-wise gate. The core recurrence is:

$\boldsymbol{h}_t = \boldsymbol{\alpha}_t \odot \boldsymbol{h}_{t-1} + (1 - \boldsymbol{\alpha}_t) \odot \boldsymbol{v}_t$

$\boldsymbol{o}_t = \boldsymbol{h}_t \odot \boldsymbol{q}_t$

where $\boldsymbol{\alpha}_t \in (0, 1)^d$ is an element-wise forget gate computed as a function of the input sequence up to position $t$, typically $\boldsymbol{\alpha}_t = f_\theta(x_{1:t})$ for some learned function $f_\theta$; $\boldsymbol{v}_t \in \mathbb{R}^d$ is a value projection of the current token; $\boldsymbol{q}_t \in \mathbb{R}^d$ is a query projection; $\boldsymbol{h}_t \in \mathbb{R}^d$ is the hidden state vector; and $\odot$ denotes the Hadamard (element-wise) product. The output $\boldsymbol{o}_t$ is the element-wise product of the current hidden state and the query vector.

What the update computes: At each time step, the model decides—on a per-channel (dimension) basis—what fraction of the previous hidden state to retain and what fraction of the new value to incorporate. The forget gate $\boldsymbol{\alpha}_t$ controls this balance: when $\alpha_{t,i}$ is close to 1, channel $i$ retains old information; when close to 0, channel $i$ overwrites with new information. The output is then a gated read from the hidden state, modulated by the current query—if the query has high magnitude in a given dimension, that dimension of the hidden state is amplified in the output.

Why this form: The element-wise gating solves a fundamental problem that plagued classical RNNs: vanishing and exploding gradients. In a vanilla RNN, the hidden state update is $\boldsymbol{h}_t = \sigma(W_h \boldsymbol{h}_{t-1} + W_x \boldsymbol{x}_t)$, where the same weight matrix $W_h$ is applied at every step; during backpropagation through time, gradients are multiplied by powers of $W_h$, causing exponential growth or decay. The gated formulation decouples the temporal dynamics from a fixed weight matrix—the forget gate $\boldsymbol{\alpha}_t$ is computed freshly at each step based on the input, allowing the model to learn when to preserve information (gate near 1) and when to reset (gate near 0). This is the same principle that made LSTMs and GRUs effective, but applied in a simpler form suitable for linear attention.

Capacity limitation: Because $\boldsymbol{h}_t$ is a single vector of size $d$, the model can store at most $d$ independent scalar values at any time. If two pieces of information compete for the same channel, the gate must average between them, leading to interference. This makes vector-state models fundamentally limited on recall tasks requiring the model to store and later retrieve many distinct associations—the classic "memory capacity" bottleneck. This limitation motivates Generation 2.

---

Generation 2: Outer-Product State with Decay

The second generation increases memory capacity by promoting the hidden state from a vector to a matrix $\mathbf{S}_t \in \mathbb{R}^{d \times n}$ (typically $n = d$), updated by accumulating rank-one outer products while applying a decay mechanism:

$\mathbf{S}_t = \mathbf{S}_{t-1} \odot (\mathbf{1}\boldsymbol{\alpha}_t^\top) + \boldsymbol{v}_t \boldsymbol{k}_t^\top$

$\boldsymbol{o}_t = \mathbf{S}_t \boldsymbol{q}_t$

where $\mathbf{S}_t \in \mathbb{R}^{d \times n}$ is the matrix-valued hidden state; $\boldsymbol{\alpha}_t \in \mathbb{R}^n$ is a decay vector (one scalar per column of $\mathbf{S}_t$); $\mathbf{1} \in \mathbb{R}^d$ is a vector of ones used to broadcast the decay across rows; $\boldsymbol{v}_t \in \mathbb{R}^d$ is the value; $\boldsymbol{k}_t \in \mathbb{R}^n$ is the key; $\boldsymbol{q}_t \in \mathbb{R}^n$ is the query; and $\odot$ denotes the Hadamard product (applied element-wise, with $\mathbf{1}\boldsymbol{\alpha}_t^\top$ being a $d \times n$ matrix where each column $j$ is the constant $\alpha_{t,j}$). The term $\boldsymbol{v}_t \boldsymbol{k}_t^\top$ is a rank-one outer product producing a $d \times n$ matrix.

What the update computes: At each time step, every column of the state matrix $\mathbf{S}_t$ is first scaled down by its corresponding decay factor $\alpha_{t,j}$—this is multiplicative decay, meaning that information stored in the state gradually fades unless refreshed. Then, a new rank-one association $\boldsymbol{v}_t \boldsymbol{k}_t^\top$ is added to the state. The key $\boldsymbol{k}_t$ determines where in the matrix the value $\boldsymbol{v}_t$ is written (each column $j$ receives $\boldsymbol{v}_t \cdot k_{t,j}$), and the query $\boldsymbol{q}_t$ determines where to read: the output $\boldsymbol{o}_t = \mathbf{S}_t \boldsymbol{q}_t$ computes a weighted sum of the columns of $\mathbf{S}_t$, with weights given by the query-key alignment. This is mathematically equivalent to linear attention without softmax: the state $\mathbf{S}_t$ accumulates key-value outer products, and reading is a dot-product attention over the accumulated keys.

Why this form—the capacity argument: Moving from a vector state ($d$ scalars) to a matrix state ($d \times d$ scalars) increases raw memory capacity from $\mathcal{O}(d)$ to $\mathcal{O}(d^2)$. For a typical head dimension of $d = 64$, this means going from 64 independent stored values to 4,096. Intuitively, the outer-product formulation allows the model to store separate value vectors indexed by different key patterns: if two keys $\boldsymbol{k}_a$ and $\boldsymbol{k}_b$ are near-orthogonal, their associated values $\boldsymbol{v}_a$ and $\boldsymbol{v}_b$ occupy approximately non-overlapping subspaces of the state matrix, reducing interference. The query mechanism then retrieves the value whose key best matches the current query—this is an associative memory, a continuous generalization of a key-value lookup table. The decay gate $\boldsymbol{\alpha}_t$ controls the rate at which old associations are forgotten, preventing unbounded state growth.

Three sub-families distinguished by gating complexity. The paper identifies three design points along the gating axis, illustrated in Table 1:

(i) RetNet (fixed, non-learned decay): The decay is a single scalar $\gamma \in (0, 1)$ shared across all positions and all channels: $\boldsymbol{\alpha}_t = \gamma \cdot \mathbf{1}$ for all $t$. This means every column of the state matrix decays at the same fixed rate, regardless of what information is being processed. The decay is learned once during training but does not adapt to the input at inference time. This is the simplest and fastest variant—only five $d^2$ FLOP passes per token per head—but it cannot selectively retain important information based on content. Lightning attention (the mechanism used in MiniMax-01) uses this same fixed-decay mechanism, so RetNet serves as the representative for this class.

(ii) GLA (fully data-dependent per-channel decay): The decay vector $\boldsymbol{\alpha}_t$ is fully learned as a function of the input at each time step and varies across channels (columns of $\mathbf{S}_t$). Formally, $\boldsymbol{\alpha}_t = g_\theta(x_{1:t}) \in (0, 1)^n$, where $g_\theta$ is a learned gating network. This provides maximum flexibility: the model can learn to decay some columns quickly (erasing irrelevant information) while preserving others nearly indefinitely (retaining long-range dependencies). The cost is higher—seven $d^2$ FLOP passes per token per head due to the two extra gating computations—and more parameters to train. Mamba-2 uses a scalar (data-dependent but shared across channels) version of this gate, and RWKV-6 modifies only the read-out while keeping the same gated update; GLA is chosen as the most expressive representative of this axis.

(iii) HGRN-2 (tied gate with hierarchical organization): The decay is shared across all channels (columns)—$\boldsymbol{\alpha}_t = \alpha_t \cdot \mathbf{1}$—but includes a hierarchical two-pathway architecture. One path (the "slow" or "coarse" memory) operates at a lower update rate, accumulating long-range context; the other path (the "fast" or "fine" memory) handles token-level details. The tied gate couples the key-value write with the forget behavior, meaning that when the model decides to write new information, it simultaneously determines how long that information persists. This is described further in the architectural determinants section (Section 4.3 of the paper), but the key property is that it achieves multi-timescale memory at minimal parameter cost—seven $d^2$ FLOP passes per token per head, same as GLA, but with a more structured inductive bias. MetaLA adopts the identical tied-gate update, making HGRN-2 the representative for this class.

The decay design space summarized: RetNet (fixed scalar) → GLA (learned per-channel vector) → HGRN-2 (learned tied scalar with hierarchical structure). This progression represents increasing sophistication in how the model decides what to forget, holding the outer-product state structure constant. The paper's empirical results test which level of gating sophistication translates into improved hybrid performance.

---

Generation 3: Delta-Rule Controlled Forgetting (GatedDeltaNet)

The third generation replaces multiplicative decay with an explicit erase-then-write operation based on the delta rule from associative memory literature:

$\mathbf{S}_t = \mathbf{S}_{t-1} \left(\mathbf{I} - \beta_t \boldsymbol{k}_t \boldsymbol{k}_t^\top\right) + \beta_t \boldsymbol{v}_t \boldsymbol{k}_t^\top$

where $\mathbf{S}_t \in \mathbb{R}^{d \times n}$ is the matrix-valued hidden state; $\mathbf{I} \in \mathbb{R}^{n \times n}$ is the identity matrix; $\beta_t \in (0, 1)$ is a scalar update strength; $\boldsymbol{k}_t \in \mathbb{R}^n$ is the key (assumed normalized, $\|\boldsymbol{k}_t\| = 1$); $\boldsymbol{v}_t \in \mathbb{R}^d$ is the value; and $\boldsymbol{k}_t \boldsymbol{k}_t^\top \in \mathbb{R}^{n \times n}$ is the key self-outer-product (a rank-one projection matrix). The read-out remains $\boldsymbol{o}_t = \mathbf{S}_t \boldsymbol{q}_t$.

What the update computes: The update proceeds in two conceptual phases. First, the erase phase: the term $(\mathbf{I} - \beta_t \boldsymbol{k}_t \boldsymbol{k}_t^\top)$ is a projection that removes from $\mathbf{S}_{t-1}$ any information aligned with the current key $\boldsymbol{k}_t$. Geometrically, $\boldsymbol{k}_t \boldsymbol{k}_t^\top$ projects vectors onto the direction of $\boldsymbol{k}_t$, so $\mathbf{I} - \beta_t \boldsymbol{k}_t \boldsymbol{k}_t^\top$ shrinks the component of each row of $\mathbf{S}_{t-1}$ that lies along $\boldsymbol{k}_t$ by a factor of $(1 - \beta_t)$. When $\beta_t$ is close to 1, this nearly erases the old value associated with key $\boldsymbol{k}_t$. Second, the write phase: the term $\beta_t \boldsymbol{v}_t \boldsymbol{k}_t^\top$ adds a new association between key $\boldsymbol{k}_t$ and value $\boldsymbol{v}_t$. The scalar $\beta_t$ trades off how much of the old memory to keep versus how much new information to write—if $\beta_t \approx 0$, the state barely changes (the old association is preserved); if $\beta_t \approx 1$, old content aligned with $\boldsymbol{k}_t$ is erased and replaced.

Why this form—controlled forgetting: The crucial difference from Generation 2 decay is that forgetting is targeted. In Generation 2, decay applies uniformly to all columns of the state matrix—information fades at a rate determined by the per-column decay factor, regardless of whether it is relevant to the current input. In the delta rule, the erase operation is a rank-one projection that specifically targets the subspace spanned by the current key. This means that only content that is similar (in the sense of key alignment) to what is currently being written gets overwritten; perpendicular content is left untouched. This is a form of controlled forgetting: the model can update an association for a particular key pattern without disturbing unrelated associations stored in orthogonal subspaces.

The update is mathematically identical to a single step of stochastic gradient descent on an online least-squares objective:

$\mathcal{L}(\mathbf{S}) = \sum_{i=1}^{t} \|\mathbf{S} \boldsymbol{k}_i - \boldsymbol{v}_i\|^2$

If we attempt to learn a linear map $\mathbf{S}$ that maps keys to values by minimizing the cumulative squared error over all past key-value pairs, one step of gradient descent with learning rate $\beta_t$ on the current pair $(\boldsymbol{k}_t, \boldsymbol{v}_t)$ yields exactly the delta-rule update. This interpretation means the hidden state $\mathbf{S}_t$ behaves like a continually trained linear associative memory—each new token triggers a single gradient update that adjusts the mapping to better fit the current key-value association. The rank-one correction injects quadratic feature interactions (the outer product $\boldsymbol{v}_t \boldsymbol{k}_t^\top$ multiplies two different projections of the input, creating a second-order feature), giving the network a nonlinear expressiveness that standard softmax attention (which is a $\mathsf{TC}^0$ circuit) cannot achieve, yet the computation remains $\mathcal{O}(L d^2)$—linear in sequence length.

Cost and variants: The delta-rule update requires approximately eight $d^2$ FLOP passes per token per head (one extra forget/restore pass compared to the seven-pass matrices of GLA/HGRN-2). GatedDeltaNet adds a learned gate to the $\beta_t$ parameter, making the update strength input-dependent, which the paper finds improves recall performance compared to the ungated DeltaNet. As of the paper's writing, DeltaNet and GatedDeltaNet are the only public implementations of this family; the paper surveys them briefly rather than selecting a representative, since they form their own distinct category.

---

Generational Progression Summary

The three generations represent increasing sophistication in how the hidden state is managed, captured visually in Figure 1:

1. Gen-1 (Vector, $\mathcal{O}(d)$ capacity): A single vector state with element-wise gating. Least capacity, simplest to compute, worst recall.
2. Gen-2 (Matrix with decay, $\mathcal{O}(d^2)$ capacity): An outer-product state with multiplicative decay applied per-column. Increased capacity, but decay is uniform or channel-wise—information fades at a rate determined by the gate, not by content relevance. Three sub-families vary the sophistication of the decay: RetNet (fixed), GLA (learned per-channel), HGRN-2 (tied with hierarchical structure).
3. Gen-3 (Matrix with targeted forgetting, $\mathcal{O}(d^2)$ capacity): An outer-product state where forgetting is a rank-one erase operation targeted at the current key subspace. This prevents interference between unrelated memories, yielding the best recall among linear models while matching Gen-2 in asymptotic complexity.

The arrow at the bottom of Figure 1 emphasizes the progression from minimal to maximal recall capability. The paper's key empirical question is: does this progression in standalone recall capability translate to the hybrid setting, or do the interactions with full-attention layers change the ranking?

---

Representative Selection Rationale (Section 3.3)

The paper does not test every existing linear attention variant. Instead, it selects one "spokesperson" for each design cluster. Understanding the taxonomy behind these choices is essential because it defines the scope of conclusions: when the paper says "HGRN-2 performs well as a hybrid backbone," it is implicitly claiming that the tied-gate hierarchical class as a whole merits investigation.

HGRN for Gen-1: HGRN is described as "the strongest vector-state model on language tasks," providing a clean low-capacity baseline. Hawk (De et al., 2024) shares the same gated vector recurrence form, so HGRN stands in for all vector-state models. The key property tested is: how much does the capacity jump from $\mathcal{O}(d)$ to $\mathcal{O}(d^2)$ matter in hybrids?

RetNet for fixed-decay Gen-2: RetNet's single scalar decay $\gamma$ is the simplest outer-product variant. Lightning attention (MiniMax-01, 2025) uses the identical fixed-decay mechanism, so RetNet represents this entire class. The key property tested is: does the lack of input-dependent gating cripple hybrid recall, or do the full-attention layers compensate?

GLA for data-dependent Gen-2: GLA applies a fully learned per-token diagonal gate, making it the most expressive member of the decay-based family. Mamba-2 reduces the gate to a scalar, and RWKV-6 modifies only the read-out while keeping the same gated update—both "live on the same design axis" with GLA as the most expressive exemplar. The key property tested is: does maximal gating flexibility translate into stronger hybrid performance, or does the added complexity provide diminishing returns?

HGRN-2 for hierarchical Gen-2: HGRN-2 ties the gate across channels and adds hierarchical organization. MetaLA adopts the identical tied-gate update, so HGRN-2 represents this class. The key property tested is: does the structured inductive bias of hierarchical recurrence outperform the unstructured flexibility of GLA in hybrids?

Gen-3 delta-rule models: DeltaNet and GatedDeltaNet are the only public implementations; they are not so much "representatives" as the entire currently-available family. The key property tested is: does controlled (targeted) forgetting outperform uniform decay in hybrids?

These selections allow the paper to study how gating strategy—from none (RetNet) to per-channel (GLA) to tied-hierarchical (HGRN-2) to delta-rule (GatedDeltaNet)—interacts with the sparse set of full-attention layers in a hybrid model, while also capturing the state-size axis (vector vs. matrix).

---

Hybridization Strategy (Section 3.4)

The hybrid architecture follows a simple interleaving pattern, illustrated in Figure 2. Understanding the mechanics—specifically what changes during training versus inference, and what costs are incurred—is essential for interpreting the ratio sweep results.

Architecture layout. An input sequence is first mapped to embeddings. The network then applies $N$ repetitions of a composite block consisting of $r$ linear-attention layers followed by 1 full-attention layer, where $r$ is the linear-to-full ratio. The paper sweeps four values of $r$: 24, 12, 6, and 3, plus a pure linear variant ($r = \infty$) and a pure Transformer baseline ($r = 0$). A final projection head maps to vocabulary logits. The block is repeated $N$ times such that the total number of full-attention layers depends on the ratio: a model with a 24:1 ratio has $N$ blocks × 1 full-attention layer per block = $N$ full-attention layers total; a model with a 3:1 ratio has $N$ blocks × 1 full-attention layer per block, but since $r$ is smaller, $N$ is larger for the same total depth (or the model depth is adjusted to maintain comparable parameter counts).

Linear-attention layer behavior. Each linear layer maintains a constant-size state (a vector for HGRN, a matrix for Gen-2/Gen-3 models) that is updated via the model-specific recurrence (as described in the generation equations above). During training, the entire sequence is processed in parallel using the efficient chunk-wise algorithms from the flash-linear-attention library (Yang and Zhang, 2024), which exploit the linear-complexity structure for hardware efficiency. During autoregressive decoding, the state is updated incrementally—each new token triggers one state update and one read operation. Critically, the state size does not grow with sequence length: for vector-state models, it is $\mathcal{O}(d)$; for matrix-state models, it is $\mathcal{O}(d^2)$ per head, or $\mathcal{O}(d_{\text{model}}^2 / H)$ total. No KV cache is needed for these layers, meaning the inference memory footprint is constant with respect to sequence length.

Full-attention layer behavior. Each full-attention layer computes standard softmax self-attention:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$

where $Q, K, V \in \mathbb{R}^{L \times d}$ are the query, key, and value matrices for all $L$ tokens in the sequence, and $d_k$ is the key dimension. During training, this is computed in parallel at $\mathcal{O}(L^2 d)$ cost. During autoregressive decoding, the KV cache stores all past keys and values, growing linearly with sequence length—this is the source of the hybrid model's inference memory bottleneck, and the reason why reducing the number of full-attention layers (by increasing the linear-to-full ratio) saves memory.

Cost implications and the ratio's role. The ratio $r$ directly controls the inference-time memory-accuracy tradeoff:

• KV-cache memory is proportional to the number of full-attention layers × the number of tokens in the cache × the model dimension. For a model with $L_{\text{full}}$ full-attention layers and $L_{\text{linear}}$ linear layers, the cache size is roughly $L_{\text{full}} \times L \times d_{\text{model}}$ (plus the fixed-size linear states, which are negligible for long sequences). Moving from $r = 3$ (one full-attention layer per 3 linear layers) to $r = 12$ (one per 12) reduces full-attention layers by a factor of 4, cutting KV-cache memory by approximately 4×.

• Training cost is less affected by the ratio because even the linear layers have $\mathcal{O}(L d^2)$ cost for matrix-state models—the asymptotic complexity is still linear in $L$ for both layer types when using efficient implementations, though full-attention has a larger constant factor at moderate sequence lengths.

• The ratio does not affect the linear layers' state management mechanism. This is crucial for the experimental design: changing the ratio changes how often global token-to-token interaction occurs but not how the linear layers store and retrieve information. This isolates the ratio as a pure "recall booster" knob, orthogonal to the choice of linear backbone.

Training and inference distinction. The paper is explicit that during training, the stack is processed exactly like a Transformer (parallel over the sequence, with causal masking). Only during autoregressive decoding do the cache differences manifest. This means the paper's training FLOP comparisons (Figure 5, Appendix A) are forward-pass counts that reflect the structural cost of each token mixer, not the inference-time memory advantage.

---

Benchmarking Framework and Training Protocol (Section 3.5)

The experimental design's credibility rests on the rigor of the training and evaluation protocol, which controls all variables except the linear attention architecture and the hybridization ratio.

Models and scales. Two model sizes are trained:

• 340M parameters, trained on 20 billion tokens.
• 1.3B parameters, trained on 100 billion tokens.

For each model size, 36 configurations are trained: 6 linear attention variants (HGRN, RetNet, GLA, HGRN-2, DeltaNet, GatedDeltaNet) × 5 hybridization ratios (24:1, 12:1, 6:1, 3:1, pure linear), plus a full Transformer baseline, totaling 72 models trained from scratch. All models are implemented using the flash-linear-attention library (Yang and Zhang, 2024).

Training data. All models are pretrained on fineweb-edu (Penedo et al., 2024), a high-quality filtered subset of CommonCrawl web text selected for educational value. Using a single dataset eliminates data composition as a confounding variable—differences in model performance cannot be attributed to one model being trained on higher-quality or more diverse data.

Optimization hyperparameters. All models use the AdamW optimizer with a cosine learning rate schedule. For the 340M-parameter models: batch size is 50k tokens, accumulating to 20 billion training tokens total. For the 1.3B-parameter models: batch size is 1 million tokens, reaching 100 billion total training tokens. The paper does not report specific learning rates, weight decay, or warmup steps explicitly in the main text, but the key design choice is that all models share identical optimization settings within each parameter scale, so differences arise from architecture rather than training dynamics.

Evaluation benchmarks—language modeling. Zero-shot accuracy is measured on six standard benchmarks:

• ARC-Challenge (ARC-c) and ARC-Easy (ARC-e) (Clark et al., 2018): grade-school multiple-choice science questions, testing reasoning and factual knowledge.
• HellaSwag (Zellers et al., 2019): commonsense natural language inference requiring choosing the most plausible continuation of a scenario.
• LAMBADA (LMB) (Paperno et al., 2016): word prediction requiring broad discourse context—a test of long-range language modeling.
• OpenBookQA (OBQA) (Mihaylov et al., 2018): open-book science question answering requiring multi-step reasoning.
• PIQA (Bisk et al., 2019): physical commonsense reasoning about everyday situations.

All benchmarks are evaluated in a zero-shot manner—no task-specific fine-tuning, no few-shot examples, no prompt engineering. The "average LM score" reported in Figure 3 and tables is the arithmetic mean of these six accuracy scores. This metric captures general language understanding capability.

Evaluation benchmarks—recall. Recall is measured using RULER (Hsieh et al., 2024), a suite specifically designed to assess long-context retrieval capabilities. RULER includes subtasks that test:

• Single-key retrieval: finding one piece of information stored earlier in the context.
• Multi-key retrieval: finding multiple pieces of information, testing interference resistance.
• Multi-hop tracing: following chains of references through the context.
• Aggregation: summarizing or combining information from multiple positions.
• Question answering (QA): answering questions that require retrieval from the long context.
• Common Word Extraction (CWE) and Frequent Word Extraction (FWE): simpler lexical retrieval tasks.

The RULER evaluation uses a 2,048-token context window (noted in Section 5 limitations). The "average RULER score" reported in Figure 3 is the aggregate across subtasks. This metric captures the model's ability to access and use information stored earlier in the sequence—precisely the capability where pure linear models are known to struggle.

Why these two metric categories: The paper's central claim is that language modeling and recall respond differently to the hybridization ratio and to the choice of linear backbone. Measuring both on every model configuration is what enables this claim to be tested. A common alternative approach—reporting only perplexity—would miss the recall dimension entirely, which the paper argues is the dimension where architectural choices matter most.

The 340M recall caveat. Table 2 includes the note: "340M models show insignificant recall capability due to their relatively small parameter count. We omit recall benchmarks for this reason." This is an important methodological choice: at 340M parameters, the models lack sufficient capacity for RULER tasks to produce meaningful signal, so recall comparisons are restricted to the 1.3B scale. The paper implicitly claims that 1.3B parameters is the minimum scale where recall differences become detectable, which bounds the generalizability of the findings downward.

FLOP accounting (Appendix A). The paper includes a detailed FLOP analysis of forward-pass token mixer costs, which is used in Section 4.4 to construct the performance-efficiency Pareto front (Figure 5). The key formulas, aggregated over all heads per token per layer, are:

For softmax self-attention, the quadratic term dominates:

$\text{FLOPs}_{\text{softmax}} \approx 2 L^2 d_{\text{model}}$

where $L$ is the sequence length and $d_{\text{model}}$ is the model dimension. The head count $H$ cancels because the per-head cost is $2L^2 d_{\text{head}}$ and $H d_{\text{head}} = d_{\text{model}}$.

For vector-state mixers (HGRN):

$\text{FLOPs}_{\text{vector}} = 5 d_{\text{model}}$

where the constant 5 reflects the five element-wise operations per token (computing the gate, applying the gate to the previous state, computing the complement gate, applying it to the value, and the read-out). For HGRN specifically, $H = 1$ (single-head by design), so $d_{\text{head}} = d_{\text{model}}$ and the cost is $5 d_{\text{model}}$.

For matrix-state mixers, the cost depends on the number of $d_{\text{head}}^2$ passes per token:

$\text{FLOPs}_{\text{matrix}} = k \frac{d_{\text{model}}^2}{H}$

where $k$ is the pass count (5 for RetNet/Mamba-2, 7 for GLA/RWKV-6/HGRN-2, 8 for DeltaNet/GatedDeltaNet) and $H$ is the number of heads. Adding more heads reduces per-token cost at fixed $d_{\text{model}}$ because each head operates on a smaller $d_{\text{head}} \times d_{\text{head}}$ matrix: $H d_{\text{head}}^2 = d_{\text{model}}^2 / H$.

What this FLOP accounting reveals: At sequence length $L = 4,096$, the quadratic softmax term ($2 \times 4096^2 \times d_{\text{model}}$) dominates the matrix-state cost ($k d_{\text{model}}^2 / H$) for typical model dimensions. At $L = 32,768$ (Figure 5b), the gap widens further. The vector-state HGRN is orders of magnitude cheaper than both—$5 d_{\text{model}}$ vs. $\mathcal{O}(d_{\text{model}}^2)$—explaining its position at the extreme efficiency end of the Pareto front. However, the paper explicitly cautions (Section 4.4) that "FLOPs in the forward pass do not directly translate into throughput or latency on conventional hardware" because HGRN's element-wise vector operations may underutilize GPU tensor cores and be bottlenecked by memory bandwidth, while the larger matrix operations in outer-product models can achieve higher hardware utilization.

Summary of the experimental design logic. The framework is constructed to answer three questions through comparison:

1. Does better standalone mean better hybrid? Compare the same linear backbone in pure vs. hybrid form; check whether the ranking changes.
2. What does the ratio control? Vary the ratio for a fixed backbone and observe which benchmarks change and which don't.
3. What architectural properties matter? Compare backbones that differ only along one design axis (gating type, hierarchy, forgetting mechanism) at the same ratio and observe performance differences.

4. Key Insights and Innovations

Innovation 1: Reframing Hybrid Architecture Selection from an Assumption-Driven to an Evidence-Driven Process

The dominant assumption in the linear attention community prior to this paper was what I'll call the transferability assumption: a linear model that performs well in standalone evaluation will naturally serve as the strongest backbone when hybridized with full attention. This assumption was not stated explicitly in prior work—it was operationalized implicitly through design decisions. Jamba (Lieber et al., 2024) chose Mamba because Mamba was state-of-the-art among linear models; MiniMax-01 (2025) chose Lightning Attention because Lightning scaled well in pure form; RecurrentGemma (Botev et al., 2024) chose a gated linear recurrence because Griffin showed strong standalone results. In each case, the linear component was selected based on its reputation in isolation, then the ratio was swept to optimize hybrid performance. The transferability assumption was baked into the methodology.

This paper's most conceptually important move is to surface this assumption and systematically falsify it. The finding is not that the assumption is sometimes wrong—it's that the ranking of linear models entirely inverts once full-attention layers are introduced. At the 340M scale (Table 2), GatedDeltaNet achieves the highest standalone accuracy (0.545 average LM score), leading both HGRN-2 (0.539) and the Transformer baseline (0.544). Yet when hybridized at a 6:1 ratio, HGRN-2 jumps to 0.556—surpassing every configuration in the study, including GatedDeltaNet's best hybrid (0.550 at 24:1). The same pattern holds at 1.3B parameters (Table 3): GatedDeltaNet leads in pure form, but HGRN-2 matches it in hybrid form and offers better recall with fewer full-attention layers.

This is not an incremental correction—it's a fundamental reframing of how the field should approach hybrid architecture design. The transferability assumption was reasonable on its face: if model A is better at language modeling and recall than model B when both use the same computational primitives, shouldn't A's advantage persist when you add a few full-attention layers? The paper's empirical answer is a clear "no," and the theoretical explanation—though not formally proven—is that hybridization introduces a new factor: the interaction between the linear layers' state management strategy and the sparse full-attention layers' global context. A model like GatedDeltaNet, which excels at recall by implementing precise targeted forgetting (the delta-rule erase-then-write mechanism), may find its sophisticated state management partially redundant when full-attention layers already provide global token-to-token interaction at regular intervals. Conversely, HGRN-2's hierarchical recurrence—with one pathway maintaining coarse long-range summaries and another handling token-level detail—may complement the full-attention layers more effectively by maintaining context between them that is structured at multiple timescales, rather than attempting to solve the entire long-range dependency problem within the linear layers themselves.

The significance of this reframing extends beyond model selection. If standalone performance doesn't predict hybrid performance, then the entire existing literature on linear model comparisons—which overwhelmingly reports results in pure form—cannot be directly applied to hybrid architecture design. This paper provides the first systematic evidence that such comparisons are necessary rather than redundant: you cannot shortcut hybrid architecture selection by looking at standalone benchmarks; you must evaluate in hybrid form. This is a methodological contribution that changes how future research on linear attention should be conducted and reported.

---

Innovation 2: Decoupling Language Modeling and Recall as Independent Axes of the Hybrid Tradeoff

Prior hybrid architecture work—Jamba, StripedHyena, Hymba, Mamba-2-Hybrid—treated the linear-to-full attention ratio as a knob that controls a single dimension of model quality, typically measured by perplexity or downstream task accuracy. The implicit model was: more full attention → better model, but with diminishing returns, so find the ratio where the perplexity curve flattens. This framing collapses language modeling capability and recall capability into a single aggregate metric, obscuring the fact that these two capabilities have qualitatively different relationships with the hybridization ratio.

The paper's second key innovation is to measure these two capabilities independently and demonstrate they respond to the ratio in fundamentally different ways. Figure 3 makes this distinction visible: the left panel shows language modeling scores clustering in a tight band (0.55–0.57 average accuracy) across all ratios from 24:1 to 3:1, with no systematic trend—the curves are flat. The right panel shows RULER recall scores rising sharply and systematically as full-attention layers are added, nearly doubling from pure linear (~0.1–0.35) to 3:1 (~0.40–0.45), with most architectures approaching the Transformer baseline at the most attention-heavy configuration.

This is not a small quantitative difference—it's a qualitative decoupling that redefines what the ratio knob actually controls. Language modeling, as measured by these benchmarks, is primarily a test of local coherence, commonsense reasoning, and factual knowledge—capabilities that linear attention mechanisms handle reasonably well on their own. Adding full-attention layers provides negligible additional benefit because the linear layers are already adequate for these tasks. Recall, on the other hand, requires the model to store information at one position in a long sequence and accurately retrieve it at a much later position—a capability where linear models' fixed-size hidden states are fundamentally limited by interference (the "state crowding" problem) and where full-attention's exact, lossless KV cache provides a qualitatively different mechanism.

This decoupling has direct practical implications that the paper makes explicit: if you tune your hybrid ratio by minimizing validation perplexity, you will observe a nearly flat loss landscape and likely settle on whatever ratio is convenient—potentially missing a 2× improvement in recall that comes from a more balanced allocation. Conversely, if you need high recall (e.g., for long-document question answering, multi-turn conversation with distant references, or code generation with long-range dependencies), you should tune the ratio explicitly on recall benchmarks like RULER, not on language modeling loss. The paper's Figure 4 further refines this picture by breaking down RULER subtasks: single-key, multi-key, and QA subtasks are sensitive to the ratio, while Common Word Extraction and Frequent Word Extraction are not—suggesting that the ratio primarily affects tasks requiring precise retrieval of specific stored information rather than simpler pattern-matching tasks.

The intellectual move here is breaking a conflation that the field had been operating under. Prior work talked about the "performance" of hybrid models as a unitary concept, sweeping ratios and reporting perplexity. This paper demonstrates that "performance" in a hybrid model has at least two orthogonal dimensions—language modeling and recall—that must be measured and optimized separately because the architectural decisions that affect them are distinct.

---

Innovation 3: Identifying a Triad of Architectural Properties Necessary for Hybrid Effectiveness

The paper's third contribution is moving beyond model rankings to identify which architectural properties enable a linear backbone to hybridize effectively. This is not a benchmarking result—it's a diagnostic analysis that extracts design principles from the empirical comparisons.

The triad identified—selective gating, hierarchical recurrence, and controlled forgetting—emerges from contrasting models that differ along specific design axes:

• Selective gating is isolated by comparing RetNet (fixed exponential decay, no input-dependent gating) against GLA and HGRN-2 (learned, input-dependent gating). RetNet's recall performance is near zero regardless of the hybridization ratio (the paper explicitly notes in Figure 4's caption that "RetNet and HGRN model families are omitted as their recall benchmark results were insignificant"). Even with full-attention layers present at a 3:1 ratio, RetNet fails to store and retrieve information effectively. This demonstrates that gating is not a nice-to-have optimization—it's necessary for a linear backbone to benefit from the sparse full-attention context. Without input-dependent gating, the linear layers' state is overwhelmed by irrelevant information and cannot preserve the long-range dependencies that the full-attention layers are supposed to complement.

• Hierarchical recurrence is isolated by comparing HGRN-2 (two-pathway hierarchical organization) against its "single-path ablation" HGRN (Gen-1 vector recurrence) and against GLA (unstructured per-channel gating). HGRN-2 achieves roughly double the recall of HGRN and substantially outperforms GLA in hybrid form despite having similar per-token FLOP costs. The paper's interpretation—which is observational, not proven through controlled ablation—is that the hierarchical structure provides multi-timescale memory that complements the intermittent full-attention layers: the slow pathway maintains coarse context between full-attention refreshes, while the fast pathway handles token-level detail. GLA's unstructured gating, while more flexible in principle (per-channel learned gates), apparently doesn't organize the state as effectively for the specific demands of a hybrid stack.

• Controlled forgetting is isolated by comparing GatedDeltaNet and HGRN-2 against models without explicit forgetting mechanisms. The paper notes that while GatedDeltaNet achieves controlled forgetting through the delta-rule erase-then-write operation, HGRN-2 achieves a similar effect through gated diagonal decay—both prevent the unbounded accumulation of information that plagues purely additive updates. Models lacking this property, such as early linear attention variants with simple additive outer-product accumulation, suffer from state crowding where old and new information interfere destructively. The paper doesn't have a pure-additive baseline in its comparison set, but it argues from the contrast between GatedDeltaNet/HGRN-2 (strong recall) and GLA (weaker recall despite similar capacity) that some form of explicit information removal—whether subtractive or decay-based—is essential.

The significance of this triad is that it provides a vocabulary and set of design criteria for evaluating future linear attention mechanisms as hybrid backbones. Rather than asking "which model performs best in our benchmark," a practitioner can ask: does this model have input-dependent gating? Does it organize memory hierarchically? Does it explicitly control what information is removed from the state? The claim is that all three properties are jointly necessary—a model missing any one will show degraded recall in hybrid form, regardless of the hybridization ratio. Table 6 supports this by showing that HGRN-2 and GatedDeltaNet—the only models possessing all three properties in the comparison set—are the only ones that achieve Transformer-level recall at any ratio.

---

Innovation 4: Characterizing the Recall Saturation Threshold as a Practical Design Constraint

The paper's fourth insight is more empirical than conceptual, but it has direct engineering significance: recall performance saturates sharply around a 3:1 linear-to-full ratio, and additional full-attention layers beyond this point provide negligible further benefit. This is visible in Figure 3's right panel, where most architectures' RULER curves rise steeply from 24:1 to 3:1 and then flatten—the Transformer baseline (dashed line at approximately 0.42) is approached or exceeded at 3:1, but moving to even more full-attention would offer minimal gains.

This saturation point has been observed qualitatively in prior work, but the paper provides the first quantitative characterization across multiple linear backbones, showing it is architecture-independent—the saturation occurs at roughly the same ratio for all models that are capable of reaching Transformer-level recall at all (HGRN-2, GatedDeltaNet, and to a lesser extent GLA and DeltaNet). This suggests the saturation is a property of the hybridization paradigm itself rather than of any particular linear attention mechanism: one full-attention layer every three linear layers provides sufficient global context to close most of the recall gap with a pure Transformer.

The practical consequence—articulated in the paper's Section 4.5 deployment recipe—is that the ratio does not need to be pushed to extreme values. Since KV-cache memory scales with the number of full-attention layers, and recall saturates around 3:1, there is a strong economic case for operating at or slightly above this ratio (the paper recommends 3:1 to 6:1) rather than the 12:1 or 24:1 configurations that minimize memory at the cost of recall. At 3:1, the KV cache is approximately 4× smaller than a pure Transformer (since only one in four layers contributes to the cache); at 6:1, it is 7× smaller. These are substantial practical savings that come with near-Transformer recall, making the saturation point a Pareto-optimal operating region for hybrid model deployment.

What distinguishes this from a simple "tune the ratio" finding is its framing as a constraint on the design space: if the ratio needs to be roughly 3:1 to achieve acceptable recall, then the minimum achievable KV-cache compression for a recall-competitive hybrid is approximately 4×—you cannot arbitrarily reduce the number of full-attention layers without crossing a sharp recall cliff. This bounds the asymptotic efficiency of the block-wise hybrid paradigm and motivates investigation into alternative hybridization strategies (head-wise mixing as in Hymba, prefill-decode hybrid as in YOCO) that might push the compression factor higher without sacrificing recall.

---

These four innovations together constitute the paper's intellectual contribution: a reframing of how hybrid architectures should be selected (away from assumed transferability and toward evidence-driven evaluation), a decoupling of language modeling and recall as independent dimensions of the hybrid tradeoff, a diagnostic framework of three necessary architectural properties for effective hybridization, and an empirically-grounded characterization of the recall saturation threshold as a practical design bound. None of these is a new mechanism or algorithm—the paper is explicitly empirical and analytical, not a methods proposal—but they collectively change how a practitioner or researcher should think about designing, evaluating, and deploying hybrid linear attention models.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All models are pretrained on fineweb-edu (Penedo et al., 2024), a high-quality filtered subset of CommonCrawl web text selected for educational value. The 340M models consume 20 billion tokens; the 1.3B models consume 100 billion tokens. Evaluation benchmarks are described below under Metrics.

• Base model(s). The study does not use a single pretrained base model. Instead, 72 models are trained from scratch: 36 at 340M parameters and 36 at 1.3B parameters, spanning six linear attention variants (HGRN, RetNet, GLA, HGRN-2, DeltaNet, GatedDeltaNet) × five hybridization ratios (24:1, 12:1, 6:1, 3:1, pure linear), plus a standard Transformer baseline at each scale. All models are implemented using the flash-linear-attention library (Yang and Zhang, 2024). The two parameter scales are chosen to bracket a range where language modeling capability is measurable at both scales but recall differences only become detectable at 1.3B—the paper explicitly notes that "340M models show insignificant recall capability due to their relatively small parameter count" (Table 2 caption), so RULER results are reported only at the larger scale.

• Metrics. Two independent capability axes are measured. Language modeling is assessed via zero-shot accuracy on six benchmarks: ARC-Challenge, ARC-Easy (Clark et al., 2018), HellaSwag (Zellers et al., 2019), LAMBADA (Paperno et al., 2016), OpenBookQA (Mihaylov et al., 2018), and PIQA (Bisk et al., 2019). The "average LM score" reported in tables and Figure 3 is the arithmetic mean of these six accuracy values. Recall is measured using the RULER suite (Hsieh et al., 2024), which includes subtasks for single-key retrieval, multi-key retrieval, multi-hop tracing, aggregation, question answering, Common Word Extraction (CWE), and Frequent Word Extraction (FWE). The "average RULER score" in Figure 3 is the aggregate across subtasks. All evaluations use a 2,048-token context window and are conducted in a zero-shot manner without task-specific fine-tuning or prompt engineering.

• Baselines. The primary baselines are the pure Transformer (standard softmax self-attention at all layers) and the pure linear variant of each architecture (no full-attention layers interleaved). These establish the upper and lower bounds for each capability axis. Within the hybrid configurations, the paper treats the transferability assumption—that a better standalone linear model produces a better hybrid—as an implicit baseline hypothesis to be tested, rather than comparing against a specific prior hybrid architecture.

• Generation budget / compute accounting. The paper does not use a per-sample generation budget (as in best-of-N or beam search studies) since it evaluates pretrained models rather than test-time inference strategies. Instead, training compute is controlled by fixing the total training tokens per model size (20B for 340M, 100B for 1.3B) and using identical optimization hyperparameters across all configurations. For efficiency analysis in Section 4.4, forward-pass FLOPs of the token mixer are computed using the formulas derived in Appendix A: softmax attention costs 2L²d_model, vector-state mixers cost 5d_model, and matrix-state mixers cost k·d_model²/H where k is the pass count (5 for RetNet/Mamba-2, 7 for GLA/RWKV-6/HGRN-2, 8 for DeltaNet/GatedDeltaNet). These FLOP counts are per-token per-layer and do not include projections, MLPs, norms, or residuals, enabling a clean comparison of the mixing mechanisms in isolation.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. Results are reported as single-point accuracy values without confidence intervals. The analysis relies on consistency across the 72-model matrix and both parameter scales to establish reliability, rather than formal statistical procedures.

Main Quantitative Results

4.1 Standalone vs. Hybrid Performance: The Transferability Assumption Is Falsified

The central empirical finding is that standalone linear model quality does not predict hybrid performance—and in key cases, the ranking inverts entirely.

At 340M parameters (Table 2): GatedDeltaNet achieves the highest standalone accuracy among linear models (0.545 average LM score), leading HGRN-2 (0.539) and matching the Transformer baseline (0.544). However, when hybridized, HGRN-2 at 6:1 ratio reaches 0.556—exceeding the Transformer baseline by 1.2 percentage points and outperforming GatedDeltaNet's best hybrid configuration (0.550 at 24:1 ratio). This is a genuine inversion: the model that ranks first in pure form does not win in hybrid form.

The full ranking at 340M for best hybrid ratios versus standalone:

The improvement from hybridization (Δ) varies dramatically by architecture: HGRN-2 gains 1.7 points, HGRN gains 1.9 points, GLA gains 1.8 points—while GatedDeltaNet gains only 0.5 points and DeltaNet gains only 0.3 points. The models that benefit most from hybridization are not the strongest in standalone form.

At 1.3B parameters (Table 3): The pattern holds but with some nuances. GatedDeltaNet again leads in standalone form (0.590), with HGRN-2 at 0.586. After hybridization, GatedDeltaNet at 24:1 and HGRN-2 at 6:1 achieve equivalent performance—the paper states "several other hybrid configurations within one percentage point." The ranking inversion is less dramatic at this scale, but the transferability assumption still fails in the sense that the standalone leader does not outperform alternatives in hybrid form. Critically, HGRN-2 achieves comparable language modeling at a more memory-efficient ratio (6:1 vs. 24:1 for GatedDeltaNet), meaning it delivers similar quality with substantially more KV-cache compression.

The delta in delta: The paper's Tables 2 and 3 do not report pure linear accuracy for all models in a single unified table, but the pattern is consistent enough to support a strong conclusion: the correlation between standalone and hybrid performance is weak or negative across architectures. The models that gain the most from hybridization (HGRN, HGRN-2, GLA) are mid-tier in standalone form; the models that are strongest standalone (GatedDeltaNet, DeltaNet) gain the least. This suggests that hybridization introduces a complementary interaction effect that is independent of—and potentially orthogonal to—standalone linear model quality.

4.2 Impact of the Linear-to-Full Attention Ratio: Language Modeling Is Flat, Recall Scales

Figure 3 provides the paper's most visually compelling result, showing the decoupling of language modeling and recall as the ratio varies.

Language modeling (Figure 3, left panel): The average LM scores cluster tightly across all architectures and all ratios. Most configurations fall in the 0.55–0.57 range at 1.3B parameters, with no systematic trend as the ratio changes from 24:1 to 3:1. The curves are effectively flat. The paper characterizes this as "language modeling performance remains largely flat across all ratio configurations" and notes that "practitioners can freely optimize the linear to full attention ratio with minimal effect on language modeling performance."

Recall (Figure 3, right panel): The RULER scores show a clear upward trend as the proportion of full-attention layers increases. At pure linear configurations, scores span roughly 0.10–0.35 depending on architecture. At the 3:1 ratio, most architectures cluster around 0.40–0.45, with HGRN-2 and GatedDeltaNet exceeding the Transformer baseline (dashed line at approximately 0.42). The paper states that "recall rises steadily as full-attention layers are added and saturates around 3:1," and that "all architectures show a clear upward trend as the proportion of full-attention layers increases."

The magnitude of improvement is substantial: moving from pure linear to 3:1 ratio nearly doubles RULER scores for architectures capable of recall (HGRN-2, GatedDeltaNet, GLA, DeltaNet), while RetNet and HGRN show near-zero recall at all ratios—their omission from Figure 4's per-subtask breakdown is explicitly noted ("RetNet and HGRN model families are omitted as their recall benchmark results were insignificant").

Subtask breakdown (Figure 4): The paper further decomposes RULER performance by subtask class, revealing that the recall improvement is not uniform across all subtask types. Single-key retrieval, multi-key retrieval, and QA subtasks are "noticeably affected by a changing ratio, with higher concentrations of full attention performing better than lower concentrations." In contrast, Common Word Extraction (CWE) and Frequent Word Extraction (FWE) do not correlate with the hybrid ratio—these simpler lexical retrieval tasks appear solvable by the linear layers alone. This granularity refines the ratio-recall relationship: the ratio knob specifically controls structured information retrieval (finding specific stored facts, following chains of references, answering questions requiring context), not generic pattern matching.

The saturation threshold: The paper makes a specific quantitative claim about the saturation point: recall "saturates around 3:1." This is visible in Figure 3's right panel, where the delta between 6:1 and 3:1 configurations is small for the best-performing models. HGRN-2 and GatedDeltaNet show essentially flat RULER curves from 6:1 to 3:1—they have already approached the Transformer baseline. This establishes 3:1 as the minimum ratio needed for Transformer-level recall, which maps to a 4× reduction in KV-cache memory compared to a pure Transformer (since only one in four layers maintains a cache).

4.3 Architectural Determinants: The Triad of Necessary Properties

Section 4.3 and Tables 4–6 present evidence that three architectural properties jointly predict hybrid effectiveness. This is an observational analysis—no controlled ablation experiments are performed—but the cross-architecture comparisons are structured to isolate each property.

Selective gating is necessary for any recall. This is demonstrated by RetNet's failure: despite having a matrix-valued state with Gen-2 capacity (d² scalars per head), RetNet's fixed exponential decay yields "near-zero recall even when full-attention layers are added" (Section 4.3). Table 5 shows RetNet's hybrid RULER scores are effectively zero regardless of ratio. GLA, with fully learned per-channel gating, achieves substantially better recall (rising from ~0.30 to ~0.40 across ratios), confirming that input-dependent gating—not just matrix capacity—is what enables the linear layers to selectively preserve information that the full-attention layers can later leverage. The HGRN (Gen-1) failure is also attributed to lack of selective gating, though at Gen-1, the vector bottleneck is confounded with the gating question.

Hierarchical recurrence provides multi-timescale memory. The paper compares HGRN-2 (two-pathway hierarchical) against HGRN (single-path vector) and GLA (unstructured per-channel gating). HGRN-2's recall in hybrid form "doubles" that of HGRN and substantially exceeds GLA. Table 6 specifies the best RULER results per architecture: HGRN-2 (6:1) achieves 0.454, GatedDeltaNet (3:1) achieves 0.430, while DeltaNet (3:1) achieves 0.412 and GLA (3:1) achieves 0.385. The paper interprets the HGRN-2 advantage over GLA—two models with comparable FLOP costs (7 passes, same asymptotic complexity)—as evidence that the hierarchical inductive bias provides a complementary structure to the intermittent full-attention layers: "widely spaced full-attention layers benefit from a recurrent hierarchy that can 'latch' information between them." This is the most interpretive (and least directly tested) of the three claims; no experiment varies the hierarchical structure independently of other factors.

Controlled forgetting prevents state crowding. GatedDeltaNet and HGRN-2 are the only models achieving Transformer-level recall in hybrid form (exceeding the dashed line in Figure 3, right panel). The paper attributes this to their forgetting mechanisms: GatedDeltaNet's delta-rule erase-then-write and HGRN-2's gated diagonal decay both "prevent the unbounded accumulation that plagues purely additive updates." GLA, despite having per-channel learned gating, only applies multiplicative decay—it cannot explicitly remove stale information aligned with a new key—and its recall saturates below the Transformer baseline. Table 5 shows GLA's best RULER is 0.385 at 3:1 versus GatedDeltaNet's 0.430 at 3:1, a meaningful gap. The interpretation is that decay alone is insufficient; explicit content-aware removal is necessary for the linear state to remain useful alongside full-attention layers.

Interaction with the ratio (Table 6): The paper makes the claim that "architectures lacking either gated or delta-style forgetting never reach Transformer-level recall, regardless of ratio." Table 6 supports this: only HGRN-2 and GatedDeltaNet exceed the Transformer recall baseline at any ratio. DeltaNet (0.412 at 3:1) approaches but does not exceed the baseline; GLA (0.385), HGRN (~0.0), and RetNet (~0.0) never reach it, even at the most attention-heavy 3:1 configuration. This implies that the ratio alone cannot compensate for missing architectural properties—adding more full-attention layers helps recall, but there is a hard ceiling determined by the linear backbone's state management capabilities.

Tables 4 and 5 provide the aggregated data: Table 4 averages LM performance across 1.3B and 340M scales per architecture per ratio, confirming the flat trend (total averages: 0.557, 0.553, 0.551, 0.555 for 24:1 through 3:1 respectively—a range of only 0.006). Table 5 averages recall at the 1.3B scale, showing the steep improvement (from pure linear averages around 0.14–0.23 for capable architectures to 0.42+ at 3:1 for the best models).

4.4 Performance-Efficiency Pareto Front

Figure 5 maps the FLOPs-versus-performance tradeoff for all architectures at two sequence lengths, using the FLOP formulas from Appendix A.

At L = 4,096 (Figure 5a): Pure HGRN occupies the extreme efficiency position (lowest FLOPs, ~10⁷, at 0.525 average LM loss, which corresponds to ~0.525 average accuracy—note the y-axis is inverted so higher is better performance), while the pure Transformer occupies the highest-performance, lowest-efficiency position (~10⁹ FLOPs, ~0.562 accuracy). The Pareto front is formed by HGRN 24:1 and HGRN-2 6:1 in the middle ground. GatedDeltaNet pure and hybrid variants sit well to the right (higher FLOPs) for comparable or slightly better performance, meaning they are Pareto-dominated by HGRN-2 and HGRN hybrids.

At L = 32,768 (Figure 5b): The quadratic cost of full attention pushes models with any attention layers further right. Pure HGRN and pure Transformer remain as the extreme endpoints. Notably, pure RetNet, pure HGRN-2, and pure GatedDeltaNet now occupy the lower-performance half of the Pareto front, with pure HGRN-2 being the most efficient matrix-state model at long sequences. The paper's interpretation is that pure matrix-state models without any full-attention layers become competitive in efficiency at long sequences—their O(L d²) cost grows linearly, while any model containing full attention pays the O(L² d) price that dominates at L = 32,768.

A critical caveat: the paper explicitly warns that "FLOPs in the forward pass do not directly translate into throughput or latency on conventional hardware." HGRN's vector operations may underutilize GPU tensor cores, making its FLOP advantage not fully realizable as wall-clock speedup on standard hardware. The Pareto front should be understood as a theoretical efficiency bound, not a throughput benchmark.

Ablation Studies and Robustness Checks

Ratio sweep as implicit ablation: The paper's primary "ablation" is the systematic sweep of linear-to-full attention ratios (24:1, 12:1, 6:1, 3:1, plus pure linear) for every architecture. This serves as an ablation of the number of full-attention layers while holding the linear backbone constant. The key finding—that LM scores remain flat while recall rises—is robust across all six architectures (Figure 3). This consistency across varied backbones strengthens the claim that the decoupling is a property of the hybridization paradigm rather than an artifact of a specific model.

Architecture vs. architecture at fixed ratio: By comparing different linear backbones at the same ratio, the paper implicitly ablates the gating mechanism while holding the amount of full attention constant. At the 6:1 ratio and 1.3B scale, HGRN-2 achieves 0.454 RULER while GLA achieves approximately 0.37—this gap isolates the effect of hierarchical recurrence versus unstructured per-channel gating, since both models have comparable state capacity and FLOP costs. However, this is not a controlled ablation in the strict sense because HGRN-2 and GLA differ in multiple ways beyond hierarchical organization (tied gate vs. per-channel gate, specific gating function formulation), so the attribution to hierarchy is inferential rather than proven.

Scale consistency check: The pattern of standalone vs. hybrid ranking inversion appears at both 340M (Table 2) and 1.3B (Table 3) scales, with the same general trend: GatedDeltaNet leads standalone, HGRN-2 leads hybrid. At 1.3B, the inversion is less dramatic—GatedDeltaNet at 24:1 and HGRN-2 at 6:1 achieve equivalent performance—but the ranking still changes relative to standalone expectations. This consistency across scales provides some evidence that the finding is not an artifact of a particular model size, though the limited range (340M–1.3B) means extrapolation to larger scales remains speculative.

Sequence length sensitivity (Figure 5a vs. 5b): The Pareto front analysis at two sequence lengths (4,096 and 32,768) serves as a robustness check on the efficiency claims. At longer sequences, the relative efficiency advantage of pure linear models grows dramatically, as expected from the asymptotic complexity analysis. This confirms that the FLOP accounting approach produces internally consistent results, even if it doesn't capture hardware-specific throughput effects.

Negative result—RetNet and HGRN fail completely on recall: The paper includes an important negative result in Figure 4's caption: "RetNet and HGRN model families are omitted as their recall benchmark results were insignificant." Even at the most attention-heavy 3:1 ratio, these architectures cannot perform structured retrieval. This negative result is crucial because it establishes a clear boundary: without input-dependent gating and matrix-valued state capacity, hybridization alone cannot recover recall. It prevents the misinterpretation that "any linear model + enough full attention = good recall."

Negative result—340M models cannot measure recall: Table 2's note that "340M models show insignificant recall capability" is itself a finding about the minimum scale required for evaluating recall in hybrid architectures. It bounds the practicality of future studies: below 1.3B parameters, RULER benchmarks may not provide meaningful signal, so recall-focused architectural comparisons need to operate at or above this scale.

FLOP analysis as ablation of confounding cost factors: Appendix A's detailed FLOP derivations serve as an ablation of computational cost from the performance comparisons—by explicitly accounting for the per-token per-layer cost of each mixer, the paper ensures that performance differences are not simply artifacts of one architecture using more compute per token. The finding that HGRN-2 (7 passes) outperforms GLA (also 7 passes) in hybrid form, and that HGRN-2 (7 passes) outperforms GatedDeltaNet (8 passes) in most hybrid configurations, demonstrates that raw FLOP count does not predict hybrid quality—the architectural properties matter independently of cost.

Critical Assessment

The experiments provide strong evidence for the paper's central claim that standalone linear model quality does not predict hybrid performance. The ranking inversion at 340M (Table 2)—GatedDeltaNet leading in pure form but HGRN-2 winning in hybrid form—is a clean, specific result that directly falsifies the transferability assumption for those two architectures at that scale. The 1.3B results (Table 3) are more ambiguous: the two models achieve equivalent performance, which does not invert the ranking but does refute the assumption that the standalone leader will dominate in hybrid form. The paper's claim that HGRN-2 matches GatedDeltaNet with a more memory-efficient ratio (6:1 vs. 24:1) is supported by the data, but the magnitudes are small enough that at this scale, the practical difference between "equivalent" and "statistically indistinguishable" matters.

The decoupling of language modeling and recall (innovation 2) is the most robustly supported finding in the paper. Figure 3 shows flat LM curves and sharply rising RULER curves across all architectures—this pattern is consistent, visually unambiguous, and would be difficult to produce as an artifact. The claim that the linear-to-full ratio primarily controls recall rather than language modeling is empirically solid within the tested conditions.

The triad of architectural properties (innovation 3) is the least directly tested of the paper's claims. The evidence for selective gating is strong—RetNet's failure is unambiguous, and GLA's improvement over RetNet isolates the effect of making gates input-dependent. The evidence for controlled forgetting is suggestive but not dispositive: GatedDeltaNet and HGRN-2 perform best in recall, and both have forgetting mechanisms, but GLA's weaker recall could also be attributed to its lack of hierarchy or to suboptimal hyperparameters rather than the specific forgetting mechanism. The evidence for hierarchical recurrence is entirely observational—HGRN-2 outperforms GLA, and the paper attributes this to hierarchy, but the two models differ in multiple ways (tied vs. per-channel gates, specific architecture details) that are not independently varied. A clean ablation would require a version of HGRN-2 with hierarchical structure removed but all other factors held constant, which the paper does not provide. The authors acknowledge this limitation in Section 4.3: "The analysis below is necessarily observational. Ablations are left to future work." Readers should treat the triad as well-motivated hypotheses derived from systematic comparison rather than as proven causal mechanisms.

The recall saturation threshold of 3:1 (innovation 4) is supported by Figure 3's right panel for the architectures that achieve meaningful recall, but the exact saturation point depends on how "saturation" is defined. If defined as "approaching or exceeding the Transformer baseline," then 3:1 is correct for HGRN-2 and GatedDeltaNet. If defined as "the point where additional full-attention layers provide negligible marginal improvement," then the data is coarser—the paper tests only four ratios (24:1, 12:1, 6:1, 3:1) plus the endpoints, so the true saturation could occur at 4:1, 5:1, or even 2:1. A finer-grained ratio sweep (e.g., 2:1, 4:1, 8:1) would be needed to precisely characterize the saturation curve.

Several significant weaknesses limit the generalizability of the findings:

Single dataset, single training paradigm. All models are trained on fineweb-edu only. Whether the findings hold for different data distributions (code, multilingual text, domain-specific corpora) is untested. The flat LM curve across ratios might reflect properties of web text language modeling specifically—tasks requiring more structured long-range reasoning (legal document analysis, scientific literature review) might show different ratio sensitivity.

Single evaluation context length (2,048 tokens). The RULER evaluation uses only 2,048-token contexts. The paper's own motivation emphasizes long-context applications, yet the recall evaluation operates at a sequence length where vanilla Transformers are not meaningfully bottlenecked. At 2,048 tokens, the quadratic attention cost is approximately 4M operations per head—hardly prohibitive—and the KV cache is only ~8MB per layer for a 1.3B model. The recall saturation at 3:1 might shift at 32k or 128k contexts, where the linear layers are operating far beyond their typical training length and the full-attention layers become proportionally more expensive. The paper acknowledges this limitation ("Whether the observed trade-offs persist at... 128k token contexts... remains open"), but it is a significant gap between the paper's motivating problem (quadratic complexity at long sequences) and the experimental conditions (short-context evaluation).

No instruction tuning or downstream task fine-tuning. All evaluations are zero-shot on pretrained models. Instruction-tuned models might exhibit different recall behavior—the fine-tuning process could teach the model to utilize the full-attention layers differently, potentially changing the optimal ratio or the ranking of linear backbones. The paper recommends practical deployment configurations based on pretraining-only results, which may not transfer to instruction-tuned deployments.

The 1.3B models are small relative to state-of-the-art. Production hybrid models operate at 7B–400B parameters. The paper's 1.3B scale, while reasonable for a systematic study, leaves open the question of whether the architectural property findings scale. Liu et al. (2025) showed that state-size scaling for RNN LLMs exhibits diminishing returns, but this paper's triad of properties (gating, hierarchy, controlled forgetting) has not been validated at larger scales. The paper acknowledges this ("Verifying the trend at larger scales remains future work"), but readers should not assume that HGRN-2 at 6:1 will be optimal at 70B parameters based on 1.3B evidence alone.

No head-wise or dynamic hybridization comparisons. The paper studies only block-wise interleaving at fixed ratios. Alternative approaches—head-wise mixing (Hymba, Dong et al., 2024), prefill-decode hybrids (YOCO, Sun et al., 2024), or dynamic routing between layer types—are not compared. The paper's claims about optimal ratios are specific to the block-wise paradigm and may not generalize to more fine-grained hybridization schemes that could potentially achieve better recall at higher compression ratios.

No formal statistical testing. All results are reported as single-point estimates without confidence intervals, standard deviations, or statistical significance measures. Given the 500-question test sets and the relatively small differences between top-performing configurations (e.g., 0.556 vs. 0.550 at 340M in Table 2), it is unclear whether the reported differences are statistically reliable or within the noise floor of training variance. Training multiple seeds per configuration—even for a subset of architectures—would have substantially strengthened the claims by quantifying the variance due to random initialization and data ordering.

Missing key ablation: hierarchical structure. The paper's most interpretive claim—that hierarchical recurrence is a necessary property for hybrid effectiveness—lacks a direct ablation. An experiment that trained HGRN-2 with and without the hierarchical pathway, holding all else constant, would be needed to establish causality. The current comparison between HGRN-2 and GLA confounds hierarchy with gating structure and other architectural details.

The 340M recall omission creates an evidence gap for scale extrapolation. Because recall is only measured at 1.3B, there is no way to assess whether the recall architecture ranking changes with scale, or whether the recall "floor" (near-zero for RetNet and HGRN) would lift at larger model sizes. If RetNet gained measurable recall at 7B parameters—perhaps because the increased capacity compensates for poor gating—the architectural property claims would need qualification.

These limitations do not invalidate the paper's contributions, but they bound them precisely: the findings provide strong evidence for the 1.3B parameter scale, 2,048-token contexts, block-wise hybridization, and web-text pretraining. Extension to different scales, contexts, hybridization strategies, and training paradigms requires independent verification. The paper is appropriately conservative in stating these boundaries, which is to its credit.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted for and Potentially Dominates the Inference Budget

The assumption or constraint. The entire compute-optimal framework depends on estimating prompt difficulty before allocating the inference budget. The paper's method for doing so—generating 2048 samples per question and averaging correctness—is enormously expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. Since 2048 samples per question exceeds the largest test-time budgets studied (256–512 generations), the difficulty estimation step alone would consume more compute than the entire problem-solving process. The 4× figure should be understood as an upper bound on achievable efficiency—not a realized deployment gain. In latency-sensitive settings, the overhead of running 2048 samples to estimate difficulty would be prohibitive even if the total FLOPs were somehow amortized.

What evidence exists in the paper. The paper provides no experiment that measures the cost of difficulty estimation or that includes it in any budget calculation. The difficulty estimation protocol—2048 samples per question, PRM scoring, binning into quintiles—is described in Section 3.2, but the compute cost of this step is never quantified. The cross-validation protocol in Section 3.2 uses two-fold splits on a 500-question test set, meaning the predicted difficulty bins are computed once on a static evaluation set—there is no online deployment scenario tested where difficulty must be estimated for novel prompts.

Mitigation status. The paper explicitly flags this as future work: "future work on training models to directly predict difficulty from the question text" (Section 8). No such model is developed or evaluated in the paper. The predicted (non-oracle) difficulty bins perform nearly as well as oracle bins (Figures 4 and 8), which shows that the PRM-based difficulty signal is informative, but does not address the cost of obtaining that signal. An adaptive scheme—starting with a few samples, assessing difficulty, and allocating the remaining budget—is mentioned as a possibility but not explored. Without cheap difficulty estimation, the compute-optimal framework remains a conceptual contribution rather than a directly deployable system.

---

6.2 All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. The paper evaluates exclusively on the MATH benchmark with PaLM 2-S* as the base model. The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is unverified. The test set consists of 500 competition-level math problems, all requiring multi-step symbolic reasoning with exact ground-truth answers.

The consequence. Several aspects of the findings could be model-specific or domain-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration or error patterns might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. Most critically, the MATH benchmark consists exclusively of problems with clean, verifiable answers—the entire experimental pipeline (PRM training via Monte Carlo rollouts, difficulty estimation via pass@1, evaluation via exact match grading) depends on having ground-truth correctness signals. Many important real-world tasks lack such clean signals, and the paper provides no evidence that the compute-optimal framework transfers to domains where correctness is ambiguous or multi-dimensional.

What evidence exists in the paper. All tables and figures (Figures 3–9, Tables 2–3 in the original paper) report results exclusively on MATH. There are no experiments on code generation, logical reasoning, scientific QA, or any domain beyond competition math. The PRM training data is generated from PaLM 2-S*'s own outputs (Section 5.1, Appendix D)—there is no cross-model transfer experiment. The paper's finding that the PRM800k dataset was "largely ineffective" for their PaLM 2 models (Section 5.1) actually demonstrates model-specificity: a PRM trained on GPT-4 outputs does not transfer to PaLM 2, suggesting the entire framework may be sensitive to the base model's output distribution.

Mitigation status. The paper does not claim generalizability beyond MATH—the limitations are not hidden. However, no replication on other benchmarks or model families is attempted. Section 8 lists extension to "other reasoning domains (code generation, logical reasoning, scientific QA)" as future work. A practitioner deploying this approach on a different model family or task domain would need to re-validate every component (PRM training, difficulty estimation, strategy selection) from scratch.

---

6.3 The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-Versus-Inference Tradeoff Claims

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, trained using the LLaMA paradigm (fixed training data, scaled parameters only). The authors explicitly acknowledge this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs would scale both parameters and data, likely outperforming a parameter-only-scaled model. This makes the pretraining baseline weaker than it could be, potentially inflating the apparent advantage of test-time compute. The reported gains—for example, +27.8% relative improvement on easy questions at low inference-to-pretraining ratios (Figure 1, top-right bar chart)—might shrink or reverse against a properly compute-optimal larger model. Furthermore, the 14× larger model uses only greedy decoding with no test-time compute augmentation of its own. A fairer comparison would give the larger model some test-time compute budget (even modest best-of-8 could close a meaningful fraction of the gap), but no such comparison is reported. The paper's FLOP accounting formulas (Equations X and Y in Section 7) correctly handle the training-inference tradeoff mathematically, but the empirical comparison implements them with a suboptimal pretraining baseline.

What evidence exists in the paper. The FLOPs-matched comparison is reported in Figure 9, with the bar charts in Figure 1 providing the headline relative differences. The paper explicitly states the design choice and limitation in Section 7, but provides no alternative comparison against a Chinchilla-optimal baseline. The sensitivity analysis over three values of the inference-to-pretraining ratio R (0.16, 0.79, 22) partially addresses this by showing when test-time compute is advantageous even with the current baseline, but does not quantify how much the advantage would shrink with a stronger pretraining baseline.

Mitigation status. The paper acknowledges the limitation and flags it as future work. A practitioner deciding between pretraining and test-time compute based on these results should recognize that the reported advantages of test-time compute represent an upper bound—a compute-optimally trained larger model would reduce or potentially eliminate the advantage in some regimes, particularly on medium-to-hard problems and at high inference-to-pretraining ratios.

---

6.4 The Hardest Problems Are Essentially Unsolved by Any Test-Time Compute Strategy, Establishing a Hard Capability Ceiling

The assumption or constraint. The paper's entire framework—both search against PRM verifiers and iterative revision—operates on the proposal distribution of the base model. It can amplify existing capability but cannot create it. This is not an oversight but a fundamental constraint: if the base model's pass@1 is near zero on a problem class, no amount of search or revision will help because there are no correct solutions in the search space to find or refine.

The consequence. On the hardest difficulty quintile (bin 5), all methods show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods across all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel ratio, even at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line remains essentially flat near 0–5% across all test-time compute budgets, and the 14× larger model consistently outperforms test-time compute on these problems. For the revision model (Figure 1, top-right bar chart), hard problems show a −37.2% relative disadvantage compared to pretraining at high inference-to-pretraining ratios.

This means the compute-optimal framework offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining a larger model remains the only viable approach. The paper's own data (Figure 9) shows that on bin 5, the pretrained 14× larger model outperforms test-time compute in nearly all regimes—these are precisely the problems where capability improvements must come from pretraining, not inference.

What evidence exists in the paper. The difficulty-bin breakdowns in Figures 3 (right), 7 (right), and 9 provide consistent evidence across search, revisions, and FLOPs-matched comparisons that bin 5 problems remain essentially unsolved. The takeaway box in Section 7 explicitly states: "hard problems are essentially unaffected by any amount of test-time compute." The paper is transparent about this limitation.

Mitigation status. The paper does not attempt to solve this limitation—it characterizes it as a fundamental boundary condition. The compute-optimal policy implicitly handles this by allocating minimal test-time compute to bin 5 problems (since no strategy helps), but this is avoidance rather than mitigation. A practitioner encountering a problem distribution skewed toward hard problems should recognize that test-time compute is not a substitute for pretraining—the paper's evidence is clear that pretraining dominates in this regime.

---

6.5 The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate with No Principled Solution

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target. The training data construction procedure (Section 6.1) pairs independently sampled incorrect solutions with a correct solution, using edit distance as a proxy for trajectory coherence. This means the model has never seen a training example where the current answer is already correct and no revision is needed.

The consequence. During inference, the revision model may encounter correct answers produced during earlier steps in the chain and incorrectly "revise" them into wrong answers. The paper reports in Section 6.1 that approximately 38% of correct answers get converted back to incorrect ones. This is a direct consequence of the training data distribution—the model has learned that the in-context answers are always wrong and that it should always produce something different. The mitigation used in the paper—majority voting or verifier-based selection across the entire chain of revisions—is a patch that picks the best answer from historical steps but does not prevent the model from producing incorrect revisions in the first place. This means that longer revision chains do not monotonically improve: later steps can be worse than earlier ones, and the system must maintain and evaluate all intermediate outputs, increasing computational overhead.

What evidence exists in the paper. The 38% reversion rate is stated in Section 6.1 (not measured in a dedicated experiment, but reported as an observed behavior). Figure 6 (left) shows that revision model pass@1 improves gradually across steps—from ~18.2% at step 1 to ~24–25% by steps 15–20—but does not measure how often correct answers are subsequently degraded. The chain-selection mechanism (majority voting or verifier) is described as the mitigation strategy.

Mitigation status. The paper does not propose a principled solution, such as training the model to recognize when no revision is needed (by including "no change required" training examples), or using the verifier's score to terminate the revision chain early when the current answer is likely correct. The acknowledge that the choice to not train on correct-to-correct trajectories creates this problem, but do not explore alternative training data constructions. This is a significant practical limitation: deploying a revision model that incorrectly modifies 38% of already-correct answers requires building infrastructure to detect and mitigate this behavior, which adds complexity and latency to any production system.

---

6.6 Sequential Revisions Are Inherently Serial, Creating a Latency Bottleneck Not Captured by the Generation-Based Compute Model

The assumption or constraint. The paper measures test-time compute in "generations"—the number of complete solutions sampled—which is a reasonable proxy for total FLOPs but ignores wall-clock time and the serial dependencies introduced by sequential revision chains. A strategy that allocates 128 generations as 64 sequential revisions in a single chain requires 64 serial forward passes of the model, each depending on the output of the previous one. In contrast, a fully parallel best-of-128 strategy can execute all 128 generations simultaneously given sufficient hardware.

The consequence. The sequential-heavy strategies favored by the compute-optimal policy—particularly on easy problems, where Figure 7 (right) shows that fully sequential revision is optimal—may be impractical for latency-sensitive applications regardless of their accuracy advantages. A 64-step revision chain takes approximately 64× the wall-clock time of a single generation, making it unsuitable for interactive assistants, real-time decision-making, or any deployment where response time matters. The paper's compute-optimal policy weights all generations equally, ignoring the fact that a sequential generation and a parallel generation have very different latency profiles even if they use the same number of FLOPs. This means the reported "optimal" strategies may not be optimal in a latency-aware setting—a practitioner would need to incorporate a latency penalty or constraint into the strategy selection process, which the paper does not explore.

What evidence exists in the paper. The paper does not report any wall-clock time measurements, latency benchmarks, or throughput analysis. The generation budget is the universal and only unit of test-time compute (Section 3.4). The sequential-to-parallel ratio sweep (Figure 7) implicitly captures the tradeoff—fully sequential (rightmost point) uses the same number of generations as fully parallel (leftmost point)—but presents them as equivalent in cost when they are not equivalent in latency. The FLOPs-matched comparison (Section 7) similarly uses total FLOPs as the cost metric without distinguishing between parallel FLOPs and serial FLOPs.

Mitigation status. The paper does not address this limitation. The revision model's inference procedure (Section 6.1) and the compute-optimal allocation framework (Section 3.1) are both defined purely in terms of generation count. A practitioner deploying these methods would need to perform their own latency analysis and potentially modify the strategy selection to incorporate a latency budget or time-to-first-token constraint. The absence of latency analysis is particularly notable given that Section 1 motivates test-time compute as relevant for "on-device deployment"—a setting where latency and power constraints are often as important as total compute.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a methodological correction to how the linear attention community evaluates and selects architectures for hybrid deployment. Before this work, the dominant assumption—operationalized implicitly in every major hybrid architecture paper from Jamba to MiniMax-01—was that a linear attention mechanism which performs well in isolation will naturally serve as the strongest backbone when interleaved with full-attention layers. This paper systematically falsifies that assumption at two model scales (340M and 1.3B parameters), demonstrating not merely that standalone performance is an imperfect predictor, but that the ranking of linear architectures can invert entirely once full-attention layers are introduced. At 340M parameters (Table 2), GatedDeltaNet leads in pure form (0.545 average LM score) but HGRN-2 wins in hybrid form (0.556 at 6:1, exceeding both the standalone leader and the Transformer baseline). At 1.3B parameters (Table 3), GatedDeltaNet again leads standalone but HGRN-2 matches it in hybrid form at a 4× more memory-efficient ratio (6:1 versus 24:1).

The magnitude of this contribution is best understood as a reframing rather than a paradigm shift. The paper does not introduce a new attention mechanism, a new hybridization scheme, or a new training objective. Instead, it changes which experiments the field should run before making architectural decisions. Prior to this work, a team designing a hybrid model could reasonably survey the linear attention literature, identify the strongest standalone model, and adopt it as their backbone—the methodology was consistent with published best practices. This paper demonstrates that such a procedure is unreliable: the interaction between a linear backbone's state management strategy and the sparse full-attention layers it is paired with can make a mid-tier standalone model outperform the standalone leader. The practical consequence is that architecture selection for hybrid models must be done in hybrid form—standalone benchmarks are necessary but insufficient. This is a methodological contribution that changes the experimental protocol for an entire subfield.

A second shift the paper effects is decoupling language modeling and recall as independent optimization dimensions in hybrid architecture design. Prior work treated the linear-to-full attention ratio as a knob controlling a single axis of model quality (typically perplexity), leading to ratio selection based on minimizing language modeling loss. The paper demonstrates (Figure 3) that language modeling scores vary by less than one percentage point across ratios from 24:1 to 3:1—the curve is essentially flat—while recall as measured by RULER nearly doubles over the same range. This is not a quantitative nuance; it means that if you tune your hybrid ratio on perplexity, you will observe no meaningful signal and likely settle on an arbitrary ratio, potentially leaving enormous recall performance on the table. The paper reframes the ratio as primarily a recall knob, and recommends that hybrid models intended for recall-intensive applications should be tuned explicitly on recall benchmarks rather than language modeling loss. This insight is directly actionable for any team building long-context language models.

The paper also resolves a latent tension in prior work. Multiple papers had noted that purely linear/recurrent models underperform Transformers on recall tasks (Jelassi et al., 2024; Sieber et al., 2024), but the hybrid literature (Jamba, Mamba-2-Hybrid, StripedHyena) demonstrated that adding a small number of full-attention layers could close much of this gap. What was missing was a systematic characterization of how many full-attention layers are needed, whether this number depends on the choice of linear backbone, and why some backbones benefit more from hybridization than others. This paper provides that characterization: recall saturates around a 3:1 linear-to-full ratio (Figure 3, right panel), this saturation point is largely architecture-independent for models capable of achieving meaningful recall, and only architectures possessing input-dependent gating and controlled forgetting mechanisms ever reach the Transformer baseline regardless of how much full attention is added (Table 6). The resolution is that hybrids can match Transformer recall, but success requires both a suitable linear backbone and sufficient full-attention allocation—neither alone suffices.

Several research directions become more attractive as a result of this work. The finding that hierarchical recurrence (HGRN-2) complements full-attention layers more effectively than unstructured per-channel gating (GLA) or delta-rule targeting (GatedDeltaNet) suggests that architecture search over linear backbone designs should optimize for complementary interaction with full attention, not for standalone capability. The STAR framework (Thomas et al., 2025) and mechanistic design approach (Poli et al., 2024) become natural vehicles for this search. The finding that recall saturates around 3:1 in block-wise hybrids suggests that alternative hybridization strategies—head-wise mixing (Hymba), prefill-decode hybrids (YOCO), or dynamic routing—deserve investigation as potential routes to pushing the compression factor higher while maintaining recall. Conversely, directions that focus on improving standalone linear model quality without evaluating hybrid interaction become less attractive: the paper shows that standalone gains can be irrelevant or even inversely correlated with hybrid performance, meaning that a linear attention paper that reports only pure-model benchmarks provides incomplete guidance for practitioners building hybrid systems.

Perhaps most importantly, the paper provides a vocabulary for discussing hybrid architecture design. The triad of properties—selective gating, hierarchical recurrence, and controlled forgetting—moves the conversation beyond "Model X outperforms Model Y" toward "Model X possesses properties A, B, and C, which we hypothesize are necessary for effective hybridization." This enables hypothesis-driven architecture design: a researcher can ask whether a proposed new linear attention mechanism possesses these three properties and, if not, what modifications would add them. The paper does not prove causality for all three properties (the hierarchy claim is observational, and the paper acknowledges this in Section 4.3), but it provides a testable framework that subsequent work can validate or refine through controlled ablations.

Follow-Up Research This Work Enables

Controlled ablation of hierarchical recurrence in hybrid backbones. The paper's claim that hierarchical recurrence is a necessary property for hybrid effectiveness rests on comparing HGRN-2 (two-pathway hierarchical) against GLA (unstructured per-channel gating), which differ in multiple ways beyond the presence or absence of hierarchy. A clean experiment would construct a version of HGRN-2 with the hierarchical pathway removed—collapsing the two-pathway architecture into a single-path variant while preserving the tied-gate mechanism, state size, and all other architectural details—and compare hybrid performance at the same ratios. If the single-path variant's recall degrades to GLA-like levels (below the Transformer baseline), this would establish hierarchy as causally necessary. If performance is unchanged, the HGRN-2 advantage over GLA would be attributable to the tied-gate structure or other confounded factors. This ablation is directly enabled by the paper's characterization of the recall saturation point and its identification of HGRN-2 as the strongest hybrid backbone.

Larger-scale verification of the architectural property triad. The paper's findings are bounded by the 1.3B parameter scale. A natural and important follow-up would replicate the core comparison—HGRN-2 versus GatedDeltaNet versus GLA, at the same linear-to-full ratios—at 7B parameters with 200B+ training tokens, evaluating on RULER with longer context windows (32k–128k tokens). The specific questions this would answer: Does the ranking of HGRN-2 over GatedDeltaNet in hybrid form persist at scale, or does GatedDeltaNet's delta-rule mechanism become more advantageous as sequence length grows? Does the recall saturation point shift—perhaps requiring a 2:1 or even 1:1 ratio at 128k contexts? Does RetNet remain at near-zero recall at 7B parameters, or does increased capacity eventually compensate for its lack of input-dependent gating? The paper's open-sourced models at 340M and 1.3B provide direct baselines for extrapolation; training three architectures at 7B with a subset of ratios (3:1, 6:1, 12:1) would cost roughly 20× the 1.3B training budget—substantial but feasible for an academic lab or industry group.

Instruction-tuning and downstream task evaluation of hybrid models. All models in this study are evaluated zero-shot on pretraining-only checkpoints. Instruction tuning changes how models utilize their context—it often teaches them to attend more precisely to specific instructions, which could alter the effective recall demands on the architecture. A follow-up would take the strongest hybrid configurations identified in this paper (HGRN-2 at 6:1 and GatedDeltaNet at 3:1, at 1.3B parameters), apply a standard instruction-tuning protocol (e.g., on a mix of FLAN, OpenOrca, and synthetic long-context data), and evaluate on downstream tasks requiring long-range recall: long-document QA (NarrativeQA, Qasper), multi-turn conversation with distant references, and retrieval-augmented generation benchmarks. The specific question is whether the pretraining recall ranking persists after instruction tuning, or whether the fine-tuning process teaches different architectures to utilize their full-attention layers differently enough to reshuffle the ranking. The paper's open-sourcing of all 72 models makes this directly feasible—an external team could perform the instruction tuning without retraining from scratch.

Head-wise versus block-wise hybridization at matched recall. The paper studies only block-wise interleaving (r linear layers, then one full-attention layer, repeated). Hymba (Dong et al., 2024) demonstrated that head-wise mixing—allocating some attention heads within a single layer to softmax and the rest to state-space updates—can halve KV-cache size while preserving accuracy. A direct comparison between block-wise HGRN-2 at 3:1 and a head-wise HGRN-2 variant at equivalent or higher compression would determine whether head-wise mixing can achieve Transformer-level recall at ratios exceeding the 3:1 saturation point observed in block-wise hybrids. The experiment would train a head-wise model where, within each layer, 25% of heads use softmax attention and 75% use HGRN-2 linear attention, at a total parameter count matching the 1.3B block-wise models. If head-wise mixing achieves comparable RULER scores at higher KV-cache compression ratios, it would establish a new Pareto frontier and suggest that the 3:1 block-wise saturation is not fundamental but an artifact of the interleaving granularity.

Cheap recall-predictive metrics for hybrid architecture selection. The paper demonstrates that standalone LM benchmarks do not predict hybrid recall performance, but evaluating every candidate architecture at scale is prohibitively expensive. A methodological follow-up would develop a lightweight proxy metric—perhaps a synthetic recall task (modeling the "needle-in-a-haystack" retrieval directly during training), or a diagnostic based on the linear layers' state statistics (e.g., the effective memory capacity as measured by key recovery accuracy from the hidden state)—that correlates with full-scale RULER performance but can be measured at small scale or early in training. The experiment would train 20–30 diverse linear attention variants at 100M parameters, measure both the proposed proxy and (for a subset) full hybrid RULER at 1.3B, and report the correlation. If a strong proxy exists, it would enable rapid iteration on linear backbone design without the full cost of the 72-model matrix used in this paper. The paper's dataset—72 models with measured LM and recall metrics—provides ground truth for validating such a proxy.

Dynamic ratio scheduling during autoregressive generation. The paper's hybridization uses a fixed, static ratio throughout the network. But during autoregressive generation, the recall demands vary: generating the first token of a long-form answer may require retrieving information from the prompt (high recall load), while generating subsequent tokens may rely primarily on local coherence (low recall load). A follow-up would explore dynamic ratio scheduling: using all full-attention layers during the prompt encoding phase, then selectively disabling some full-attention layers during token generation, effectively varying the effective ratio over the course of a single sequence. This could be implemented by a learned gating mechanism that decides per-layer whether to route through the full-attention path or skip it (using only the linear layer's output), trained with a regularization penalty on the fraction of full-attention layers used. The specific metric to optimize would be RULER score at a target average KV-cache size—if dynamic scheduling can achieve Transformer-level recall while using full attention for only 15% of generation steps (versus the 25% implied by a 3:1 static ratio), it would push the compression frontier beyond what static block-wise hybrids can achieve.

Practical Applications and Downstream Use Cases

Memory-constrained long-context deployment on consumer hardware. The paper's deployment recipe—HGRN-2 or GatedDeltaNet at a 3:1 to 6:1 linear-to-full ratio—provides specific guidance for teams building LLM inference systems where GPU memory is the primary constraint. At the 1.3B parameter scale, a pure Transformer with 2,048-token context requires a KV cache of approximately (number of layers × sequence length × model dimension × 2 bytes per float16) for keys and values. Moving to a 3:1 hybrid ratio cuts the number of full-attention layers to one-quarter of the total layers, reducing KV-cache memory by roughly 4×. For a 7B-parameter model at 32k-token context—a realistic deployment target—this translates to saving gigabytes of GPU memory, potentially enabling long-context inference on a single consumer GPU (e.g., RTX 4090 with 24GB) that would otherwise require an A100 or model offloading. The paper's finding that language modeling quality is flat across ratios (Figure 3, left) means that for applications where recall is not the primary bottleneck—document summarization, creative writing, simple question answering—practitioners can push the ratio even higher (12:1 or 24:1) for maximum memory savings with negligible quality degradation, reserving attention-heavy configurations only for recall-critical tasks.

Cost-efficient inference for retrieval-augmented generation (RAG) pipelines. RAG systems combine a retriever (which fetches relevant documents) with a language model (which reads the documents and generates an answer). The language model in a RAG pipeline often processes long retrieved contexts that require precise recall of specific facts—exactly the capability where the paper shows hybrid models with sufficient full-attention layers match Transformers while reducing KV-cache costs by 4–7×. A RAG deployment processing millions of queries per day could adopt HGRN-2 at 6:1 as the generator backbone, achieving comparable answer accuracy to a pure Transformer (as measured by retrieval-specific recall benchmarks like RULER's multi-key and QA subtasks) while reducing per-query GPU memory by roughly 7×. This enables either higher batch sizes on existing hardware (improving throughput) or deployment on cheaper GPU instances (reducing cost per query). The paper's Figure 4 subtask breakdown—showing that single-key and multi-key retrieval subtasks benefit most from additional full attention—maps directly onto RAG workloads where the model must retrieve specific facts from retrieved documents.

Training data generation for self-improvement with long-context reasoning. When using LLMs to generate training data for long-context reasoning tasks (e.g., synthesizing multi-hop QA pairs from long documents, generating code with inter-file dependencies), the generator model must maintain accurate recall over long inputs to produce valid training examples. Using a hybrid model with optimized recall—HGRN-2 at 3:1 or 6:1—as the data generator provides near-Transformer recall quality at a fraction of the inference memory cost, enabling longer context windows or higher-throughput generation within the same hardware budget. The synthetic data can then be used to fine-tune smaller, purely-linear models (which the paper shows have negligible recall but strong language modeling) or to distill a larger Transformer, creating a cost-efficient self-improvement loop. The paper's finding that pure linear models (especially HGRN) are orders of magnitude cheaper in FLOPs than hybrids (Figure 5) but have near-zero recall suggests a division of labor: use the expensive hybrid for data generation where recall matters, and deploy the cheap pure-linear model for token generation where local coherence dominates.

</response>