Hungry Hungry Hippos: Towards Language Modeling with State Space Models

ArXiv: 2212.14052

🎯 Pitch

State space models were thought to lag behind attention for language modeling—until H3 revealed the gap was due to only two missing mechanisms, not a fundamental limit. By designing SSMs that can recall tokens and compare them across a sequence, a hybrid H3-attention model actually outperforms pure Transformers by 1.0 perplexity, while FLASHCONV makes it 2.4× faster at text generation.

---

1. Executive Summary

This paper analyzes the expressivity and efficiency gap between state space models (SSMs) and attention in language modeling, then introduces H3 (Hungry Hungry Hippo), a new SSM layer that stacks two discrete SSMs with multiplicative interactions to capture two missing capabilities—remembering tokens after specific events (via a shift SSM that maintains a memory buffer) and comparing tokens across the sequence (via multiplicative gating between projections). On synthetic language tasks, H3 matches attention perfectly—100% accuracy on induction heads and 99.8% on associative recall where prior SSMs (S4D, Gated State Spaces) largely fail—while on natural language, a hybrid H3-attention model with just two attention layers outperforms Transformers by 1.0 perplexity on OpenWebText and by up to 1.0 PPL across 125M–2.7B parameter scales on the Pile. The paper further proposes FLASHCONV, a fused block-FFT algorithm with a novel state-passing scheme that exploits the recurrent properties of SSMs to achieve 2× speedup on the Long Range Arena benchmark and 2.4× faster text generation than Transformers, establishing that SSMs can match or exceed attention on language modeling—but only when explicitly designed with mechanisms for token recall and comparison that attention provides natively.

2. Context and Motivation

The Core Problem: A Growing Chasm Between Capability and Efficiency

The paper addresses a fundamental tension at the heart of modern language modeling. Transformers, built on the self-attention mechanism, dominate virtually every benchmark in natural language processing. Their success is not accidental—attention provides two critical capabilities that make it exceptionally well-suited to language: the ability to recall specific tokens from arbitrary positions in the input, and the ability to compare any pair of tokens through the quadratic attention matrix $\mathbf{QK}^\top$. These capabilities underpin everything from simple copying behavior to the sophisticated in-context learning that allows language models to adapt to new tasks without weight updates.

However, this power comes at a steep cost. Standard self-attention scales as $O(N^2)$ in both time and memory with respect to sequence length $N$—a cost that becomes prohibitive for the long sequences increasingly demanded by real-world applications: processing entire books, analyzing hours-long audio recordings, or modeling continuous physiological signals like EEG. The research community has invested enormous effort in developing efficient attention variants (e.g., Reformer, Performer, linear attention), specialized hardware (tensor cores, transformer-specific chips), and memory-optimized implementations (FlashAttention). Yet the quadratic scaling is a theoretical lower bound for exact attention—you cannot fundamentally escape it within the attention paradigm.

State space models (SSMs) offer a tantalizing alternative. These models trace their lineage to classical control theory and signal processing, where dynamical systems described by differential equations map input signals to outputs through latent state variables. Unlike attention, SSMs can be computed as a convolution with a learned filter, enabling $O(N \log N)$ scaling through FFT-based algorithms. During inference, they admit a recurrent formulation where each new token requires only constant-time computation—independent of sequence history. This makes SSMs dramatically more efficient than Transformers for long sequences in principle.

The Performance Gap: SSMs Lag Behind in Language

Despite their theoretical elegance and strong results in continuous-signal domains like audio generation and time series forecasting, SSMs have consistently underperformed attention on language modeling by multiple perplexity points. The paper cites this gap explicitly: existing SSMs—including carefully designed variants like S4 and its derivatives—trail Transformers by 3–4 PPL on benchmarks like OpenWebText (Table 3). For a field where single-point perplexity improvements often drive architectural decisions, this is a substantial deficit.

This gap is puzzling. Language, like audio, is a sequential modality. SSMs' ability to model long-range dependencies through continuous-time hidden states—formalized through the HiPPO framework that projects sequence history onto orthogonal polynomial bases—should in theory capture the temporal structure of text. Yet in practice, something about how language works appears to require capabilities that standard SSMs lack.

The paper identifies two specific missing capabilities through synthetic diagnostic tasks:

1. Token Recall After Events: The induction head task (Table 1) tests whether a model can remember a token that appeared immediately after a special marker (e.g., $\vdash$) earlier in the sequence. When the model encounters this marker again at the end, it must output that remembered token. This requires detecting a specific event (the marker), then faithfully copying a subsequent token for later retrieval. Existing SSMs achieve only 35.6% accuracy (S4D) or 6.8% (Gated State Spaces) on this task—barely above random for the latter.

2. Token Comparison Across the Sequence: The associative recall task (Table 1) presents key-value pairs scattered through the input, then asks for the value associated with a specific key queried at the end. This requires comparing the query key against all previously seen keys to identify the relevant pair, then recalling its associated value. While S4D reaches 86.0%, this is still well below perfect accuracy, and GSS achieves only 78.0%.

These failures are not edge cases. Olsson et al. (2022) demonstrated that induction head-like mechanisms account for the majority of in-context learning capability in Transformers—the ability that enables few-shot prompting, instruction following, and other emergent behaviors that make large language models useful. If SSMs cannot perform these operations, they are fundamentally limited in their ability to capture the structure of language.

Why This Gap Matters Beyond Academic Interest

The SSM-attention gap has real consequences. Several research directions and applications depend on being able to process very long sequences efficiently:

• Long-context language understanding: Legal documents, scientific papers, and code repositories often span tens or hundreds of thousands of tokens. Quadratic attention becomes infeasible at these scales, forcing practitioners to truncate context—potentially losing critical information referenced early in the document.

• Multimodal foundation models: Continous signals like audio, video, and physiological recordings (EEG, fMRI) naturally produce sequences of 10,000–100,000+ time steps. Transformers struggle with these lengths; SSMs handle them easily—but only if they can match the representational power of attention across modalities.

• Efficient on-device deployment: The recurrent formulation of SSMs enables constant-time-per-token generation, making them ideal for mobile and edge devices where memory and compute are constrained. But if SSMs produce meaningfully worse language, this efficiency advantage is moot for text-based applications.

• Self-improvement and iterative refinement: Systems that generate, evaluate, and revise their own outputs need both the ability to attend to specific past outputs (recall) and to compare alternative generations (comparison). SSMs that lack these capabilities may be fundamentally unsuitable for such pipelines.

The paper's framing suggests that closing this gap could enable a unified architecture that handles both discrete language and continuous signals—potentially serving as the backbone for truly multimodal foundation models.

Prior Approaches and Their Shortcomings

The paper situates itself within a rich landscape of work attempting to either improve SSMs for language or reduce the cost of attention.

SSM variants for language. The dominant prior work is S4 (Gu et al., 2022) and its parameterization improvements—S4D (Gu et al., 2022), which constrains the state matrix $\mathbf{A}$ to be diagonal, and S4 variants with diagonal-plus-low-rank structure. These models achieve strong results on the Long Range Arena benchmark, which tests long-range dependency modeling across synthetic and simplified tasks. However, LRA success has not translated to competitive language modeling perplexity: the gap remains multiple points.

Gated State Spaces (GSS; Mehta et al., 2022) specifically target language modeling by adding gating mechanisms reminiscent of LSTMs and GRUs to SSMs. The paper evaluates GSS and finds it performs worse on synthetic tasks than even S4D (Table 2), and trails Transformers by 3.4 PPL on language modeling (Table 3). The gating mechanism, while intuitively useful for controlling information flow, does not address the fundamental inability to perform token recall and comparison that the synthetic tasks reveal.

A critical limitation of all prior SSMs is that none were explicitly evaluated on mechanistic language tasks like induction heads or associative recall. The community had observed the perplexity gap but lacked a diagnostic framework to understand why it exists. By introducing these synthetic tasks as probes, the paper reframes the problem from "SSMs are worse at language" to "SSMs lack specific computational primitives that attention provides natively"—a more actionable diagnosis.

Efficient attention approximations. A parallel line of work attempts to reduce attention's quadratic cost while preserving its representational power. Linear attention (Katharopoulos et al., 2020) replaces the softmax similarity with a kernelized dot product $\phi(Q)^\top \phi(K)$, enabling a recurrent formulation with $O(N)$ complexity. The Performer (Choromanski et al., 2021) approximates the softmax using random feature projections. Reformer (Kitaev et al., 2020) uses locality-sensitive hashing to reduce the set of key-query pairs considered. These approaches show promise but generally underperform exact attention on language modeling perplexity, and can be slower in wall-clock time than optimized exact attention implementations like FlashAttention (Dao et al., 2022).

The paper includes linear attention baselines on WikiText-103 (Table 10) and PG-19 (Table 11), where it achieves 25.6 and 19.1 PPL respectively—substantially worse than both Transformers (18.6, 17.0) and H3 hybrids (18.5, 16.2). This suggests that the kernelized formulation, while computationally appealing, loses important inductive biases present in exact softmax attention. The paper's insight is that rather than approximating attention, one might instead augment SSMs with the specific missing primitives that make attention effective.

Hybrid architectures. The idea of combining SSMs with attention is not new: GSS (Mehta et al., 2022) also explores hybrid models where some layers use gated state spaces and others use attention. The paper validates this direction (Table 3) but argues that the specific H3 formulation—with its shift SSM for memory and multiplicative interactions for comparison—provides a stronger SSM building block that makes hybrids more effective. The hybrid H3-attention model (2 attention layers out of 12) outperforms the GSS-attention hybrid by 0.2 PPL despite having the same number of attention layers, suggesting H3's design choices matter beyond simply "add attention to SSMs."

How This Paper Positions Itself

The paper makes a deliberate methodological choice: use synthetic diagnostic tasks to identify specific computational primitives, then design an architecture that implements those primitives, then validate on real language. This is a "mechanistic interpretability" approach applied to architecture design, inspired by work from the Anthropic interpretability team (Elhage et al., 2021; Olsson et al., 2022) that reverse-engineered how Transformers implement induction heads.

The paper's position is that the SSM-attention gap is not inherent to the SSM formalism—it is a consequence of specific architectural choices (or omissions) in existing SSM designs. By augmenting SSMs with a shift matrix (creating explicit memory of recent tokens) and multiplicative interactions (enabling token comparison), H3 recovers the missing primitives while retaining SSMs' $O(N \log N)$ complexity and recurrent inference. The fact that H3 matches attention perfectly on synthetics (Table 2) and a hybrid H3-attention model surpasses Transformers on language (Table 3) is evidence for this position.

The paper also explicitly positions SSMs as facing a hardware barrier independent of their modeling capabilities. Despite asymptotically better scaling, SSMs run slower than attention on modern GPUs for typical sequence lengths because their FFT-based convolution cannot utilize specialized matrix multiplication hardware (tensor cores) and suffers from memory bandwidth bottlenecks. The FLASHCONV contribution addresses this directly—it is not an incremental speedup but a targeted intervention to remove the IO and compute bottlenecks that make SSMs practically slower than their theoretical advantage suggests. By achieving 2× speedup on LRA and 2.4× faster generation than Transformers, the paper demonstrates that the hardware barrier is surmountable with careful algorithm-hardware co-design.

In summary, the paper frames itself as bridging two gaps simultaneously: an expressivity gap (what SSMs can compute vs. what attention can compute) and an efficiency gap (what SSMs theoretically cost vs. what they practically cost on modern hardware). The synthetic task methodology provides diagnostic clarity for the first gap, while the fused block-FFT and state-passing algorithms constitute engineering contributions that address the second. The dual focus—modeling and systems—reflects the paper's conviction that SSMs will only become competitive with Transformers when both gaps are closed.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

This is primarily a systems-and-modeling paper that designs a new neural network layer—H3 (Hungry Hungry Hippo)—for sequence modeling, along with a specialized algorithm (FLASHCONV) to make it run fast on modern GPUs. The core idea is that state space models (SSMs) underperform attention on language modeling because they lack two specific computational primitives—recalling tokens after events and comparing tokens across the sequence—and that adding these primitives via a shift-SSM and multiplicative interactions bridges the expressivity gap, while a fused block-FFT with state-passing bridges the hardware efficiency gap.

3.2 Big-picture architecture (diagram in words)

The system has three major components:

1. H3 Layer — a drop-in replacement for self-attention in a Transformer-style architecture. It takes a sequence of vectors as input and produces a transformed sequence of the same shape as output, using two SSMs with different matrix structures (shift and diagonal) connected by multiplicative interactions. Two H3 layers stacked with an MLP between them form one "H3 block," analogous to an attention block.

2. FLASHCONV — a fused GPU kernel for computing the FFT-based convolution that underlies SSM training. It combines kernel fusion (eliminating intermediate memory reads/writes), block FFT (using matrix multiplication hardware for the Fourier transform), and a state-passing algorithm (splitting long sequences into chunks processed with a recurrent state handoff) to achieve near-linear wall-clock scaling.

3. Hybrid H3-Attention Model — a language model where most layers use H3, but two specific layers (positions 2 and $N/2+2$ out of $N$ total layers) use standard self-attention. This architecture is the one that achieves the best language modeling results, outperforming pure Transformers.

Information flows through an H3 layer as follows: the input sequence $u \in \mathbb{R}^{N \times d}$ (where $N$ is sequence length and $d$ is hidden dimension) is linearly projected into three matrices $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$. The $\mathbf{K}$ projection passes through a shift SSM that delays each token by one time step and maintains a memory buffer of recent inputs. The output of this shift SSM is combined with $\mathbf{V}$ through an elementwise product, then passed through a diagonal SSM that acts as a cumulative sum over the entire sequence. The resulting tensor is multiplied by $\mathbf{Q}$ at each time step. The overall architecture stacks these H3 layers alternating with standard MLPs, connected by residual connections and layer normalization (pre-norm style).

3.3 Roadmap for the deep dive

• First, the two synthetic language tasks (induction heads and associative recall) that diagnose why existing SSMs fail—since understanding the gap is necessary to understand the solution.
• Second, the shift SSM and diagonal SSM components as atomic building blocks—because H3's entire architecture is built by composing these two specialized SSMs with distinct memory properties.
• Third, the multiplicative interaction mechanism and the full H3 layer algorithm—since the multiplicative interactions are what connect the two SSMs and enable token comparison, the second missing primitive.
• Fourth, the formal connection between H3 and linear attention/time-varying systems—since this reveals why H3 works and how it relates to prior approaches.
• Fifth, the FLASHCONV system: kernel fusion, block FFT, and state-passing—since efficient training at scale depends on these algorithms, not just the layer design.
• Sixth, the hybrid architecture and model scaling choices—since the final language models use a specific mixture of H3 and attention layers with particular hyperparameter configurations.

3.4 Detailed, sentence-based technical breakdown

Synthetic Language Tasks as Diagnostic Probes

The paper uses two synthetic tasks to identify exactly what primitives existing SSMs lack. These are not natural language benchmarks but controlled experiments that isolate specific computational abilities.

Induction Head Task. The task (Table 1) presents a sequence containing a special marker token (denoted $\vdash$), followed by one token, followed by additional unrelated tokens, and then the marker again at the end. The model must predict the token that originally followed the marker. For example, given the sequence $[a, b, c, d, e, \vdash, f, g, h, i, \dots, x, y, z, \vdash]$, the correct next-token prediction is $f$. The sequence length is fixed at 30 and the vocabulary size is 20.

What makes this task revealing: it requires two operations in sequence. First, detect an event—recognize that $\vdash$ has appeared. Second, recall a token—remember the token that appeared immediately after $\vdash$ and reproduce it much later when $\vdash$ appears again. Attention solves this trivially: the query for the final $\vdash$ position can attend to the earlier $\vdash$ position (event detection through similarity), then the value at the position one step after that $\vdash$ is directly copied to the output (token recall through the weighted sum over values). SSMs lack the explicit comparison mechanism that lets them find "the position after $\vdash$" and the explicit copy mechanism that lets them retrieve the token at that position.

Associative Recall Task. The task (Table 1) presents key-value pairs scattered through the input: the sequence alternates between keys and values (e.g., $[a, 2, c, 4, b, 3, d, 1]$), ending with a key (e.g., $a$). The model must predict the value associated with that key that appeared earlier in the sequence (e.g., for key $a$, output $2$). The sequence length is 20 and vocabulary size is 10.

This task requires two subtler operations. First, compare tokens: the model must check whether the current key matches each previously seen key. Second, recall the associated value: once a match is found, retrieve the token that followed that key. Attention implements this through the quadratic $\mathbf{QK}^\top$ matrix—each query key is compared against all previous keys, producing high attention scores at matching positions, which then gate the values at those positions. Existing SSMs achieve only 86.0% (S4D) or 78.0% (Gated State Spaces) on this task with two layers (Table 2), compared to 100% for attention and 99.8% for H3.

Why These Tasks Matter for Language. The paper cites Olsson et al. (2022), who showed that induction head mechanisms account for the majority of in-context learning capability in Transformers. In-context learning—the ability to learn a new task from examples provided in the prompt without weight updates—is arguably the most important emergent capability of large language models. If an architecture cannot implement induction heads, it fundamentally cannot support one of the key behaviors that make LLMs useful. The synthetic tasks are thus not arbitrary benchmarks; they are minimal testbeds for the computational primitives that underpin practical language capability.

Baseline Results. Table 2 shows the diagnostic results for 2-layer models. S4D achieves only 35.6% on induction heads and 86.0% on associative recall. Gated State Spaces (GSS) performs even worse: 6.8% on induction heads (barely above the 5.0% random baseline) and 78.0% on associative recall. Both models solve neither task. Attention achieves 100% on both. H3 achieves 100% on induction heads and 99.8% on associative recall—essentially closing the gap completely on the synthetic tasks.

Shift SSM: A Memory Buffer for Recent Tokens

The shift SSM is the first of two specialized SSM components in H3, designed to provide the "recall tokens after events" capability. An SSM is defined by four matrices: $\mathbf{A} \in \mathbb{R}^{m \times m}$ (state transition), $\mathbf{B} \in \mathbb{R}^{m \times 1}$ (input projection), $\mathbf{C} \in \mathbb{R}^{1 \times m}$ (output projection), and $\mathbf{D} \in \mathbb{R}^{1 \times 1}$ (feedthrough). The state evolves as $x_i = \mathbf{A}x_{i-1} + \mathbf{B}u_i$ and the output is $y_i = \mathbf{C}x_i + \mathbf{D}u_i$. The key design choice in the shift SSM is the structure of $\mathbf{A}$.

Shift Matrix Structure. The shift SSM constrains $\mathbf{A}$ to be a shift matrix of size $m \times m$, defined as:

$\mathbf{A}_{i,j} = \begin{cases} 1 & \text{for } i - 1 = j \\ 0 & \text{otherwise} \end{cases}$

where $m$ is the hidden state dimension (the SSM state size, set to $m = 64$ in the language model experiments) and $i, j$ index the rows and columns of $\mathbf{A}$ (1-indexed in the original description; equivalently, $\mathbf{A}$ has ones on the sub-diagonal and zeros elsewhere).

What It Computes. When $\mathbf{A}$ is a shift matrix, applying $\mathbf{A}$ to the state $x_i$ shifts every element of the state vector down by one position, discarding the last element and inserting a zero at the top. If we also set $\mathbf{B} = e_1$ (the first standard basis vector, i.e., a vector with 1 in the first position and 0 elsewhere), then the state update at each timestep $i$ is:

$x_i = \mathbf{A}x_{i-1} + e_1 u_i$

Starting from a zero initial state, after $t$ timesteps the state becomes:

$x_t = [u_t, u_{t-1}, u_{t-2}, \dots, u_{t-m+1}]$

That is, the state is a sliding window of the $m$ most recent input values, with the current input in the first position, the previous input in the second position, and so on. The output is $y_i = \mathbf{C}x_i + \mathbf{D}u_i$, where $\mathbf{C}$ and $\mathbf{D}$ are learned.

Why a Shift Matrix. The shift matrix provides a finite, explicit memory buffer of the exact inputs from the last $m$ steps—not a compressed or lossy representation, but the literal input values themselves. This is exactly what is needed for token recall: when a special event occurs (e.g., the marker $\vdash$ appears at position $i$), the very next input $u_{i+1}$ enters the state buffer. The shift matrix ensures this value persists in the state at a known position (shifting down by one each subsequent step) until it is needed. In contrast, the standard HiPPO-initialized $\mathbf{A}$ matrices used in S4/D project the input history onto orthogonal polynomial bases—a compressed representation that is excellent for capturing long-range smooth dependencies but cannot preserve the exact value of a single token for later verbatim retrieval.

Connection to Convolution. The shift SSM with $\mathbf{B} = e_1$ is equivalent to a 1D convolution with kernel size $m$ along the sequence dimension. The filter $f = [\mathbf{CB}, \mathbf{CAB}, \mathbf{CA}^2\mathbf{B}, \dots, \mathbf{CA}^{m-1}\mathbf{B}]$ has $m$ non-zero entries (since powers of the shift matrix beyond $m-1$ become zero matrices). This means the shift SSM has strictly finite memory—it can only look back $m$ steps—but within that window, it captures exact input values in order. The paper notes that $\mathbf{B}$ can be learned (giving the model flexibility over which inputs enter the buffer and in what linear combination) or fixed to $e_1$ (simplifying to a standard 1D convolution with kernel size $m$).

In H3 Specifically. The shift SSM in H3 processes the $\mathbf{K}$ projection of the input. The output $\bar{\mathbf{K}} = \text{SSM}_{\text{shift}}(\mathbf{K})$ therefore contains, at each position $i$, information about the $\mathbf{K}$ vectors from the previous $m$ time steps—specifically, the $m$ most recent $\mathbf{K}$ vectors in reverse chronological order (newest first), transformed by the learned $\mathbf{C}$ matrix.

Diagonal SSM: Cumulative Memory Over the Entire Sequence

The diagonal SSM provides the second memory mechanism: persistent, lossy memory over the entire sequence, complementing the shift SSM's exact but limited-duration memory. The key design choice is in the initialization and structure of $\mathbf{A}$.

Diagonal Structure. The diagonal SSM constrains $\mathbf{A}$ to be a diagonal matrix—all off-diagonal entries are zero, only the diagonal $\mathbf{A}_{i,i}$ entries are non-zero and learned. This makes the state evolution completely decoupled: each of the $m$ dimensions of the hidden state evolves independently according to $x_i^{(j)} = a_j x_{i-1}^{(j)} + b_j u_i$, where $a_j$ is the $j$-th diagonal entry of $\mathbf{A}$ and $b_j$ is the $j$-th entry of $\mathbf{B}$. This decoupling makes training and computation efficient.

HiPPO Initialization. The diagonal entries are initialized from the diagonal version of HiPPO (S4D; Gu et al., 2022). The HiPPO framework projects the sequence history onto a basis of orthogonal polynomials (specifically, Legendre polynomials for a uniform weighting of history). This initialization encodes an inductive bias that the state should compress the input history in a way that approximately preserves information for linear function approximation over time. Concretely, the diagonal entries produced by S4D are complex numbers with negative real parts, meaning the state evolution has a decaying memory: older inputs are gradually forgotten, but the rate of decay is spread across different timescales (some dimensions forget quickly, others retain information for very long). This is why the diagonal SSM can maintain information over the entire sequence, unlike the shift SSM's hard cutoff at $m$ steps.

What It Computes. Given the sequence of inputs, the diagonal SSM computes the convolution of the input with a learned filter $f = [\mathbf{CB}, \mathbf{CAB}, \mathbf{CA}^2\mathbf{B}, \dots, \mathbf{CA}^{N-1}\mathbf{B}]$, where each term $\mathbf{CA}^k\mathbf{B}$ can be exponentially decaying (if the diagonal entries of $\mathbf{A}$ have magnitude less than 1) or oscillatory (if they are complex). The output at position $i$ is a weighted combination of all previous inputs, with the weights determined by the learned dynamics. In H3 specifically, the diagonal SSM processes the tensor $\bar{\mathbf{K}}^{(h)}(\mathbf{V}^{(h)})^\top$ (explained in the next section), acting as a cumulative sum with forgetting over the entire sequence.

Why Diagonal + HiPPO Instead of General $\mathbf{A}$. Prior work (Gu et al., 2022; Gupta et al., 2022) established that diagonal SSMs with HiPPO initialization match or exceed the performance of more general SSMs (e.g., with dense or diagonal-plus-low-rank $\mathbf{A}$) while being substantially more efficient to compute. The diagonal structure reduces the per-step state update from $O(m^2)$ (general matrix-vector multiply) to $O(m)$ (elementwise multiply). The HiPPO initialization provides a principled starting point that captures long-range dependencies better than random initialization. The paper adopts this choice directly.

H3 Layer: Composing Shift and Diagonal SSMs with Multiplicative Interactions

The H3 layer combines the two SSMs with multiplicative interactions in a specific pattern inspired by linear attention. The full algorithm is laid out in Algorithm 1.

Linear Projections (Step 1 of Algorithm 1). The input $u \in \mathbb{R}^{N \times d}$ (a sequence of $N$ vectors, each of dimension $d$) is linearly projected into three matrices:

$\mathbf{Q} = u\mathbf{W}_Q, \quad \mathbf{K} = u\mathbf{W}_K, \quad \mathbf{V} = u\mathbf{W}_V$

where $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V \in \mathbb{R}^{d \times d}$ are learned weight matrices. This tripartite projection mirrors the query/key/value decomposition in attention, which is why H3 uses the same notation—though the subsequent computation is entirely different. The projections give the model flexibility to extract different features from the input for different purposes: $\mathbf{K}$ for event detection (fed to the shift SSM), $\mathbf{V}$ for value storage (combined with the shift SSM output), and $\mathbf{Q}$ for querying (combined with the diagonal SSM output).

Shift SSM on Keys (Step 2). The $\mathbf{K}$ projection passes through the shift SSM:

$\bar{\mathbf{K}} = \text{SSM}_{\text{shift}}(\mathbf{K}) \in \mathbb{R}^{N \times d}$

where $\text{SSM}_{\text{shift}}$ uses a shift matrix $\mathbf{A}$ with state size $m$. At each position $i$, $\bar{\mathbf{K}}_i$ is a learned function of the $m$ most recent $\mathbf{K}$ vectors $(K_i, K_{i-1}, \dots, K_{i-m+1})$. Because the shift SSM shifts elements down by one, $\bar{\mathbf{K}}_i$ contains the $\mathbf{K}$ vector from position $i-1$ (with a one-step delay) in its first "slot," the vector from position $i-2$ in the second slot, and so on. This is crucial for the associative recall mechanism: when a value token appears at position $i$, its associated key is at position $i-1$. By the time the value is processed, the shift SSM has the key one step back, enabling the model to condition the value's treatment on whether the preceding token was a particular key.

Head Splitting (Step 3). The matrices $\mathbf{Q}$, $\bar{\mathbf{K}}$, and $\mathbf{V}$ are split into $H$ "heads" of dimension $d_h = d/H$:

$\mathbf{Q}^{(h)}, \bar{\mathbf{K}}^{(h)}, \mathbf{V}^{(h)} \in \mathbb{R}^{N \times d_h} \quad \text{for } h = 1, \dots, H$

This multi-head decomposition is inherited from attention, where it allows the model to attend to different representational subspaces. In H3, the heads each perform independent SSM computations, giving the model multiple parallel channels for learning different event-detection and memory patterns. The paper uses head dimension $d_h = 1$ for hybrid models and $d_h = 8$ for pure H3 models, with $H$ scaling accordingly (e.g., $d_h = 1$ means $H = d$ heads of dimension 1).

Outer Product and Diagonal SSM (Step 5). Within each head $h$, the system computes the batched outer product of $\bar{\mathbf{K}}^{(h)}$ and $\mathbf{V}^{(h)}$:

$\bar{\mathbf{K}}^{(h)} (\mathbf{V}^{(h)})^\top \in \mathbb{R}^{N \times d_h \times d_h}$

For each position $i$, this produces a $d_h \times d_h$ matrix that is the outer product of the (shift-delayed) key vector at position $i$ and the value vector at position $i$. The outer product can be interpreted as storing an association: if $\bar{\mathbf{K}}^{(h)}_i$ indicates "the previous token was key $a$" and $\mathbf{V}^{(h)}_i$ is the encoding of value $2$, then their outer product records the key-value pair $(a, 2)$.

This tensor is then passed through the diagonal SSM along the sequence ($N$) dimension:

$\mathbf{KV}^{(h)} = \text{SSM}_{\text{diag}}\left(\bar{\mathbf{K}}^{(h)} (\mathbf{V}^{(h)})^\top\right) \in \mathbb{R}^{N \times d_h \times d_h}$

Because the diagonal SSM is initialized with HiPPO and acts as a cumulative sum with forgetting, $\mathbf{KV}^{(h)}_i$ at each position $i$ is a compressed representation of all the key-value outer products from positions $1$ through $i$. This is the mechanism for remembering key-value pairs over the entire sequence: each pair is deposited into the SSM state via the outer product, and the diagonal SSM's cumulative nature retains a summary of all pairs seen so far.

Query Multiplication (Step 6). The output for each head is computed by matrix-vector multiplication of $\mathbf{Q}^{(h)}_i$ with $\mathbf{KV}^{(h)}_i$ at each position:

$\mathbf{O}^{(h)} = [\mathbf{Q}^{(h)}_1 \mathbf{KV}^{(h)}_1, \mathbf{Q}^{(h)}_2 \mathbf{KV}^{(h)}_2, \dots, \mathbf{Q}^{(h)}_N \mathbf{KV}^{(h)}_N] \in \mathbb{R}^{N \times d_h}$

When the current input is a key token (e.g., query key $a$), $\mathbf{Q}^{(h)}_i$ is a vector that, when multiplied by $\mathbf{KV}^{(h)}_i$, extracts the value associated with that key from the compressed key-value store. This is the mechanism for comparing tokens across the sequence: the query vector selectively reads from the accumulated association matrix. If the model has learned to encode "key $a$" as a specific direction in the $d_h$-dimensional space, and "value for key $a$" as the corresponding row of the stored outer products, then the query multiplication performs a form of associative lookup.

Output Concatenation and Projection (Steps 7-8). The per-head outputs are concatenated and multiplied by an output projection matrix $\mathbf{W}_O \in \mathbb{R}^{d \times d}$, producing the final layer output of shape $\mathbb{R}^{N \times d}$—the same shape as the input, enabling residual connections.

The H3 Formula (Single Head, $d_h = 1$). The paper provides a concise formula for the entire computation in the scalar case ($d_h = 1$, meaning one-dimensional heads):

$\mathbf{Q} \odot \text{SSM}_{\text{diag}}(\text{SSM}_{\text{shift}}(\mathbf{K}) \odot \mathbf{V})$

where $\odot$ denotes elementwise (Hadamard) product. From inside out: $\mathbf{K}$ passes through the shift SSM (getting a delayed and buffered version of the keys), this is multiplied elementwise with $\mathbf{V}$ (producing a gated value signal—values only pass through when the preceding token was a relevant key), the result passes through the diagonal SSM (accumulating gated values over the whole sequence), and the output is multiplied elementwise with $\mathbf{Q}$ (gating the final output based on the current query). Each $\odot$ is a multiplicative interaction that enables comparison: $\text{SSM}_{\text{shift}}(\mathbf{K}) \odot \mathbf{V}$ compares the delayed key against the current value (essentially asking "does the previous token trigger this value's storage?"), and $\mathbf{Q} \odot \text{SSM}_{\text{diag}}(\cdots)$ compares the current query against the accumulated memory (essentially asking "is the current token the key for which we stored a value?").

Concrete Mechanism for Associative Recall. Appendix D.1 constructs an explicit parameterization of H3 that solves the associative recall task. For a language with 4 keys and 4 values, the construction uses embedding dimension $d = 8$, state size $m = 2$, and $H = 4$ heads (one per key). The embedding layer maps each key $k_i$ to the $e_i$ basis vector (one-hot in the first 4 dimensions) and each value $v_i$ to the $e_{4+i}$ basis vector (one-hot in dimensions 5–8). The projection matrices $\mathbf{W}_Q$ and $\mathbf{W}_K$ assign each key to a different head (e.g., $\mathbf{Q}^{(i)}_t$ is non-zero only when the input at time $t$ is key $k_i$). The shift SSM delays $\mathbf{K}$ by one step so that $\bar{\mathbf{K}}_t$ encodes whether the token at position $t-1$ was a particular key. The outer product $\bar{\mathbf{K}}^{(i)}_t (\mathbf{V}^{(i)}_t)^\top$ is non-zero only when the preceding token was key $k_i$ AND the current token is the associated value. The diagonal SSM (with $\mathbf{A}$ set to the identity matrix, meaning perfect accumulation without forgetting) sums all these non-zero deposits over the sequence. At the final position where the query key $k_i$ appears, $\mathbf{Q}^{(i)}$ is non-zero, extracting the accumulated value from the diagonal SSM output. The output projection decodes this to the correct value token. Critically, this construction shows that the shift SSM provides the one-step memory needed to detect "key followed by value" patterns, the diagonal SSM provides the accumulation needed to remember values across long gaps, and the multiplicative interactions provide the gating that ensures only the right key-value pair is retrieved.

Connection to Linear Attention. The paper explicitly connects H3 to linear attention in Appendix B. Linear attention (Katharopoulos et al., 2020) defines the output as $O_i = \frac{\phi(Q_i)^\top \sum_{j=1}^i \phi(K_j)V_j^\top}{\phi(Q_i)^\top \sum_{j=1}^i \phi(K_j)}$, where $\phi$ is a feature map (e.g., ReLU or ELU+1). If we ignore the denominator (which provides approximate normalization), linear attention is:

$O_i = \phi(Q_i)^\top \sum_{j=1}^i \phi(K_j)V_j^\top$

This is a linear time-varying system: the matrix $S_i = \sum_{j=1}^i \phi(K_j)V_j^\top$ is incrementally updated as $S_{i+1} = S_i + \phi(K_{i+1})V_{i+1}^\top$ (a state variable that is a cumulative sum), and the output is $O_i = \phi(Q_i)^\top S_i$ (a time-varying readout). H3 replaces $\phi(K_i)$ with $\text{SSM}_{\text{shift}}(K_i)$ (adding a finite memory buffer) and replaces the exact cumulative sum $S_i$ with $\text{SSM}_{\text{diag}}$ (adding learned forgetting dynamics via HiPPO). The paper describes this as "converting a linear time-varying system into a linear time-invariant system with post-processing," where the time-invariant SSM handles the memory, and the multiplicative interaction with $\mathbf{Q}$ handles the time-varying readout.

Computational Complexity. Proposition 1 proves that the H3 layer takes $O(d^2 N + dN \log N)$ time and $O(dN)$ space for sequence length $N$ and hidden dimension $d$, assuming head dimension $d_h = O(1)$. The $O(d^2 N)$ term comes from the four linear projections (three input projections and one output projection), each multiplying an $N \times d$ matrix by a $d \times d$ weight matrix. The $O(dN \log N)$ term comes from the two SSMs: the shift SSM involves one FFT convolution along the sequence dimension across $d$ channels ($O(dN \log N)$), and the diagonal SSM involves $H$ convolutions each of size $N \times d_h \times d_h$, totaling $O(H d_h^2 N \log N) = O(d d_h N \log N) = O(dN \log N)$. The space complexity is $O(dN)$ because all operations (matrix multiplies, FFTs) work on tensors of size proportional to $dN$. Compare this to standard attention, which requires $O(N^2 d)$ time and $O(N^2)$ space (or $O(N^2 d)$ for multi-head attention). For long sequences, the $O(N \log N)$ scaling of H3 is asymptotically much more efficient than attention's $O(N^2)$.

FlashConv: Fused Block-FFT with State-Passing

The FLASHCONV system addresses the practical inefficiency of SSMs on modern GPU hardware. Despite $O(N \log N)$ theoretical complexity, standard FFT-based SSM implementations run slower than attention for typical sequence lengths due to poor hardware utilization. FLASHCONV has three components: kernel fusion, block FFT, and state-passing, applied in different regimes of sequence length.

The Problem: cuFFT is Memory-Bound. The standard SSM computation uses the convolution theorem: the filter $f$ and input $u$ are transformed via FFT, multiplied pointwise, and inverse-transformed. Using NVIDIA's cuFFT library for the FFT steps, this requires:

• Reading $u$ from GPU memory into compute units
• Computing $\text{FFT}(u)$, writing result back to GPU memory
• Reading $f$ from GPU memory, computing $\text{FFT}(f)$, writing result back
• Reading both FFT results, multiplying pointwise, writing result back
• Reading the product, computing inverse FFT, writing result back

Each intermediate read/write goes to GPU global memory (HBM, high-bandwidth memory with ~1.5 TB/s on an A100), not the much faster on-chip SRAM (~19 TB/s per streaming multiprocessor). The compute cost of the FFT is small relative to the memory movement, making the operation IO-bound. This is wasteful because tensor cores on modern GPUs (providing up to 312 TFLOPS of FP16 compute on an A100) sit idle while waiting for data.

Kernel Fusion. Following the approach of FlashAttention (Dao et al., 2022), FLASHCONV fuses the entire FFT convolution into a single CUDA kernel that runs entirely in GPU SRAM for sequences up to 8K tokens. The fused kernel performs the FFT, pointwise multiply, and inverse FFT without writing any intermediate results to global memory. The input $u$ and filter $f$ are read from global memory once, all the FFT operations are computed on-chip, and only the final output $y$ is written back to global memory. This eliminates the intermediate reads and writes that dominate the cuFFT implementation. The paper reports this yields up to 3.4× speedup over cuFFT for short sequences (up to 512 tokens), where the IO bottleneck is most severe (Figure 2).

Block FFT: Using Matrix Multiplication Units. Even with kernel fusion, the FFT computation itself cannot utilize tensor cores—specialized hardware units that perform fast $16 \times 16$ matrix multiplications in FP16. Tensor cores are the primary source of compute throughput on modern GPUs, providing roughly 15× more FLOPS than standard CUDA cores for matrix operations. To leverage tensor cores for the FFT, FLASHCONV uses a classical technique: the Cooley-Tukey FFT decomposition (also known as the four-step FFT algorithm).

The key insight: an $N$-point DFT matrix $\mathbf{F}_N$ can be decomposed into block-diagonal matrices and permutations. If $N = N_1 N_2$ for integers $N_1, N_2$, the decomposition is (using the rotated DFT matrix formulation):

$\mathbf{F}_N = \mathbf{P}(\mathbf{I}_{N_2} \otimes \mathbf{F}_{N_1})\mathbf{P}^\top \mathbf{D}(\mathbf{I}_{N_1} \otimes \mathbf{F}_{N_2})\mathbf{P}$

where:

• $\mathbf{P}$ is a fixed permutation matrix that reshapes the input as an $N_1 \times N_2$ array and transposes it
• $\mathbf{D}$ is an $N \times N$ diagonal matrix containing the "twiddle factors" — complex exponentials $e^{-2\pi i k / N}$ that connect the smaller FFTs
• $\mathbf{I}_{N_2} \otimes \mathbf{F}_{N_1}$ is a block-diagonal matrix with $N_2$ copies of the $N_1 \times N_1$ DFT matrix $\mathbf{F}_{N_1}$ on the diagonal
• $\mathbf{I}_{N_1} \otimes \mathbf{F}_{N_2}$ is similarly block-diagonal with $N_1$ copies of $\mathbf{F}_{N_2}$ on the diagonal

What this decomposition enables: The block-diagonal matrices $(\mathbf{I}_{N_2} \otimes \mathbf{F}_{N_1})$ and $(\mathbf{I}_{N_1} \otimes \mathbf{F}_{N_2})$ consist of independent $N_1 \times N_1$ and $N_2 \times N_2$ matrix multiplications, which can be batched and executed on tensor cores. The permutation $\mathbf{P}$ can be implemented as a fast transpose operation. The twiddle factor multiplication $\mathbf{D}$ is a simple elementwise scaling. If $N = N_1 N_2 N_3$, the decomposition can be applied recursively, yielding a series of block-diagonal multiplications interleaved with permutations.

Computational Cost: The block FFT algorithm incurs $O(N r \log N / \log r)$ FLOPs if $N$ can be written as $r^p$ for integers $r, p$. This is slightly more FLOPs than the standard Cooley-Tukey FFT ($O(N \log N)$), because the block decomposition introduces additional permutation and twiddle factor overhead. However, the increased arithmetic is more than compensated by the use of tensor cores, which execute matrix multiplications 15× faster than standard cores execute the same arithmetic as part of a general FFT. The paper reports that block FFT yields up to 2× speedup for medium-length sequences (1K to 8K tokens), where the tensor core advantage dominates over the increased FLOP count.

State-Passing Algorithm: Scaling to Arbitrary Length. The fused block FFT kernel has a hard limit: the entire computation must fit in GPU SRAM, which on an A100 is about 40 MB per streaming multiprocessor. For sequences longer than ~8K tokens (with typical hidden dimensions), the working set exceeds SRAM capacity, and the fused kernel cannot run. To handle longer sequences, FLASHCONV introduces a state-passing algorithm that exploits the recurrent nature of SSMs.

The Key Insight. An SSM can be computed either as a convolution over the whole sequence (using FFT) or as a recurrent step-by-step computation. The state-passing algorithm combines both: process the sequence in chunks small enough to fit in SRAM using the efficient block FFT convolution, and connect the chunks using the recurrent formulation. The "state" being passed is the hidden state $x$ at the boundary between chunks.

Algorithm 2 in Detail. The state-passing algorithm takes a long input $u \in \mathbb{R}^N$ and splits it into $C = N / N'$ chunks $u^{(c)} \in \mathbb{R}^{N'}$, where $N'$ is the largest sequence length that fits in SRAM with the fused block FFT kernel. The algorithm requires computing the SSM output $y \in \mathbb{R}^N$ by processing each chunk sequentially:

Precomputation (Step 1). Before the chunk loop, the algorithm precomputes three matrices from the SSM parameters:

• $\mathbf{A}^{N'} \in \mathbb{R}^{m \times m}$ — the state transition matrix raised to the power $N'$, representing how the state evolves over a full chunk
• $\mathbf{M}_{ux} = [\mathbf{A}^{N'-1}\mathbf{B}, \mathbf{A}^{N'-2}\mathbf{B}, \dots, \mathbf{B}] \in \mathbb{R}^{m \times N'}$ — a matrix that, when multiplied by a chunk of inputs $u^{(c)}$, computes the contribution of those inputs to the final state of the chunk
• $\mathbf{M}_{xy} = [\mathbf{C}, \mathbf{CA}^\top, \dots, (\mathbf{CA}^{N'-1})^\top]^\top \in \mathbb{R}^{N' \times m}$ — a matrix that, when multiplied by an initial state, computes the contribution of that state to every output in the chunk

What each matrix means operationally. $\mathbf{M}_{ux}$ encodes "given a chunk of $N'$ input values, what is the cumulative effect on the final hidden state?" The columns of $\mathbf{M}_{ux}$ are $\mathbf{A}^{N'-1}\mathbf{B}, \mathbf{A}^{N'-2}\mathbf{B}, \dots, \mathbf{B}$. When multiplied by the input chunk $u^{(c)} = [u_1, u_2, \dots, u_{N'}]$, each column $\mathbf{A}^{N'-j}\mathbf{B}$ is scaled by $u_j$, and the sum $\sum_{j=1}^{N'} \mathbf{A}^{N'-j}\mathbf{B} u_j$ is exactly the term that would be added to the state starting from zero. $\mathbf{M}_{xy}$ encodes "given an initial state at the start of a chunk, what output does it produce at each position?" The rows of $\mathbf{M}_{xy}$ are $\mathbf{C}, \mathbf{CA}, \dots, \mathbf{CA}^{N'-1}$. When multiplied by an initial state $x_{N'}^{(c-1)}$, each row $\mathbf{CA}^{i-1}$ computes the output at position $i$ due to that initial state propagating through the dynamics.

Chunk Processing Loop (Steps 2–7). Starting with initial state $x_{N'}^{(0)} = 0 \in \mathbb{R}^m$, for each chunk $c = 1, \dots, C$:

Output Computation (Step 5). The output for chunk $c$ is:

$y^{(c)} = \mathbf{M}_{xy} x_{N'}^{(c-1)} + \text{BLOCKFFTCONV}(f, u^{(c)}) + \mathbf{D} u^{(c)}$

where $\text{BLOCKFFTCONV}(f, u^{(c)})$ is the fused block FFT convolution of the SSM filter $f = [\mathbf{CB}, \mathbf{CAB}, \dots, \mathbf{CA}^{N'-1}\mathbf{B}]$ with the chunk input $u^{(c)}$ (assuming zero initial state). The three terms have clear interpretations:

• $\mathbf{M}_{xy} x_{N'}^{(c-1)}$ accounts for the residual effect of all inputs from previous chunks, as captured by the state $x_{N'}^{(c-1)}$ handed off from chunk $c-1$
• $\text{BLOCKFFTCONV}(f, u^{(c)})$ accounts for the effect of the current chunk's inputs on the current chunk's outputs (the convolution of $f$ with $u^{(c)}$, assuming initial state zero)
• $\mathbf{D} u^{(c)}$ is the direct feedthrough term (the skip connection that passes the input directly to the output)

State Update (Step 6). After computing the chunk's output, the algorithm updates the state for the next chunk:

$x_{N'}^{(c)} = \mathbf{A}^{N'} x_{N'}^{(c-1)} + \mathbf{M}_{ux} u^{(c)}$

This is the discrete-time state update propagated for $N'$ steps. $\mathbf{A}^{N'} x_{N'}^{(c-1)}$ propagates the state from the end of the previous chunk through $N'$ steps, and $\mathbf{M}_{ux} u^{(c)}$ adds the contribution of the current chunk's inputs. Together they produce the state at the end of chunk $c$, which becomes the initial condition for chunk $c+1$.

Why this works (Proposition 2). The correctness proof in Appendix D.4 establishes by induction that the chunked computation produces exactly the same output as computing the entire SSM in one large FFT of size $N$. The induction relies on the linearity of the state space model: the output at any position is a sum of the contribution from the initial state (which is exactly the state handed off from the previous chunk) and the convolution of the filter with inputs from the current chunk. Because the initial state $x_{N'}^{(c-1)}$ for chunk $c$ is computed exactly from the previous chunks using the correct recurrence, the output matches the non-chunked computation.

Computational Benefit. The state-passing algorithm reduces the FFT size from $N$ (which may not fit in SRAM) to $N'$ (which does fit). The FFT cost per chunk is $O(N' \log N')$, and there are $N/N'$ chunks, so the total FFT cost is $O(N \log N')$. This is slightly higher than the $O(N \log N)$ cost of a single large FFT (because $\log N' < \log N$), but the practical speedup from running entirely in SRAM using fused block FFT more than compensates. The algorithm also introduces $O(m^2 + mN')$ additional work per chunk for the matrix multiplications involving $\mathbf{M}_{xy}$ and $\mathbf{M}_{ux}$, but since $m$ (the state size, typically 64) is much smaller than $N'$, this overhead is negligible. Figure 2 shows that state-passing provides up to 2.3× speedup for sequences of 16K and above compared to the next-best method.

Hardware-Specific Details. The fused kernel and block FFT are designed for NVIDIA A100 GPUs, which have the following relevant specs: 40 GB or 80 GB of HBM2e global memory (bandwidth ~1.5–2.0 TB/s), 108 streaming multiprocessors each with 192 KB of SRAM, and tensor cores providing 312 TFLOPS of FP16 matrix multiply-accumulate. The block FFT uses FP16 precision on tensor cores for the matrix multiplications, while the pointwise operations and permutations use FP32. The paper notes that training language models uses FP32 for the FFTConv to maintain numerical stability, while the MLP and attention components use bf16 (brain floating point). The state-passing algorithm's chunk size $N'$ is chosen to be the maximum that fits in SRAM given the batch size and hidden dimension—the paper uses $N'$ up to 8K on A100 for typical configurations.

Hybrid H3-Attention Architecture and Scaling

The paper's best language modeling results come from a hybrid architecture that combines H3 layers with a small number of attention layers. This section describes how the hybrid is constructed and what hyperparameters are used at different scales.

Layer Configuration. The hybrid model replaces most self-attention layers in a standard Transformer with H3 layers, keeping only two attention layers. For an $N$-layer model (where $N$ is even), the attention layers are placed at layer 2 (the second layer from the bottom) and layer $2 + N/2$ (roughly the middle of the network). Specifically:

• 12-layer model (125M): attention at layers 2 and 7
• 24-layer models (355M, 1.3B): attention at layers 2 and 13
• 32-layer model (2.7B): attention at layers 10 and 21

Why two attention layers? The paper does not extensively ablate this choice, but the motivation is that attention provides capabilities (arbitrary token-to-token comparison via the quadratic $\mathbf{QK}^\top$ matrix) that H3's multiplicative interactions can approximate but not exactly replicate. By retaining two attention layers—one early (layer 2, acting on relatively local features) and one in the middle (acting on more abstract representations)—the model gets the best of both worlds: H3's efficient long-range memory and attention's precise token-level comparison. The paper shows (Table 3) that this hybrid (PPL 19.6) outperforms both the pure H3 model (PPL 21.0) and the pure Transformer (PPL 20.6) on OpenWebText.

Per-Hyperparameter Breakdown by Model Size. The paper follows the GPT-3 scaling recipe and specifies architecture hyperparameters for each model size:

125M parameters: 12 layers, hidden dimension $d = 1024$, MLP dimension 4096, 12 attention heads (for the attention layers). For H3 layers: SSM state size $m = 64$, head dimension $d_h = 1$ (meaning $H = d/d_h = 1024$ heads of dimension 1). The hybrid uses attention at layers 1 and 7 (using 1-indexing in the paper's description; equivalently positions 2 and 7 in 0-indexed notation).

355M parameters: 24 layers, hidden dimension $d = 1024$, MLP dimension 4096, 16 attention heads. H3: $m = 64$, $d_h = 1$. Attention at layers 1 and 13 (1-indexed).

1.3B parameters: 24 layers, hidden dimension $d = 2048$, MLP dimension 8192, 16 attention heads. H3: $m = 64$, $d_h = 1$. Attention at layers 1 and 13.

2.7B parameters: 32 layers, hidden dimension $d = 2560$, MLP dimension 10240, 20 attention heads. H3: $m = 64$, $d_h = 1$. Attention at layers 10 and 21.

Across all sizes, the pure H3 model uses $d_h = 8$ heads instead of $d_h = 1$. The paper does not explain the motivation for this difference, but it likely relates to the pure H3 model needing more expressivity per head since it has no attention layers to fall back on.

Training Configuration. Language model training for OpenWebText experiments (125M models in Table 3) uses the Megatron-LM recipe: effective batch size 512, AdamW optimizer, learning rate $6 \times 10^{-4}$ for the 125M models and $1.5 \times 10^{-4}$ for 355M models, weight decay 0.1, training for 100K steps with sequence length 1024. Pile training (all model sizes in Table 4) follows the GPT-3 recipe: batch size 256 (512 for 1.3B+), sequence length 2048, AdamW, learning rate $6 \times 10^{-4}$ (125M) or $3 \times 10^{-4}$ (355M) with cosine decay to 10% by 300B tokens then constant for another 100B, 800K total steps, residual dropout 0.0, embedding dropout 0.1. All models use mixed precision: bf16 for MLPs and attention, FP32 for the FFT convolution (to maintain numerical stability in the frequency domain). The paper explicitly states it did not tune hyperparameters for H3 due to resource constraints, using Transformer-optimized settings.

Inference Speed Advantage. Because the H3 layers use SSMs with a recurrent formulation, text generation (token-by-token autoregressive decoding) can be performed by maintaining the SSM state and computing $y_{i+1}$ from $x_i$ and $u_{i+1}$ in $O(1)$ time per token, rather than attending over the entire prefix. The attention layers in the hybrid still require caching keys and values, but these constitute only 2 out of 12+ layers. Table 7 reports that the 1.3B hybrid model achieves 1980 tokens/second at prompt length 512 (vs. 1340 for a pure Transformer), 1580 tokens/s at prompt length 1024 (vs. 770), and 1240 tokens/s at 1536 (vs. 520)—a speedup factor of 1.5× to 2.4×, with the gap widening as sequence length increases because the attention layers' $O(N)$ KV-cache management becomes a larger fraction of total compute relative to the $O(1)$ SSM computation.

Pre-norm Architecture. The H3 layer is used in a pre-norm Transformer architecture, meaning layer normalization is applied before each sub-layer (H3/attention, then MLP), with residual connections wrapping around both. This follows the standard GPT-2/GPT-3 architectural convention.

Design Choices Summary

The paper makes several non-obvious design choices worth highlighting:

Shift SSM instead of a learned general $\mathbf{A}$. The shift matrix is a hard constraint—the model cannot learn to have different dynamics. This is deliberately restrictive because the inductive bias it provides (exact memory of recent tokens) is exactly what is needed for token recall. A general $\mathbf{A}$ could in principle learn to approximate a shift (if that were optimal), but in practice with gradient-based training, SSMs with general $\mathbf{A}$ struggle to preserve exact token-level information over long horizons—the HiPPO initialization encourages compression and smoothing, not verbatim copying.

Multiplicative interactions instead of additive. In H3, the outputs of SSMs are multiplied (elementwise) with projections of the input, not added. This is crucial because multiplication provides gating: the output of the shift SSM is non-zero only when the previous token matches a pattern, and this gates whether the value is passed to the diagonal SSM. Addition would mean values are always passed through, regardless of context, making associative recall impossible. The multiplicative interaction enables H3 to implement conditional computation—"if the previous token was key $a$, then store the current value"—without any learnable control flow.

Last-step aggregation for PRM is absent (not applicable). The paper does not use process reward models or scoring—this is not a reinforcement learning paper. I include this note because the reference example's PRM discussion is not relevant here. The analogous design choice would be the choice to use two SSMs with different structures rather than a single more complex SSM. A single SSM with a richer state matrix (e.g., combining shift and diagonal components in a block structure) could theoretically implement both memory mechanisms. However, separating them into distinct SSMs with different inductive biases (shift for exact short-term memory, diagonal for compressed long-term memory) makes the architecture more interpretable and easier to optimize, since each SSM specializes in one function.

Hybrid attention placement at specific layers (2 and $N/2+2$). The paper places attention layers not at the very bottom (where token-level operations dominate) nor at the very top (where high-level reasoning occurs), but at one early layer and one middle layer. This suggests—although the paper does not ablate it—that attention's token-comparison ability is most useful at intermediate levels of abstraction: too early and the representations are too local (individual tokens), too late and the representations are too compressed (semantic concepts), but in the middle, attention can compare subsequences and learned features that are neither purely syntactic nor purely semantic.

State size $m = 64$ across all scales. The shift SSM's memory buffer holds 64 tokens, which for sequence length 2048 covers only 3% of the context. This means the shift SSM provides only very local context for the gating mechanism—it can only detect patterns where the relevant key is within 64 tokens of its value. Longer-range associations rely entirely on the diagonal SSM's compressed memory. The fact that this restriction does not prevent H3 from solving associative recall on sequences of length 20 (Table 2) or achieving competitive language modeling perplexity suggests that most token-level associations in language are indeed relatively local, or that the diagonal SSM successfully captures longer-range structure despite its compressed representation.

4. Key Insights and Innovations

Innovation 1: Synthetic Diagnostic Tasks as an Architecture Design Methodology

The paper's most distinctive intellectual contribution is not H3 itself, but the methodological move of using mechanistic interpretability concepts as architecture design tools rather than merely as post-hoc explanatory devices. This reframes the relationship between understanding and building neural networks.

What the field did before. The dominant approach to architecture design—particularly in the efficient attention and SSM literature—has been to propose a new mechanism (kernelized attention, gating, structured matrices), train it on standard benchmarks, and report whether perplexity improves. When an architecture underperforms (as SSMs did on language), the typical diagnosis is "insufficient capacity" or "poor optimization," leading to hyperparameter tuning or scaling up. This is a black-box optimization paradigm: treat the model as opaque, vary architectural choices, and measure outcomes.

The alternative—mechanistic interpretability—has historically been a separate research thread focused on understanding already-trained models. Work on transformer circuits (Elhage et al., 2021) and induction heads (Olsson et al., 2022) reverse-engineered why Transformers can do in-context learning, but these insights were not systematically used to design new architectures. They explained success, not failure.

What makes this paper's approach distinctive. The paper inverts this relationship: it uses synthetic tasks that isolate specific computational primitives (induction heads, associative recall) to diagnose failure modes in existing architectures, then designs a new architecture specifically to implement those missing primitives. The sequence is:

1. Identify that SSMs fail on induction heads (35.6% for S4D, 6.8% for GSS) and associative recall (86.0%, 78.0%)—tasks that attention solves perfectly.
2. Diagnose why: SSMs lack (a) a mechanism to store exact token values after detecting events, and (b) a mechanism to compare tokens across the sequence.
3. Design architectural components (shift SSM for memory buffer, multiplicative interactions for comparison) that implement exactly these primitives.
4. Verify that the design solves the synthetic tasks (100%, 99.8%).
5. Show that solving the synthetic tasks translates to improved natural language modeling (0.4 PPL gap vs. 3.4 PPL gap).

This is not merely "synthetic tasks exist and we solved them." It is a closed-loop methodology where mechanistic understanding directly drives architectural innovation, and synthetic task performance serves as an intermediate validation signal for whether the architecture has the right computational primitives. The paper demonstrates that you can debug a neural network architecture the way you'd debug software: write a unit test for a specific capability, observe failure, add the missing component, and verify the test passes before running the full integration test (language modeling).

Significance beyond raw performance. This methodology matters more than the specific H3 architecture for several reasons. First, it provides a principled answer to "why does architecture X work better than architecture Y?"—a question that is usually answered with hand-waving about inductive biases. H3 works better because it provably implements the associative recall primitive (the construction in Appendix D.1 is an existence proof, not just empirical evidence). Second, it suggests a general research program: for any target capability (e.g., multi-step reasoning, factual recall, compositional generalization), one could design a synthetic diagnostic task, identify which existing architectures fail, and design mechanisms that enable the capability. Third, it explains the negative results in the literature—GSS (Mehta et al., 2022) fails on synthetics not because gating is a bad idea, but because gating alone does not implement token-level copy and comparison.

Evidence anchoring. Table 2 is the linchpin: it shows that two prominent SSM variants fail the synthetic tasks while H3 and attention solve them. Table 3 then shows the translation to language modeling. The synthetic-to-real transfer is the critical empirical link that validates the methodology.

Fundamental vs. incremental. This is a fundamental shift in architecture design methodology—applying mechanistic interpretability prospectively to design rather than retrospectively to explain. It's not a small refinement of existing SSM training recipes.

---

Innovation 2: Decomposing the Attention Mechanism into Transferable Primitives

The paper's second conceptual contribution is the decomposition of attention's success into two separable computational primitives—token recall and token comparison—and the demonstration that these primitives can be implemented in a fundamentally different computational framework (linear time-invariant systems) without approximating attention itself.

What the field assumed before. The dominant assumption in the efficient attention literature has been that attention's power comes from the softmax over dot-product similarities—the specific nonlinear interaction between queries and keys. This assumption motivated approaches that approximate this interaction: Performer uses random features to approximate the softmax kernel, Reformer uses locality-sensitive hashing to approximate the argmax, linear attention replaces softmax with a kernelized dot product. All these approaches attempt to preserve the form of attention (query-key interaction producing a distribution over values) while reducing its cost.

The paper challenges this assumption by asking: what if attention's success comes not from the softmax specifically, but from two more fundamental operations that the softmax happens to implement? The induction head construction (Appendix D.2) shows that a two-layer attention model solves associative recall through: (1) a shift-like operation in the first layer that copies the previous token, and (2) a token-comparison operation in the second layer that matches current tokens against the shifted history. Neither operation requires the specific softmax nonlinearity—just a mechanism to detect matches and a mechanism to copy based on those matches.

Why this decomposition is non-obvious. It's tempting to think of attention as one thing—a sophisticated similarity-weighted retrieval mechanism. The paper shows it can be understood as two things that happen to be coupled in the $\text{softmax}(\mathbf{QK}^\top)\mathbf{V}$ formula. By decoupling them and reimplementing each with different machinery (shift matrix for copy, multiplicative gating for comparison), H3 achieves near-parity with attention while retaining SSMs' asymptotic efficiency. This is a conceptual reframing, not an optimization trick.

The significance: SSMs don't need to approximate attention; they need to implement its primitives. This shifts the research agenda from "how can we make SSMs more like attention?" (which led to hybrid models and gating mechanisms) to "what are the atomic operations that make any sequence model effective, and how can we implement them in the SSM framework?" The fact that H3's solution looks nothing like attention internally—two SSMs with multiplicative interactions vs. a quadratic comparison matrix—yet achieves the same synthetic task performance is strong evidence that the primitives, not the specific mechanism, are what matter.

Evidence anchoring. The constructive proof in Appendix D.1 is critical here: it shows explicitly that a shift SSM (for the copy primitive) plus multiplicative gating (for the comparison primitive) plus a diagonal SSM (for the accumulation primitive) can implement associative recall. This is not an empirical claim about what the model learns—it's a mathematical demonstration that the architecture supports the solution. Table 2 then shows that the model actually converges to this solution during training (achieving 99.8% accuracy).

Fundamental vs. incremental. This is a fundamental conceptual reframing—it changes what "closing the gap with attention" means from "better approximation of softmax" to "implementation of specific computational primitives." It's not incremental because it suggests a completely different research direction for SSM-based language models.

---

Innovation 3: State-Passing as a General Technique for Bridging SSM Theory and Hardware Reality

The FLASHCONV system's most original contribution is not the individual techniques (kernel fusion is borrowed from FlashAttention, block FFT is a classical algorithm), but the state-passing algorithm that exploits the dual recurrent-convolutional nature of SSMs to overcome a fundamental hardware constraint. This is a systems insight with architectural implications.

What the field did before. The standard approach to scaling SSMs to long sequences was to either (a) use a single large FFT (via cuFFT), which becomes IO-bound and cannot fit in GPU SRAM for long sequences, or (b) use the recurrent formulation alone, which is $O(N)$ per step and cannot leverage the parallelism of FFT-based convolution during training. This forced a hard tradeoff: train with FFT (fast but memory-hungry, limiting sequence length) or train with recurrence (slow per-step but constant memory). No prior work had a solution that elegantly spanned both regimes.

The insight that makes state-passing work is specific to SSMs and would not apply to attention. SSMs have a privileged mathematical property: they are simultaneously a convolution (over the whole sequence) and a recurrence (step-by-step), and these two views are exactly equivalent. The state-passing algorithm exploits this by using the efficient convolutional view within chunks (where the FFT fits in SRAM) and the recurrent view between chunks (where only a small state vector needs to be handed off). The correctness proof (Proposition 2) guarantees zero approximation error—this is not a heuristic.

Why this is more than an engineering optimization. The state-passing algorithm fundamentally changes what sequence lengths are practical for SSM training. Before FLASHCONV, training on sequences longer than 8K required falling back to pure recurrence, which is dramatically slower. After FLASHCONV, sequence length is essentially unlimited—it can scale to any length that fits in GPU global memory with near-linear complexity, since the chunk size is determined by SRAM, not total sequence length. This transforms SSMs from "efficient in theory but limited in practice" to "efficient in both theory and practice."

The algorithm also has an elegant recursive structure: because the state-passing loop only needs the end-state of the previous chunk, it could in principle be distributed across multiple GPUs or even across multiple machines, with only the small state vector being communicated. The paper does not explore this, but the architecture naturally supports model parallelism along the sequence dimension—something that is much harder for attention due to its all-to-all token interactions.

Comparison to prior chunking approaches. Chunked processing of sequences is not new—Transformer-XL and Compressive Transformers use segment-level recurrence—but those approaches are approximations that discard information between chunks. State-passing is exact because of the SSM's linearity: the hidden state is a sufficient statistic for all previous inputs with respect to future outputs. No information is lost when compressing the chunk into a state vector. This is a mathematical guarantee, not an empirical observation, and it distinguishes SSMs from any attention-based approach to long sequences.

Evidence anchoring. Figure 2 shows the practical impact: for sequences of 16K and above, state-passing provides 2.3× speedup over the next-best method, and the scaling is nearly linear. Table 8 shows the end-to-end system benefit: S4 with FLASHCONV achieves 5.8× speedup over Transformers on LRA, compared to 2.9× for S4 without FLASHCONV.

Fundamental vs. incremental. The individual components (kernel fusion, block FFT) are incremental engineering borrowed from or adapted from prior work. The state-passing algorithm is a fundamental systems contribution because it establishes a new paradigm for how SSMs can be computed—not as a single FFT or pure recurrence, but as a principled hybrid of both that exactly matches the mathematical definition while adapting to hardware constraints. It's the kind of contribution that could be relevant even if H3 as an architecture is eventually superseded, because it applies to any SSM.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary language modeling benchmark is the Pile (Gao et al., 2021), an 800 GB diverse text corpus, with models trained for 400B tokens. Zero-shot transfer is evaluated on OpenWebText (Gokaslan et al., 2019) and WikiText-103 (Merity et al., 2016). Downstream task evaluation uses the SuperGLUE benchmark in zero-shot and 3-shot settings, with rank classification on logits of possible choices (unless generation is specified). Additional evaluations use PG-19 (Rae et al., 2019), the Long Range Arena benchmark (Tay et al., 2020), and two non-text sequence modeling tasks: seizure classification from raw EEG on the TUSZ v1.5.2 corpus and raw speech classification on the SC10 dataset.
• Base model(s). The paper trains H3-based language models from scratch at four scales: 125M, 355M, 1.3B, and 2.7B parameters. Comparison baselines use publicly available Transformer checkpoints: GPT-2 (Radford et al., 2019), GPT-Neo (Black et al., 2021), and OPT (Zhang et al., 2022). The base SSM components within H3 build on S4D (Gu et al., 2022) and the HiPPO framework (Gu et al., 2020). For the LRA benchmark, the paper uses S4 (Gu et al., 2022) as the base SSM architecture to which FLASHCONV is applied.
• Metrics. Language modeling is evaluated using perplexity (PPL)—the exponential of the average negative log-likelihood per token. Downstream SuperGLUE tasks are evaluated with accuracy (%) using rank classification (comparing logit scores for each possible answer choice). The Long Range Arena benchmark reports accuracy on each subtask. Non-text tasks use accuracy (speech classification) and AUROC (seizure classification). Inference throughput is measured in tokens per second. Speedup on LRA is measured as total wall-clock time relative to a Transformer baseline.
• Baselines. For language modeling: GPT-2 Small (125M), GPT-2 Medium (355M), GPT-2 XL (1.5B) from Radford et al. (2019); GPT-Neo-125M, GPT-Neo-1.3B, GPT-Neo-2.7B from Black et al. (2021); and OPT-125M, OPT-350M, OPT-1.3B, OPT-2.7B from Zhang et al. (2022). For synthetic tasks: S4D (Gu et al., 2022) and Gated State Spaces (GSS; Mehta et al., 2022). For WikiText-103 language modeling: Performer (Choromanski et al., 2021), Reformer (Kitaev et al., 2020), Linear Attention (Katharopoulos et al., 2020), Perceiver AR (Hawthorne et al., 2022), and Transformer-XL (Dai et al., 2019). For FLASHCONV speed benchmarks: cuFFT-based FFTConv, a kernel-fusion-only variant, a block-FFT-only variant, and FlashAttention (Dao et al., 2022). For non-text sequence modeling: Dense-CNN, CNN-LSTM, LSTM, 1D-CNN (seizure); WaveGan-D, Transformer, Performer, CKConv (speech); S4D (LRA accuracy baseline).
• Generation budget / compute accounting. For language modeling, all models are trained for the same number of tokens (400B tokens on the Pile; 50B tokens on OpenWebText for the 125M comparison) using identical optimizer settings and batch sizes. Speed benchmarks measure forward+backward pass time for a fixed batch size and hidden dimension, varying sequence length. For inference throughput (Table 7), batch size is fixed at 64, prompt lengths at 512/1024/1536, generating 128 tokens per sequence. For LRA speedup (Table 8), total training time is compared at fixed hyperparameters. No test-time compute budget variation is studied—this is purely a training-time architecture comparison.
• Cross-validation / statistical protocol. The paper does not employ cross-validation for language modeling results—models are trained once and evaluated on standard test splits. For downstream SuperGLUE evaluation, the paper uses the standard task-specific test sets with prompts from the GPT-3 paper (Brown et al., 2020). For fMRI experiments, 20 training runs with different random seeds are used, and results are reported with 95% confidence intervals. For the synthetic language tasks, 5000 training examples and 500 test examples are sampled from the same distribution, with models trained for 200 epochs.

Main Quantitative Results

Synthetic Language Tasks: Closing the Expressivity Gap

The synthetic tasks serve as diagnostic probes to validate that H3 implements the token recall and comparison primitives that existing SSMs lack. Results are in Table 2 for 2-layer models.

Induction Head Task. H3 achieves 100.0% accuracy, matching attention's 100.0%. In contrast, S4D achieves only 35.6% and Gated State Spaces (GSS) achieves 6.8%—the latter barely above the random baseline of 5.0% (given 20 vocabulary items). The gap between H3 and existing SSMs is stark: GSS performs 93.2 percentage points worse, S4D 64.4 points worse. This task specifically requires detecting a special marker token and recalling the subsequent token—exactly the capability that the shift SSM (with its finite memory buffer of exact inputs) is designed to provide.

Associative Recall Task. H3 achieves 99.8%, compared to 100.0% for attention, 86.0% for S4D, 78.0% for GSS, and 25.0% for random. While S4D's 86.0% might appear reasonable, it represents failure on approximately 1 in 7 test examples—a substantial error rate for a diagnostic task with only 10 vocabulary items and sequence length 20. GSS's 78.0% is worse, suggesting that gating mechanisms alone do not address the missing comparison and recall primitives. H3's near-perfect performance demonstrates that the combination of shift SSM (for detecting key-value adjacency), diagonal SSM (for accumulating key-value pairs), and multiplicative interactions (for gating based on key matches) successfully implements the associative recall computation, consistent with the constructive proof in Appendix D.1.

Translation to Language Modeling. Table 3 shows that solving the synthetic tasks correlates with improved natural language modeling on OpenWebText. A 12-layer, 125M-parameter H3 model achieves 21.0 PPL, compared to 24.9 for S4D and 24.0 for GSS—improvements of 3.9 and 3.0 PPL respectively. The gap to Transformers (20.6 PPL) narrows to 0.4 PPL, compared to 4.3 PPL for S4D. This supports the paper's central thesis: the capabilities diagnosed by the synthetic tasks (token recall and comparison) are not merely academic curiosities but are genuinely important for modeling real language.

Language Modeling: H3 and Hybrid Models at Scale

OpenWebText (125M scale, Table 3). The pure H3 model (21.0 PPL) closes the gap with Transformers (20.6 PPL) to 0.4 PPL from the 3.4–4.3 PPL gap of prior SSMs. The hybrid H3-attention model with 2 attention layers achieves 19.6 PPL, outperforming Transformers by 1.0 PPL. This is the paper's headline language modeling result: a model where 10 out of 12 layers use H3 surpasses the standard Transformer architecture. The GSS hybrid achieves 19.8 PPL, 0.2 PPL worse than the H3 hybrid despite also having 2 attention layers, suggesting H3 is a stronger SSM building block than GSS even within hybrid architectures.

The Pile (125M–2.7B scale, Table 4). Across all four model sizes, hybrid H3-attention models trained on the Pile for 400B tokens achieve lower perplexity than comparably-sized Transformer baselines:

• 125M: Hybrid H3 achieves 8.8 PPL vs. 9.4 for GPT-Neo-125M (0.6 PPL improvement). GPT-2 Small achieves 19.0 PPL, but this is not directly comparable since GPT-2 was trained on WebText, not the Pile.
• 355M: Hybrid H3 achieves 7.1 PPL vs. 13.9 for GPT-2 Medium (not directly comparable due to training data differences). No GPT-Neo baseline exists at this size.
• 1.3B: Hybrid H3 achieves 6.0 PPL vs. 6.2 for GPT-Neo-1.3B (0.2 PPL improvement) and 12.4 for GPT-2 XL.
• 2.7B: Hybrid H3 achieves 5.4 PPL vs. 5.7 for GPT-Neo-2.7B (0.3 PPL improvement).

The perplexity advantage is most pronounced at the smallest scale (0.6 PPL at 125M) and narrows at larger scales (0.2–0.3 PPL at 1.3B–2.7B). This could reflect that larger Transformers develop better internal mechanisms for long-range dependencies, reducing H3's relative advantage, or that H3's hyperparameters (borrowed from GPT-3 without tuning) become suboptimal at larger scales. On zero-shot transfer to OpenWebText and WikiText-103 (Table 4, columns 2–3), hybrid H3 models consistently outperform GPT-Neo and GPT-2 models of the same size—e.g., Hybrid H3-2.7B achieves 11.0 PPL on OpenWebText vs. 11.7 for GPT-Neo-2.7B, and 10.6 PPL on WikiText-103 vs. 11.5.

Zero-Shot SuperGLUE (Table 5). Hybrid H3 models achieve the highest average accuracy at three of four model sizes:

• 125M: Hybrid H3 averages 53.9% vs. 50.8% for GPT-Neo-125M and 50.1% for OPT-125M. The H3 hybrid wins on 5 of 8 tasks (RTE, CB, ReCoRD, COPA, WSC tied with OPT).
• 355M: Hybrid H3 averages 54.7% vs. 53.1% for OPT-350M and 52.6% for GPT-2 Medium. Wins on 5 of 8 tasks.
• 1.3B: Hybrid H3 averages 56.5% vs. 52.9% for OPT-1.3B and 52.1% for GPT-Neo-1.3B. Wins on 5 of 8 tasks, with notably strong performance on ReCoRD (67.8% vs. 61.8% for OPT) and COPA (74.0% vs. 69.0% for OPT).
• 2.7B: Hybrid H3 averages 56.8% vs. 55.5% for OPT-2.7B and 54.6% for GPT-Neo-2.7B. Wins on 4 of 8 tasks (WIC, RTE, ReCoRD, COPA).

A notable exception: the pure H3-125M model (Table 14) performs erratically, achieving 61.5% on WSC (far above all baselines) but catastrophic failure on MultiRC (4.6%) and ReCoRD (15.8%). The authors attribute the low performance on some tasks to the model generating long, unparseable text responses rather than brief answers—a failure mode partially addressed by the few-shot setting and the hybrid architecture.

Three-Shot SuperGLUE (Table 6). The pattern is similar to zero-shot:

• 125M: Hybrid H3 averages 53.7% vs. 47.9% for OPT-125M and 47.2% for GPT-Neo-125M. Wins on 6 of 8 tasks.
• 355M: Hybrid H3 averages 52.6% vs. 50.1% for OPT-350M and 45.6% for GPT-2 Medium.
• 1.3B: Hybrid H3 averages 53.0% vs. 51.2% for GPT-Neo-1.3B and 49.4% for OPT-1.3B.
• 2.7B: Hybrid H3 averages 55.5% vs. 53.0% for OPT-2.7B (tied) and 51.9% for GPT-Neo-2.7B. Wins on 4 of 8 tasks.

The hybrid models' advantage appears robust across both zero-shot and few-shot settings, though the gap is often small (1–3 percentage points average) and task-specific variation is substantial. The strong and consistent performance on ReCoRD (55.0%→67.8%→71.3% at 125M→1.3B→2.7B in zero-shot; 55.0%→67.6%→71.1% in 3-shot) and COPA (67.0%→74.0%→81.0% in zero-shot; 67.0%→76.0%→77.0% in 3-shot) suggests H3 hybrids may be particularly effective at tasks requiring reading comprehension and causal reasoning over relatively long contexts.

Inference Throughput (Table 7)

At the 1.3B scale with batch size 64 and generating 128 tokens per sequence, the hybrid H3 model achieves substantially higher throughput than a pure Transformer:

• Prompt length 512: 1980 tokens/s (H3) vs. 1340 tokens/s (Transformer) — 1.48× speedup
• Prompt length 1024: 1580 tokens/s vs. 770 tokens/s — 2.05× speedup
• Prompt length 1536: 1240 tokens/s vs. 520 tokens/s — 2.38× speedup

The speedup increases with prompt length because the attention layers in the Transformer must attend over increasingly long key-value caches (cost scaling with $O(N)$ for each new token), while the SSM layers in the hybrid model maintain constant-time-per-token generation via the recurrent formulation. The two attention layers in the hybrid still incur some KV-cache overhead, but this is amortized across the 12+ total layers. The paper does not report pure H3 inference throughput, which would presumably show even larger speedups.

Non-Text Sequence Modeling

Seizure Classification from EEG (Table 18). H3 achieves 83.2 AUROC on 60-second EEG clips (12,000 time steps at 200 Hz), outperforming all baselines: Dense-CNN (78.0), 1D-CNN (69.7), LSTM (69.3), and CNN-LSTM (68.6). The Transformer baseline is marked "x" (cannot process sequences of length 12,000 due to memory constraints). This demonstrates that H3's linear complexity enables modeling sequence lengths that are simply infeasible for Transformers.

Raw Speech Classification (Table 19). On the SC10 10-class speech commands task with sequences of length 16,000, H3 achieves 97.04% accuracy, slightly below S4 (97.50%) but well above CKConv (71.66%) and Performer (30.77%). The Transformer baseline cannot process sequences of this length. This is noteworthy because H3 was designed for autoregressive language modeling—not bidirectional classification—yet it performs competitively with S4, which was specifically optimized for LRA-style tasks.

Functional MRI (Table 20, Figures 3–4). On mental state decoding from pre-trained fMRI models, H3 and GPT achieve statistically indistinguishable performance. On HCP (20-way classification): H3 achieves 88.75% accuracy (±0.33) vs. GPT at 88.44% (±0.39). On MDTB (26-way classification): H3 achieves 88.25% (±0.45) vs. GPT at 89.47% (±0.44). The slight H3 disadvantage on MDTB (1.22 percentage points) is within overlapping confidence intervals. The upstream pre-training loss (Figure 3) is also comparable, with H3 (dropout 0.2) tracking closely with GPT. The error distributions across brain regions (Figure 4) are visually similar. These results suggest H3 matches Transformer performance on yet another modality where sequence length is moderate but the number of training samples is small—a regime where overfitting is a primary concern.

WikiText-103 and PG-19: Comparison to Efficient Attention Variants

WikiText-103 (Table 10). The hybrid H3-125M achieves 18.5 PPL, slightly outperforming the Transformer-125M (18.6 PPL) and dramatically outperforming Performer (26.8), Reformer (26.0), and Linear Attention (25.6). The larger Perceiver AR (358M, 18.4 PPL) and Transformer-XL (285M, 18.4 PPL) are roughly comparable but use different tokenizers and may not be directly comparable. The 7–8 PPL gap between H3/Transformers and efficient attention variants (Performer, Reformer, Linear Attention) quantifies the cost of approximating attention's softmax with kernelized or sparse alternatives—a cost H3 avoids by not approximating attention at all but instead implementing the underlying primitives natively in the SSM framework.

PG-19 (Table 11). The hybrid H3-125M achieves 16.2 PPL, outperforming both the Transformer-125M (17.0 PPL) and Linear Attention-125M (19.1 PPL). The 0.8 PPL gap between H3 and the Transformer is consistent with the pattern observed on OpenWebText and the Pile. The 2.9 PPL gap between H3 and Linear Attention on this long-form book dataset (PG-19 contains full-length books, with much longer average document length than WebText) suggests H3's advantage may be particularly pronounced on long-context language modeling, though this is not systematically investigated across varying context lengths.

FlashConv Speed Benchmarks

Long Range Arena Speedup (Table 8). S4 with FLASHCONV achieves 5.8× speedup over the Transformer baseline on the LRA benchmark, compared to 2.9× for S4 without FLASHCONV, 2.4× for FlashAttention, and 2.8× for Block-sparse FlashAttention. This represents a doubling of S4's training speed (2.9× → 5.8×) purely through systems optimizations, without any change to the model architecture or accuracy. The comparison is on total training time, meaning FLASHCONV accelerates the entire training pipeline end-to-end, not just individual operations.

Component Ablation of FFTConv Speed (Figure 2). The paper measures forward+backward pass time on an A100-SXM4-40GB GPU with batch size 8 and hidden dimension 1024, varying sequence length from 256 to 32K:

• Kernel fusion provides up to 3.4× speedup over naive cuFFT for short sequences (256–512 tokens). This is the regime where IO dominates: cuFFT incurs multiple reads/writes of intermediate results to global memory, and fusion eliminates these.
• Block FFT (on top of kernel fusion) adds up to 2× further speedup for medium sequences (1K–8K tokens). This is the regime where compute becomes the bottleneck, and block FFT's use of tensor cores for matrix multiplications within the FFT decomposition provides the advantage despite the slightly higher FLOP count ($O(N r \log N / \log r)$ vs. $O(N \log N)$).
• State-passing (on top of fusion + block FFT) provides up to 2.3× speedup for long sequences (16K–32K) compared to the best alternative that fits in SRAM. At these lengths, the fused block FFT can no longer fit entirely in SRAM, making state-passing essential.

Critically, Figure 2 shows that FLASHCONV with state-passing maintains near-linear scaling in wall-clock time even to 32K sequence length, while FlashAttention (the fastest exact attention implementation) scales superlinearly and is approximately 15–20× slower than FLASHCONV at 32K. This visualizes the asymptotic advantage of SSMs over attention when implemented with hardware-efficient algorithms: at $N = 256$, FlashAttention and FLASHCONV are comparable; at $N = 32\text{K}$, the gap is roughly 20×.

H3 Training Speed vs. Attention (Figure 2, interpretation). The FFTConv benchmarks are measured on the SSM convolution operation itself, not the full H3 layer. For H3 specifically, the additional matrix multiplications ($\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$ projections, output projection) add $O(d^2 N)$ overhead that is shared with attention. The paper claims H3 is "4–8× times faster than attention for long sequences" (Section 1) based on these benchmarks, but this figure includes only the SSM-vs-attention comparison, not the full layer. In practice, the hybrid model's inference speedup (Table 7) is 1.5–2.4× because the attention layers and MLP layers in the Transformer become the bottleneck at moderate sequence lengths.

LRA Accuracy (Table 9). Although FLASHCONV does not change model accuracy (it is a systems optimization), the paper verifies that H3 maintains competitive LRA accuracy: H3 averages 84.8% across the six LRA tasks vs. 85.0% for S4D, with H3 outperforming S4D on Text (88.2% vs. 87.3%) and Retrieval (91.0% vs. 90.7%) while slightly underperforming on ListOps (57.5% vs. 58.3%), Image (87.3% vs. 87.5%), Pathfinder (93.0% vs. 93.6%), and Path-X (91.8% vs. 92.3%). This establishes that H3 does not sacrifice long-range modeling capability—for which S4 was explicitly designed—in exchange for its language modeling improvements.

Length Extrapolation and Data Scaling

Length Extrapolation (Table 12). An H3 model trained on associative recall sequences of length 20 achieves 99.8% accuracy at length 20 and 98.4% accuracy at length 40—twice the training length. This demonstrates that H3 retains the length extrapolation property characteristic of SSMs, thanks to the recurrent formulation that does not depend on absolute position encodings (unlike standard Transformers, which require positional embeddings that may not generalize beyond training lengths). The 1.4 percentage point degradation at 2× length suggests some mild distribution shift but largely preserved capability.

Data Scaling (Table 13). Hybrid H3-125M and Transformer-125M models are trained on the Pile for 5B, 10B, and 15B tokens. H3 maintains a consistent perplexity advantage: 11.8 vs. 12.7 at 5B tokens, 10.7 vs. 11.3 at 10B, and 10.2 vs. 10.7 at 15B. The gap (0.5–0.9 PPL) is roughly constant across data scales, suggesting that H3's advantage is not merely a small-data phenomenon that would vanish with more training. However, the data scales studied (5B–15B tokens) are far below the 400B tokens used for the main results, so extrapolation to much larger data regimes is speculative. The paper does not study H3's scaling behavior with respect to model size and data quantity jointly (a Chinchilla-style scaling law), which would be necessary to determine whether the H3 advantage persists at compute-optimal training scales.

Generation Quality: Qualitative Differences

Tables 16 and 17 report SuperGLUE performance using free-form generation (rather than rank classification). The hybrid H3 models show a consistent pattern: strong few-shot performance but weak zero-shot on tasks requiring specific answer formats. For example, Hybrid H3-125M achieves 0.0% on WSC and WIC in zero-shot—the model generates text but not in a form that contains the expected "yes"/"no" answer. In the 3-shot setting, performance recovers: 43.3% on WSC and 49.1% on WIC. This suggests H3 models can learn the expected output format from in-context examples but struggle to infer it from the task description alone, potentially because the H3 layers' inductive bias toward sequence continuation (rather than classification) makes them less likely to produce isolated answer tokens without explicit demonstration.

Ablation Studies and Robustness Checks

Pure H3 vs. Hybrid H3-Attention: Table 3 shows that adding 2 attention layers to a 12-layer H3 model improves OpenWebText perplexity from 21.0 to 19.6 (1.4 PPL improvement), surpassing the Transformer baseline (20.6 PPL). This demonstrates that H3 alone, while competitive (within 0.4 PPL of Transformers), does not fully replace attention—the hybrid architecture is strictly better. The pure H3-125M model's erratic zero-shot SuperGLUE performance (Table 14: 61.5% WSC, but 4.6% MultiRC, 15.8% ReCoRD) further suggests that H3 alone has specific failure modes that attention layers mitigate. The placement of attention layers (layers 2 and $N/2+2$) is not ablated, so it is unknown whether these specific positions are optimal or whether any 2 attention layers would produce similar results.

H3 vs. Gated State Spaces (GSS): Table 3 shows H3 (21.0 PPL) outperforms GSS (24.0 PPL) by 3.0 PPL, and the H3 hybrid (19.6 PPL) outperforms the GSS hybrid (19.8 PPL). Table 2 shows H3 dramatically outperforms GSS on synthetic tasks: 100.0% vs. 6.8% on induction heads and 99.8% vs. 78.0% on associative recall. These comparisons establish that H3's specific design (shift SSM + diagonal SSM + multiplicative interactions) is substantially more effective than GSS's gating-based approach, despite both being SSM variants targeting language modeling. The synthetic task results suggest that GSS's gating mechanism does not provide the token recall and comparison primitives needed for these tasks.

S4D vs. H3 on LRA Accuracy (Table 9): H3 achieves 84.8% average LRA accuracy vs. 85.0% for S4D, a negligible 0.2 percentage point difference. This is important because S4D was explicitly designed for and optimized on LRA—H3 was not. The fact that H3 matches S4D on LRA while dramatically outperforming it on language modeling (Table 3: 21.0 vs. 24.9 PPL) suggests that H3's architectural modifications (shift SSM, multiplicative interactions) improve language-specific capabilities without sacrificing general long-range sequence modeling ability.

Kernel Fusion Alone vs. Fusion + Block FFT vs. Full FLASHCONV (Figure 2): This ablation quantifies the marginal benefit of each FLASHCONV component across sequence lengths. Kernel fusion provides the largest relative gains at short lengths (3.4× at 256), block FFT provides the largest gains at medium lengths (2× at 1K–8K), and state-passing enables scaling beyond 8K with additional speedup (2.3× at 32K). The ablation validates that each component addresses a distinct bottleneck: IO at short lengths, compute at medium lengths, and SRAM capacity at long lengths.

Dropout in fMRI H3 Models (Figure 3): For the fMRI pre-training experiment, the paper evaluates three dropout rates for H3: 0.1, 0.2, and 0.3. The 0.2 dropout variant performs best and is used for downstream evaluation, while 0.1 appears to underfit and 0.3 to overfit. The dropout sensitivity is comparable to Transformers (which use 0.1), though the optimal value differs.

Efficient Attention Baselines (Tables 10–11): On WikiText-103, Performer (26.8 PPL), Reformer (26.0 PPL), and Linear Attention (25.6 PPL) all dramatically underperform H3 (18.5 PPL) and Transformers (18.6 PPL). On PG-19, Linear Attention achieves 19.1 PPL vs. 16.2 for H3. These comparisons demonstrate that efficient approximations of attention sacrifice substantial modeling quality—a cost H3 avoids by not being an attention approximation at all. The 7–8 PPL gap on WikiText-103 is larger than the gap between SSMs and Transformers that the paper aims to close, highlighting that the efficient attention path may be more challenging than the SSM path for language modeling.

Generation vs. Rank Classification for SuperGLUE (Tables 5–6 vs. 16–17): The paper evaluates SuperGLUE with both rank classification (scoring each answer choice's logit given the prompt) and free-form generation (generating text and searching for the answer). In the zero-shot setting (Table 16 vs. Table 5), H3 hybrids perform substantially worse with generation—e.g., Hybrid H3-125M achieves 0.0% on WSC and WIC via generation vs. 39.4% and 51.4% via rank classification. This gap largely closes in the 3-shot setting (Table 17 vs. Table 6), suggesting that H3 models can learn output formatting from examples. This is a practical limitation: H3 models require few-shot prompting to produce parseable outputs on classification tasks, whereas Transformers (particularly OPT and GPT-2) are more likely to produce clean answers zero-shot. The paper does not explore why this is the case—it may relate to H3's training data or to architectural differences in how uncertainty is represented.

H3 Head Dimension (Appendix F.6 context): The pure H3 model uses head dimension $d_h = 8$, while the hybrid uses $d_h = 1$. The paper does not ablate this choice. With $d_h = 1$, each head is a scalar, meaning the outer product $\bar{\mathbf{K}}^{(h)} (\mathbf{V}^{(h)})^\top$ produces a $1 \times 1$ "matrix" (a scalar), and the diagonal SSM accumulates scalar key-value associations. With $d_h = 8$, each head processes 8-dimensional key and value vectors, enabling richer interactions. The hybrid model's better performance despite lower head dimension suggests that attention layers compensate for any expressivity lost by the scalar heads.

Training Data Quantity (Table 13): The relative H3-vs-Transformer gap (0.5–0.9 PPL) is stable across 5B, 10B, and 15B training tokens, suggesting the advantage is not an artifact of small-data regimes. However, this is only 3.75% of the full 400B training run, making extrapolation to full training scale uncertain.

Critical Assessment

The experimental results provide strong support for the paper's central narrative—that SSMs underperform attention on language because they lack specific computational primitives (token recall and comparison), and that adding these primitives via H3's architecture bridges most of the gap. However, the evidence for this narrative is stronger at some levels than others, and several important claims rest on thin experimental support or suffer from confounds.

On the causal link between synthetic task performance and language modeling. The paper's core methodology is to use synthetic tasks as diagnostics, design H3 to solve them, and then demonstrate improved language modeling. Table 2 (synthetics) and Table 3 (language) provide the before-and-after: existing SSMs fail synthetics and have poor language PPL; H3 solves synthetics and has much better language PPL. This is compelling as a correlation—architectures that solve the synthetic tasks also model language better. But it falls short of establishing causation—that solving the synthetic tasks causes better language modeling. The missing experiment is an ablation where H3 is modified to prevent it from solving the synthetic tasks (e.g., removing the shift SSM, removing multiplicative interactions, or reducing state size $m$ to 1) and showing that language PPL degrades accordingly. Without such an ablation, alternative explanations are possible: H3 might simply be a better-optimized architecture (different initialization, better gradient flow) that happens to both solve synthetics and model language well, or the multi-head structure with outer products might provide a general representational benefit unrelated to the specific recall and comparison primitives. The constructive proof in Appendix D.1 shows H3 can implement associative recall, but does not prove that this is why it achieves better language PPL—the model could implement the task using a different mechanism that the proof does not capture.

On the claim that hybrid H3-attention "outperforms Transformers." Tables 3 and 4 provide evidence that hybrid H3 models achieve lower perplexity than GPT-Neo and OPT baselines of the same parameter count. Several caveats apply. First, the training recipes are not identically controlled: GPT-Neo used a different training infrastructure (Mesh-Tensorflow) and potentially different hyperparameters; OPT used a different training data mixture and preprocessing. The paper trains H3 models with GPT-3 hyperparameters, which may be better tuned than those used for GPT-Neo. The fair comparison is between H3 and a Transformer trained with identical code, data, and hyperparameters—the paper provides this only for the 125M OpenWebText comparison (Table 3), where the hybrid H3 indeed outperforms the Transformer (19.6 vs. 20.6 PPL). For the Pile results, no identically-trained Transformer baseline exists. Second, the perplexity gaps are small at larger scales (0.2–0.3 PPL at 1.3B–2.7B), and it is unclear whether they would persist with hyperparameter tuning optimized for each architecture. The paper acknowledges not tuning H3 hyperparameters due to resource constraints. A well-tuned Transformer might close or reverse the gap. Third, the SuperGLUE results (Tables 5–6) are mixed: hybrid H3 wins on average but not decisively, and the per-task variation is large. The pure H3 model's catastrophic failures on MultiRC and ReCoRD raise questions about whether the architecture has specific weaknesses on tasks requiring integration of information across many sentences.

On the scalability of H3 beyond 2.7B parameters. The paper does not train models larger than 2.7B parameters or on more than 400B tokens, leaving open whether the H3 advantage persists at the scales of contemporary LLMs (10B–100B+ parameters, 1T+ tokens). The narrowing gap from 125M to 2.7B (0.6 PPL → 0.3 PPL) is concerning: if the trend continues, H3 and Transformers might converge at larger scales, or H3 might even underperform. The paper's data scaling experiment (Table 13) shows a stable gap up to 15B tokens, but this is too small a range to extrapolate reliably. A Chinchilla-style scaling law analysis (training multiple model sizes on multiple data quantities and fitting parametric curves) would be needed to make stronger claims about H3's scalability. Without this, the paper's results are best viewed as evidence that H3 is competitive at moderate scales (up to 2.7B parameters), not that it will necessarily outperform Transformers at frontier scales.

On FLASHCONV speedup claims. The 5.8× speedup on LRA (Table 8) and the near-linear scaling to 32K in Figure 2 are clear and well-supported. However, the paper's claim of "4–8× speedup" for H3 training uses a narrow definition of "speedup"—it measures the SSM convolution operation alone, not the end-to-end training throughput of an H3 model. In practice, training throughput depends on many factors: the matrix multiplications in linear projections and MLPs, the attention layers in the hybrid model, communication overhead in distributed training, and data loading. Table 7 shows that inference throughput improvement is 1.5–2.4×, not 4–8×, because the attention layers and MLPs dominate at moderate sequence lengths. The paper does not report end-to-end training throughput for H3 vs. Transformers at equivalent batch sizes and sequence lengths, which would be the fairest metric for practitioners.

On the single-model-family concern. All language modeling experiments use the GPT-2 tokenizer and roughly GPT-3-style architecture (pre-norm, specific MLP ratios, specific initialization). It is unknown whether H3's advantage would transfer to other tokenizers, other architectural conventions (post-norm, parallel attention/MLP), or other model families (encoder-decoder, mixture-of-experts). The non-text experiments partially address this by showing H3 works on fMRI, EEG, and speech, but these use totally different architectures optimized for each domain. The synthetic tasks, while informative, use tiny vocabularies (10–20 tokens) and fixed sequence lengths (20–30)—far from the complexity of real language. A skeptical reading is that H3 works well on the specific training setup used in the paper, and generalization to other setups is plausible but unproven.

On the 2-attention-layer hybrid design. The paper places attention layers at fixed positions (layers 2 and $N/2+2$) without ablating the number of attention layers (would 1 or 4 be better?) or their positions (would attention at layers 1 and $N$ be better?). The hybrid architecture is therefore a point solution—one specific configuration that works well—rather than a principled optimization. The fact that GSS hybrids also benefit from attention layers (19.8 PPL in Table 3) suggests that hybrid SSM-attention models are generally effective, but H3's specific contribution is the stronger SSM building block, not the hybrid design itself.

Missing experiments that would have strengthened the paper:

• Ablation of the shift SSM: Replace the shift SSM with a diagonal SSM (making H3 use two diagonal SSMs) and measure synthetic and language PPL. This would isolate the contribution of explicit finite memory.
• Ablation of multiplicative interactions: Replace the elementwise products with additions (making the information flow additive rather than gated) and measure performance. This would test whether the gating mechanism is crucial or incidental.
• H3 trained with Transformer-identical code and data at multiple scales: For the Pile results, train a Transformer baseline with identical infrastructure, data preprocessing, and hyperparameters to enable clean comparison. The current baselines (GPT-Neo, OPT) introduce uncontrolled variables.
• Scaling laws: Train H3 and Transformer models at multiple sizes (e.g., 125M, 350M, 760M, 1.3B) on multiple data quantities and fit parametric scaling laws to predict whether the H3 advantage persists or diminishes at frontier scales.
• Long-context language modeling evaluation: Given H3's theoretical advantage for long sequences, evaluate perplexity at sequence lengths beyond 2048 (e.g., 4096, 8192, 16384) on datasets like PG-19 that contain long documents. This would directly test whether H3's asymptotic efficiency translates to practical long-context language modeling benefits.
• Attention layer count sweep: Evaluate hybrid models with 0, 1, 2, 4, and all attention layers to map the performance frontier and determine the optimal ratio. The current 2-attention-layer design is justified only by a single data point.

On claims that are well-supported:

• The claim that existing SSMs (S4D, GSS) fail on associative recall and induction heads is conclusively demonstrated in Table 2.
• The claim that H3 solves these synthetic tasks is demonstrated with near-perfect accuracy (Table 2) and an explicit constructive proof (Appendix D.1).
• The claim that H3 substantially closes the SSM-Transformer gap on language modeling is supported by Table 3 (21.0 vs. 20.6 PPL for Transformers, compared to 24.0–24.9 for existing SSMs).
• The claim that FLASHCONV accelerates SSM training and inference is supported by Figure 2 and Tables 7–8, with careful component-wise ablation showing where each technique helps.
• The claim that hybrid H3-attention models are competitive with or outperform Transformers at moderate scales is supported by Tables 3–6, with the caveat that the baselines are not identically trained for the Pile results.
• The claim that H3 works across modalities is supported by Tables 18–20 and Figure 3, showing strong performance on EEG, speech, and fMRI.

Bottom line: The paper convincingly demonstrates that architectural innovations inspired by mechanistic understanding can substantially close the gap between SSMs and attention on language modeling at moderate scales. Whether this advantage persists at the scales of frontier LLMs, whether it generalizes beyond the specific training setup used, and whether the synthetic-task-to-language-modeling link is causal or merely correlational remain open questions. The paper is best viewed as a compelling proof of concept for the diagnostic-synthetic-task methodology and for H3's architectural approach, rather than as definitive evidence that SSMs will replace attention in large-scale language modeling.

6. Limitations and Trade-offs

6.1 The Gap Between Synthetic Task Solutions and Real Language Modeling Improvement Is Correlational, Not Proven Causal

The assumption or constraint. The paper's central methodology is to identify missing primitives via synthetic tasks, design H3 to implement them, and then demonstrate improved language modeling. The implicit assumption is that solving the synthetic tasks causes the improved language modeling—that the shift SSM and multiplicative interactions close the language gap specifically because they provide token recall and comparison. The paper provides a constructive proof that H3 can implement associative recall (Appendix D.1) and shows empirically that H3 solves the synthetic tasks (Table 2) while achieving better language PPL than SSMs that fail them (Table 3). However, the paper never establishes the causal direction through the obvious control experiment: modifying H3 to remove the specific primitives it claims are responsible (e.g., replacing the shift SSM with a diagonal SSM, or replacing multiplicative interactions with additive ones) and measuring how much of the language modeling gain disappears.

The consequence. Without this causal evidence, alternative explanations for H3's language modeling improvement remain viable. H3 might perform better simply because it is a deeper, more expressive architecture—stacking two SSMs with outer products provides more parameters and more nonlinear interactions than a single SSM, regardless of the specific shift-vs-diagonal or multiplicative-vs-additive design choices. It might be that the multi-head outer product structure provides a beneficial representation learning effect (analogous to how the paper's PRM training in the reference example improved performance even when only the last-step score was used) that is unrelated to the token recall and comparison primitives. Or it might be that H3's initialization or gradient flow properties are simply better than S4D's or GSS's, and the synthetic task performance is a side effect rather than the mechanism. For a practitioner deciding whether to adopt H3, this distinction matters: if the specific primitives are causal, then carefully preserving them in any variant or extension is critical; if they are incidental to a generally better architecture, then similar gains might be achievable with different design choices that are simpler or more efficient.

What evidence exists in the paper. The paper's evidence for the primitives-to-language link is the correlation in Tables 2 and 3: architectures that fail synthetics (S4D: 35.6% induction, 86.0% associative recall; GSS: 6.8%, 78.0%) have worse language PPL (24.9, 24.0), while architectures that solve synthetics (H3: 100%, 99.8%; attention: 100%, 100%) have better language PPL (H3: 21.0, attention: 20.6). This is a consistent pattern but only across four data points (S4D, GSS, H3, attention), and it does not isolate the specific architectural components. The paper provides no ablation where H3's shift SSM is replaced with a diagonal SSM (keeping all else equal) to see whether synthetic task performance drops and language PPL degrades accordingly. The constructive proof (Appendix D.1) shows H3 can solve the task—it does not show that H3 does solve the task in practice using the constructed mechanism, nor that this mechanism is responsible for language modeling gains.

Mitigation status. The paper does not address this limitation or acknowledge it as such. It presents the synthetic-to-real transfer as validated by the correlation, without discussing alternative explanations or proposing the necessary ablations. The authors do not frame the causal link as an open question requiring further work.

---

6.2 Difficulty Estimation and Compute-Optimal Allocation of Architectural Resources Are Not Explored

The assumption or constraint. The paper demonstrates that a hybrid architecture—2 attention layers amid many H3 layers—outperforms both pure H3 and pure Transformers (Table 3: 19.6 PPL hybrid vs. 21.0 H3 vs. 20.6 Transformer). However, the hybrid configuration is presented as a single fixed design (attention at layers 2 and $N/2+2$) rather than as a problem of compute-optimal architecture selection. The paper never studies how the optimal number or placement of attention layers varies with model size, data scale, task difficulty, or sequence length. This is precisely the kind of allocation problem that the reference example paper would have approached by estimating "difficulty" (task characteristics) and selecting the best strategy per instance—here, the analogous question would be determining which tokens or which layers benefit most from attention's quadratic comparison vs. H3's linear-time memory.

The consequence. For a practitioner building a language model, the fixed hybrid design leaves critical questions unanswered. Is 2 attention layers optimal, or would 1, 3, or 4 work better? The pure H3's erratic zero-shot performance on SuperGLUE (Table 14: 61.5% WSC but 4.6% MultiRC, 15.8% ReCoRD) suggests that H3 has specific failure modes—likely on tasks requiring integration of information across many sentences—that attention layers mitigate. But without understanding which tasks or tokens need attention, a practitioner cannot adapt the architecture to their specific deployment distribution. If the deployment involves mostly short-form classification (where H3 struggles zero-shot), more attention layers might be needed; if it involves long-form generation, fewer might suffice. The fixed hybrid also provides no guidance on whether the optimal ratio of H3 to attention layers changes with model scale—the narrowing perplexity gap from 125M to 2.7B parameters (0.6 PPL → 0.3 PPL) could mean that larger models need proportionally fewer attention layers, or that the specific placement tested becomes increasingly suboptimal.

Furthermore, the paper does not account for the cost of including attention layers in the efficiency analysis. The headline speedups (2.4× inference in Table 7) are for a hybrid model where attention layers still incur KV-cache overhead. A pure H3 model with the same quality as the hybrid—if achievable—would be significantly faster. The paper never investigates this tradeoff: how much perplexity must one sacrifice for what inference speedup by removing attention layers entirely? This is directly analogous to the compute-optimal tradeoff the reference example paper studies—but here, the allocation is over architectural components rather than test-time strategies.

What evidence exists in the paper. The paper provides exactly one comparison relevant to this: Table 3 shows that pure H3 (21.0 PPL) underperforms the hybrid (19.6 PPL) and the hybrid outperforms the Transformer (20.6 PPL). The difference between pure H3 and the hybrid (1.4 PPL) is the cost of removing attention—but this is at exactly one model size (125M) on one dataset (OpenWebText) with one placement of attention layers. Table 14 shows the pure H3's zero-shot failures, providing task-level evidence for where attention is needed. No sweep over the number of attention layers is performed. No study of how attention layer placement affects performance exists. No analysis of what fraction of tokens or sequence positions benefit from the attention layers' token-comparison capability is provided.

Mitigation status. The paper acknowledges implicitly that pure H3 is not sufficient (the hybrid outperforms it), but does not frame this as a resource allocation problem requiring further study. The complementarity of attention and SSMs is noted positively in Section 7 and Appendix A as a promising direction, but the paper treats the fixed hybrid as a solution rather than an instance of a more general optimization problem. The limitation is not flagged as one requiring future work on adaptive or difficulty-conditioned architectural allocation.

---

6.3 The Difficulty Estimation Cost for the PRM Search and Revision Models Is Not Accounted For in Efficiency Claims

(Note: This limitation is about a methodological pattern in the paper, not about PRMs specifically—the paper does not use PRMs. Rephrased to match the paper's actual content.)

The assumption or constraint. The paper's synthetic-to-real methodology incurs a hidden cost that is never quantified: the computational expense of developing and validating the synthetic tasks themselves. The paper identifies the missing primitives (token recall, token comparison) through synthetic tasks designed with knowledge of how attention solves them (Appendix D.2 provides the attention construction for associative recall). This diagnostic process—designing synthetic tasks, evaluating architectures on them, iterating on design—required substantial researcher computation and insight that is not amortized in any reported metric. More concretely, the paper does not account for the cost of training the models on synthetic tasks (5000 training examples, 200 epochs each, Table 2) as part of the development process—these are treated as free diagnostic experiments rather than as part of the total compute required to arrive at the H3 architecture.

The consequence. This is not merely a philosophical concern about research methodology. For a practitioner attempting to extend H3 to a new domain (e.g., code generation, multilingual modeling, mathematical reasoning), the paper provides no guidance on what new synthetic tasks would diagnose the missing primitives in that domain, nor any estimate of how much experimentation would be needed to identify and implement the right architectural modifications. The synthetic-to-real transfer worked once for English language modeling—but the paper provides no evidence that the same two primitives (token recall, token comparison) are the primary missing capabilities in other domains, or that the same methodology would yield similarly effective architectures elsewhere. A team attempting to apply this approach to, say, protein sequence modeling or music generation would need to rediscover the relevant primitives from scratch, with no guarantee of success. The development cost is effectively externalized from the paper's efficiency claims.

Furthermore, the synthetic tasks themselves are simplified to the point where their relationship to real language is questionable. Induction heads use a vocabulary of 20 tokens with a single special marker; associative recall uses 10 tokens with fixed key-value structure. These tasks isolate specific primitives, but they also strip away the complexity (nested dependencies, ambiguous references, long-range discourse structure) that makes language modeling hard. The paper shows that solving these simple tasks correlates with better PPL, but does not demonstrate that architectures good at these tasks will necessarily be good at the other operations language requires—which could be just as important but are not captured by the two chosen synthetics.

What evidence exists in the paper. The paper provides no accounting of the researcher time or compute spent on synthetic task development, architecture iteration, or hyperparameter search during the design process. The synthetic task results (Table 2) are presented as validation of the final architecture, not as part of a development loop whose cost should be considered. The paper does not discuss whether the chosen synthetic tasks are sufficient—or whether additional tasks probing other capabilities (e.g., long-range discourse coherence, numerical reasoning, multi-hop factual recall) would reveal further gaps in H3 that the current synthetics miss.

Mitigation status. Not addressed. The paper treats the synthetic-to-real methodology as a completed contribution rather than as a process whose cost and generalizability require analysis. There is no discussion of how to extend the methodology to new domains, how to determine which synthetic tasks are necessary and sufficient, or how much experimentation is typical when applying this approach.

---

6.4 Scaling Behavior Beyond 2.7B Parameters and 400B Tokens Is Unknown, and the Trend Suggests Diminishing Returns

The assumption or constraint. The paper's largest model is 2.7B parameters trained on 400B tokens. This is nearly two orders of magnitude smaller than frontier LLMs at the time of writing (~175B parameters, 1T+ tokens). The paper claims H3 "outperforms Transformers" across scales (Section 5.1), but the evidence for this claim is limited to the scales tested—and critically, the data shows that the perplexity advantage is shrinking as model size increases.

The consequence. If the trend of diminishing returns continues, H3 and Transformers would converge in perplexity at larger scales, or H3 could even underperform. The paper's own numbers show the gap narrowing:

• 125M: Hybrid H3 vs. GPT-Neo gap of 0.6 PPL (8.8 vs. 9.4)
• 1.3B: gap of 0.2 PPL (6.0 vs. 6.2)
• 2.7B: gap of 0.3 PPL (5.4 vs. 5.7)

The non-monotonicity at 2.7B (0.3 vs. 0.2 at 1.3B) makes extrapolation unreliable, but the overall direction is concerning. If the gap is truly closing, then at the 10B–100B scale where most production LLMs operate, H3 might offer no perplexity advantage at all—while still incurring the architectural complexity of a hybrid design and the systems complexity of FLASHCONV.

This matters enormously for a practitioner deciding whether to invest in H3-based training. Training a 100B+ parameter model costs millions of dollars; committing to a non-standard architecture with unproven scaling behavior is a major risk. The paper provides no scaling law analysis (à la Chinchilla) that would allow extrapolation of the H3-vs-Transformer gap to larger scales. The data scaling experiment (Table 13) shows a stable gap up to 15B tokens, but this is at 125M parameters—the gap at larger model sizes might evolve differently with data.

Additionally, the narrowing gap could reflect that Transformers at larger scales naturally develop better internal mechanisms for long-range dependencies—effectively learning within their quadratic attention matrix some of the structure that H3 explicitly encodes. If this is the case, H3's inductive bias becomes less valuable at scale, and the simpler Transformer architecture (with standardized, highly optimized implementations) would be the pragmatically better choice even if H3 maintains a small theoretical advantage.

What evidence exists in the paper. Table 4 provides perplexity at four model sizes, showing the shrinking gap. Table 13 provides data scaling at 125M, showing stable gap. No parametric scaling law is fit. The paper does not train models larger than 2.7B parameters. No prediction is made about performance at larger scales. The paper notes that "scaling SSMs to larger sizes is a promising avenue" (Section 7) but does not characterize the observed scaling trend or warn about extrapolation risk.

Mitigation status. The paper acknowledges in passing (Section 7) that scaling to larger sizes is future work, but does not highlight the diminishing-returns trend as a limitation. The headline claim that H3 "outperforms Transformers" is presented without the caveat that this has been demonstrated only up to 2.7B parameters and that the advantage is shrinking. A Chinchilla-style scaling law analysis is suggested as future work (Appendix A mentions understanding scaling behavior as an open direction) but not flagged as critical for validating the main claims.

---

6.5 Training Efficiency Claims Are Based on Micro-Benchmarks, Not End-to-End Training Throughput

The assumption or constraint. The paper claims FLASHCONV makes SSM training "4–8× faster than attention for long sequences" (Section 1) and reports 5.8× speedup on LRA (Table 8) and the near-linear scaling in Figure 2. However, these speedup numbers measure only the SSM convolution operation (FFTConv), not the end-to-end training throughput of a complete H3 model. The $O(d^2 N)$ cost of the linear projections ($\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$) and the MLP layers is shared between H3 and Transformers and is not accelerated by FLASHCONV. For the hybrid models that achieve the best results, the attention layers incur the same quadratic cost as standard Transformers. The paper never reports the wall-clock time to train a complete H3 or hybrid model end-to-end, compared to a Transformer trained with the same hardware and batch size.

The consequence. The 4–8× figure is misleading for a practitioner trying to estimate training costs. In a real training run, the FFT convolution is only one operation among many. The linear projections, MLP layers, attention layers, data loading, optimizer steps, and distributed communication all contribute to total training time. The paper's own inference benchmark (Table 7) shows that the hybrid model's end-to-end generation throughput advantage is only 1.5–2.4×—not 4–8×—because the attention layers and other components become bottlenecks at moderate sequence lengths. Training throughput is likely to show a similar pattern: significant speedup for the SSM layers, but a smaller overall speedup once all components are included.

Furthermore, the speedup numbers are measured on an A100 GPU with specific batch sizes and hidden dimensions (Figure 2 uses batch size 8, hidden dimension 1024). In distributed training across many GPUs, the communication patterns for SSM layers (which involve FFTs with global all-to-all-like permutation patterns in the Cooley-Tukey decomposition) may be quite different from those for attention (which involves all-to-all communication of keys and queries). The paper provides no analysis of how FLASHCONV scales in distributed settings—whether the fused kernel works efficiently with model parallelism, data parallelism, or sequence parallelism. A practitioner setting up a multi-node training run for a billion-parameter model needs to understand communication costs, not just single-GPU kernel speed.

What evidence exists in the paper. Figure 2 shows FFTConv operation speed, not end-to-end training throughput. Table 7 shows end-to-end inference throughput (1.5–2.4× speedup), not training throughput. Table 8 shows LRA total training time speedup (5.8×), but LRA uses small models where the convolution may be a larger fraction of total compute. No end-to-end training throughput comparison for language models at scale is provided. The paper does not discuss distributed training considerations.

Mitigation status. The paper presents the 4–8× figure prominently in the abstract and introduction without sufficient qualification. Section 6 and Figure 2 make clear that these are FFTConv benchmarks, but the distinction between operation speed and end-to-end training speed is not explicitly discussed. The inference benchmark (Table 7) partially addresses the end-to-end question but for generation only, not training. The distributed training question is entirely unaddressed.

---

6.6 The Hybrid Architecture's Placement of Attention Layers Is Unablated, and the Pure H3 Model Has Catastrophic Failure Modes on Specific Tasks

The assumption or constraint. The hybrid H3-attention model places its two attention layers at fixed positions: layer 2 and layer $N/2 + 2$) (using 1-indexed layer numbering as described in Section 5). The paper does not ablate this choice—it never tests 1, 3, or 4 attention layers, different layer positions, or attention layers concentrated at the bottom/top of the network. The implicit assumption is that these specific positions are either optimal or near-optimal, and that 2 attention layers strike the right balance between expressivity (more attention enables more token-level comparison) and efficiency (more attention increases quadratic cost).

Simultaneously, the pure H3 model exhibits erratic, catastrophic failures on specific SuperGLUE tasks—most notably MultiRC (4.6% zero-shot, 32.6% 3-shot in Table 14–15, compared to 58.9–59.9% for Transformer baselines) and ReCoRD (15.8% zero-shot, 15.8% 3-shot, compared to 39.6–55.0% for baselines). These are not small degradations; they represent near-complete inability to perform the task, while achieving state-of-the-art performance on WSC (61.5%, surpassing all baselines by >20 percentage points).

The consequence. For a practitioner, the unablated hybrid design means there is no way to know whether the chosen configuration is near-optimal or whether substantially better performance is achievable with a different attention/H3 ratio. If adding a third attention layer would recover the pure H3's MultiRC and ReCoRD failures while preserving inference speed, a practitioner would want to know. If the attention layers can be placed differently to achieve the same PPL with fewer attention heads, that changes the efficiency tradeoff.

More fundamentally, the pure H3 model's catastrophic failure modes suggest that H3 has specific, task-dependent blind spots that attention layers patch. But the paper provides no analysis of what about these tasks makes them hard for H3 and what about attention fixes them. MultiRC requires integrating evidence from multiple sentences to answer a question; ReCoRD requires identifying which entity in a long context fills a masked slot in a query. Both tasks demand precise token-level comparisons across long contexts—exactly the capability the paper claims H3's multiplicative interactions provide. The fact that H3 fails so dramatically on precisely these tasks, while excelling at WSC (which requires resolving pronoun references, a more local operation), suggests that the token comparison primitive implemented by H3 is qualitatively different from attention's token comparison in ways the synthetic tasks do not capture. H3's multiplicative interactions might work for the simple key-value lookups in the associative recall task but fail for the complex, ambiguous, multi-hop comparisons required by real reading comprehension.

What evidence exists in the paper. Tables 14–15 (Appendix F.6) show the catastrophic failures quantitatively. The paper does not ablate attention layer count or placement. The paper notes the MultiRC and ReCoRD failures briefly (attributing them to format issues in generation, Section F.7) but does not analyze why H3 fails on these tasks even in the rank classification setting (which does not involve generation formatting). The gap between H3's associative recall performance (99.8%) and its reading comprehension failures is not discussed.

Mitigation status. The generation format explanation (Section F.7) partially addresses the generation-based failures (Tables 16–17) but does not explain the rank classification failures (Tables 14–15), where the model only needs to score answer choices rather than produce formatted text. The unablated hybrid design is presented as a working configuration rather than as an open optimization problem. The paper does not discuss what task properties make H3 fail or whether additional attention layers would recover the lost performance.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper instigates a methodological shift in how the field approaches the design of efficient sequence models, moving from black-box architecture search and attention approximation toward mechanism-driven architecture synthesis. The central insight—that synthetic diagnostic tasks can identify missing computational primitives, which can then be implemented in a fundamentally different computational framework—reframes the relationship between mechanistic interpretability and architecture design.

Before this work, these two enterprises occupied largely separate intellectual spheres. Mechanistic interpretability (Elhage et al., 2021; Olsson et al., 2022) reverse-engineered trained Transformers to understand why they work, producing beautiful explanations of induction heads, copy suppression, and name-mover heads, but these insights rarely fed back into building new architectures. Architecture design—particularly for efficient attention alternatives like Performers, Reformers, and linear attention—operated largely through the lens of approximation theory: "attention is too expensive, so let's find a cheaper way to compute something similar." The underlying assumption was that attention's specific mechanism (softmax over dot products) was what made it effective, and therefore efficient alternatives should approximate that mechanism.

The paper inverts this logic with a concrete demonstration. Rather than approximating attention's mechanism, the authors ask: what specific operations does attention implement that SSMs lack? They answer with two diagnostics: token recall after events (induction heads) and token comparison across the sequence (associative recall). They then show that these operations can be implemented through entirely different machinery—a shift matrix for buffering, multiplicative interactions for gating, a diagonal SSM for accumulation—without any softmax, any quadratic comparison matrix, or any dot-product attention at all. The fact that H3 matches attention on the synthetics (Table 2: 100% vs. 100% on induction heads; 99.8% vs. 100% on associative recall) while looking nothing like attention internally is the crucial piece of evidence: the primitives matter, not the mechanism that implements them.

This reframing has two deep consequences for the field.

First, it converts the question "can SSMs match attention on language?" from an empirical arms race into a structured design problem. Prior work approached this question by training SSM variants on language benchmarks and hoping perplexity improved. This paper shows that the question decomposes into (a) identifying which primitives are missing, (b) implementing them in the SSM framework, and (c) verifying the implementation on diagnostics before scaling to language. This is analogous to how hardware design uses functional verification: you don't tape out a chip and hope it works; you verify that each subsystem implements the required logical operations before integration. The paper effectively provides the first functional verification suite for sequence model architectures, and shows that passing it correlates with real language modeling improvement (the 0.4 PPL gap in Table 3 vs. the 3.4+ PPL gap of architectures that fail the suite).

Second, it rescues SSMs from the position of being "efficient but weak" and positions them as genuinely competitive with attention—with asymptotic advantages that attention cannot match. Before this work, the dominant narrative was that SSMs offer elegant long-range modeling with $O(N \log N)$ scaling and constant-time inference, but at the cost of a significant perplexity gap that might be inherent. The paper demonstrates that the gap is not inherent to the SSM formalism—it was an artifact of missing primitives that can be supplied architecturally. This effectively removes the "yes, but they don't work well enough" objection that kept practitioners using attention despite its quadratic cost.

The paper also reconciles a tension in the literature between SSM successes on continuous-signal tasks (audio, time series) and failures on language. The synthetic tasks explain this discrepancy: continuous signals rarely require exact token recall or precise token comparison—they benefit from the smooth, compressed long-range memory that HiPPO-initialized SSMs provide natively. Language does require these discrete operations, which standard SSMs lack. By adding them via H3, the paper shows that SSMs can serve both types of tasks, opening the door to truly multimodal foundation models that handle continuous signals and discrete language within a unified architecture.

The FLASHCONV contribution also shifts the efficiency narrative: the hardware barriers to SSM training are surmountable with careful co-design. The state-passing algorithm is particularly significant because it establishes a new paradigm for computing SSMs—not as a single FFT or pure recurrence, but as a principled hybrid of both that is exactly equivalent to the mathematical definition (Proposition 2). This is qualitatively different from chunked attention approximations (Transformer-XL, Compressive Transformers), which discard information across chunk boundaries. State-passing loses no information because the SSM's hidden state is a sufficient statistic for the past. This makes SSMs not just asymptotically more efficient than attention in theory, but practically faster in wall-clock time for long sequences, as Figure 2 demonstrates.

Research directions that become more attractive:

• Mechanistic architecture design for other missing capabilities. The paper shows that you can diagnose a missing primitive (token comparison), design a component to supply it (multiplicative interactions), and verify it works (synthetic task). This template can be applied to other capabilities—multi-hop reasoning, factual recall, compositional generalization—by designing appropriate diagnostics and architectural interventions. The field now has a methodology, not just a point solution.
• Unified multimodal SSM architectures. Since H3 matches S4D on LRA (Table 9) while dramatically outperforming it on language, it demonstrates that language-optimized SSM design need not sacrifice continuous-signal performance. A single architecture handling text, audio, video, and sensor data becomes feasible.
• Hardware-software co-design for sequence models. FLASHCONV shows that classical signal processing algorithms (Cooley-Tukey FFT decomposition) combined with modern GPU hardware features (tensor cores, SRAM fusion) can dramatically change the practical efficiency landscape. This invites further exploration of what other classical algorithms can be similarly adapted—and what future hardware should support.

Research directions that become less attractive:

• Pure attention approximation approaches for language. Tables 10–11 show that Performer (26.8 PPL), Reformer (26.0 PPL), and Linear Attention (25.6 PPL) dramatically underperform both H3 (18.5 PPL) and the Transformer baseline (18.6 PPL) on WikiText-103. These 7–8 PPL gaps are larger than the gap this paper set out to close. If efficient alternatives that approximate softmax attention lose this much performance, while an SSM-based approach that implements the underlying primitives (without approximating attention) nearly matches exact attention, the approximation route becomes harder to justify for language modeling.
• Naively scaling SSMs without understanding failure modes. The paper shows that GSS (Mehta et al., 2022), despite being explicitly designed for language, fails dramatically on the synthetic diagnostics (Table 2: 6.8% induction heads, 78.0% associative recall) and underperforms even S4D on language PPL (Table 3: 24.0 vs. 24.9 for S4D). This suggests that architectural sophistication alone—without mechanistic validation—is unlikely to close the gap. Future SSM variants should be screened on the paper's synthetic tasks before being scaled.

Follow-Up Research This Work Enables

Ablating which H3 components are necessary for language modeling gains, and which are sufficient. The paper provides a constructive proof that shift SSM + diagonal SSM + multiplicative interactions can implement associative recall (Appendix D.1), and shows that H3 trains to solve the task (Table 2). But it never establishes that these specific components cause the language modeling improvement. A critical follow-up would train H3 variants on OpenWebText where: (a) the shift SSM is replaced with a diagonal SSM initialized identically to the existing diagonal SSM (removing the finite exact-memory buffer), (b) the multiplicative interactions are replaced with additive connections (removing the gating for token comparison), and (c) both changes are applied simultaneously. If language PPL degrades by amounts that correspond roughly to the synthetic task performance drops, this would establish the causal primitives-to-language link. If language PPL is unaffected despite synthetic task degradation, the architectural complexity would be unnecessary for language and the interpretation of H3's success would need revision. The experiment is straightforward—it requires retraining only the 125M H3 on OpenWebText with each modification, a few hundred GPU-hours—but it would transform the paper's correlational evidence into causal evidence.

Chinchilla-style scaling laws for H3 vs. Transformers to determine whether the advantage persists at frontier scales. The paper's own data (Table 4) shows the H3-vs-GPT-Neo perplexity gap narrowing from 0.6 PPL at 125M parameters to 0.2–0.3 PPL at 1.3B–2.7B. Whether this gap asymptotes, continues shrinking to zero, or even reverses at larger scales is the single most important open question for practitioners considering H3 adoption. A proper scaling law study would train H3, hybrid H3, and Transformer models at 5–7 model sizes spanning 10M to 10B parameters (or more), each trained on 4–5 data quantities (Chinchilla-style), and fit parametric curves predicting optimal perplexity as a function of compute. The key output would be: at 100B parameters and 1T tokens, does the hybrid H3 model have higher or lower perplexity than a Transformer trained with the same compute? If the lines cross, at what scale? This experiment is expensive (~hundreds of thousands of GPU-hours) but essential for derisking large-scale training investments.

Determining whether the hybrid attention-H3 ratio and placement can be optimized per-task or per-token. The paper uses a fixed hybrid design: 2 attention layers at specific positions (layers 2 and $N/2+2$). The pure H3 model's catastrophic failure on MultiRC (4.6% zero-shot, Table 14) and ReCoRD (15.8%)—tasks requiring integration of information across many sentences—suggests that attention layers are doing something specific that H3 layers cannot on these tasks. A natural follow-up would characterize which tokens, positions, or operations benefit from attention's precise quadratic comparison vs. H3's gated linear-time memory. One approach: train a fully-attention model, measure the attention entropy or attention head importance at each layer and position, and correlate this with where attention layers are most valuable in the hybrid. Another: use a learned router (mixture-of-experts style) that selects H3 vs. attention per-layer or per-token, and analyze which tokens get routed to attention—are they tokens at long-range dependency boundaries? Tokens involved in entity tracking? Tokens where the model needs to resolve ambiguity? This would transform the hybrid from a fixed architecture into an adaptive one, and the analysis would reveal whether H3's token comparison mechanism is fundamentally limited or merely needs more capacity (more heads, larger state size) on certain tasks.

Extending the synthetic-task methodology to diagnose and fix H3's reading comprehension failures. The paper's synthetic tasks isolate two primitives (token recall, token comparison) and H3 solves them. Yet H3-125M fails catastrophically on MultiRC and ReCoRD—precisely the tasks that require integrating evidence across multiple sentences. This suggests there is a third missing primitive that these tasks demand but the synthetics do not capture. Candidates include: (a) multi-hop comparison—comparing token A to B, and based on that result, comparing to C (cascaded comparisons), (b) soft key matching—associative recall requires exact key matches, but real reading comprehension requires matching semantically similar but lexically distinct keys (synonyms, paraphrases), (c) suppression of distractors—MultiRC includes distractor sentences that are relevant but don't support the correct answer; the model must not store or must later suppress these. A synthetic diagnostic could be designed for each candidate: e.g., a multi-hop associative recall task where the query key depends on the result of a prior key-value lookup, or a distractor-injected recall task where irrelevant key-value pairs are scattered among the relevant ones. Testing H3 on these new diagnostics would reveal which primitive is missing, and the architecture could be extended accordingly (e.g., stacking additional H3 layers with learned suppression, or adding an explicit two-stage comparison mechanism). A successful follow-up would show H3 recovering to near-Transformer performance on MultiRC and ReCoRD after the targeted architectural intervention.

Training a fully H3-based model (no attention) that matches the hybrid's language modeling performance through better optimization and hyperparameter tuning. The hybrid outperforms pure H3 (Table 3: 19.6 vs. 21.0 PPL), but the pure H3 model uses different head dimensions ($d_h=8$ vs. $d_h=1$) and was trained with Transformer-optimized hyperparameters without tuning. It is possible that a pure H3 model with appropriate hyperparameters (higher learning rate? different initialization? larger state size $m$? different head count?) could match or exceed the hybrid, eliminating the need for attention layers entirely and realizing the full efficiency benefits of the SSM formulation. A concrete experiment: train pure H3 models at 125M on OpenWebText with a hyperparameter sweep over state size $m \in \{16, 32, 64, 128, 256\}$, head dimension $d_h \in \{1, 2, 4, 8, 16\}$, and learning rate, using a budget comparable to the original experiment. If any configuration closes the 1.4 PPL gap to the hybrid, it would demonstrate that attention is not strictly necessary—it was merely compensating for suboptimal H3 configuration. If no configuration closes the gap, it would establish a fundamental limit on what H3's primitives can achieve without attention, and motivate investigation of what additional primitive attention provides.

Applying state-passing to distributed SSM training across multiple GPUs. The state-passing algorithm (Algorithm 2) has an elegant property: each chunk can be processed independently given only the end-state of the previous chunk, which is a vector of size $m$ (typically 64)—negligible communication cost. This suggests a natural sequence-parallel training strategy where different GPUs process different chunks of the sequence, passing only the $m$-dimensional state vector between them. The paper does not explore this, but it could dramatically reduce communication overhead compared to attention's sequence parallelism, which requires all-to-all communication of keys and queries. A concrete experiment: implement state-passing-based model parallelism for training a 1.3B H3 model on sequences of length 8192 across 8 GPUs, and compare training throughput to both (a) standard data-parallel H3 training and (b) sequence-parallel Transformer training at the same model size and sequence length. If the communication reduction translates to meaningful speedup, state-passing becomes a systems contribution as significant as FlashAttention—enabling long-sequence training that would otherwise be infeasible.

Practical Applications and Downstream Use Cases

Long-context document understanding and summarization. The $O(N \log N)$ training and $O(1)$-per-token inference of H3 makes it practical to process entire legal documents, scientific papers, or books as single sequences—something Transformers struggle with due to quadratic memory and KV-cache costs. A legal tech company processing 100-page contracts (roughly 20,000–30,000 tokens with typical subword tokenization) could train a hybrid H3 model on full documents without truncation, potentially capturing cross-references and definitions that a sliding-window Transformer would miss. The inference speedup (2.4× at prompt length 1536 in Table 7, likely larger at 20K tokens) translates directly to lower latency and cost per document. The pure H3 model, if its reading comprehension failures (Tables 14–15) can be addressed through finer tuning or task-specific prompting, would provide even larger speedups by eliminating the attention layers entirely.

On-device and edge deployment of language models. The recurrent formulation of SSMs enables constant-time-per-token generation with a fixed-size state (size $m=64$ per H3 layer, so $12 \times 64 = 768$ floats per layer for a 12-layer 125M model), compared to a Transformer's growing KV cache that reaches tens of thousands of vectors for long generations. This makes H3-based models attractive for mobile keyboards, voice assistants, and other on-device applications where memory is severely constrained and latency must be minimal. A 125M hybrid H3 model could run on a smartphone CPU with a fixed memory footprint regardless of conversation length, whereas a comparable Transformer would slow down and consume more memory as the conversation grows. The paper's Pile results (8.8 PPL for 125M hybrid H3, Table 4) suggest the quality is competitive with cloud-scale models from a few years prior (GPT-2 Small at 19.0 PPL, though not directly comparable), making on-device deployment viable for common use cases.

Real-time processing of biosignals and sensor data. The paper's non-text experiments demonstrate H3's effectiveness on sequences that are simply too long for Transformers: 12,000-time-step EEG (83.2 AUROC, Table 18) and 16,000-time-step raw audio (97.04% accuracy, Table 19). A seizure detection system processing continuous EEG streams from hospitalized patients could use an H3 model to flag abnormal segments in real time, with the model maintaining a rolling state that incorporates the full patient history without recomputation. The state-passing algorithm's chunked processing maps naturally to streaming data: as each new chunk of EEG samples arrives, the model updates its state and produces predictions, with latency determined by the chunk size (tens of milliseconds) rather than the total recording length (hours to days). This is a genuine deployment scenario where the Transformer baseline literally cannot run at all (marked 'x' in Table 18), making H3 not just better but the only feasible option.

Efficient inference for large-scale batch processing. Organizations running batch inference on large text corpora—e.g., classifying millions of support tickets, scoring generated text for quality filtering, or extracting entities from web-scale documents—face a tradeoff between model quality and throughput. The hybrid H3 model offers inference throughput 1.5–2.4× higher than an equivalently-sized Transformer (Table 7), with the gap widening for longer documents. For a batch job processing 100 million documents of average length 1500 tokens, this 2× speedup halves the GPU-hours required—from, say, 5,000 GPU-hours to 2,500. If the quality is equivalent or better (Tables 5–6 show competitive SuperGLUE performance), the cost savings are direct and the switching cost is low (the model is a drop-in replacement at inference time, assuming weights are available).

When to Prefer This Method

The paper provides enough evidence to establish clear preference criteria between pure Transformers, hybrid H3-attention, and pure H3 models:

• Prefer hybrid H3-attention over pure Transformers when: You are training a language model at moderate scale (125M–2.7B parameters) and care about a mix of perplexity, downstream task performance, and inference efficiency. The hybrid achieves lower perplexity than comparable Transformers (Tables 3–4) while offering 1.5–2.4× faster generation (Table 7). The training cost is similar (same hidden dimensions, same MLP sizes, same token budgets), so the inference speedup is essentially free. The 0.2–0.6 PPL advantage may narrow at larger scales, but within the tested regime, the hybrid dominates on both quality and speed metrics.

• Prefer pure H3 or H3-dominant hybrids when: You are processing very long sequences (8K+ tokens for training, or unbounded length for streaming inference) where Transformers become memory-bound or infeasible. The state-passing algorithm (Algorithm 2) enables training on arbitrary-length sequences with near-linear scaling, and the recurrent inference formulation provides constant-time-per-token generation. This is the regime of EEG, audio, long-document processing, and streaming sensor data—exactly where the paper shows Transformers either fail (Table 18) or are dramatically slower (Figure 2: ~20× gap at 32K tokens). The paper's non-text results (Tables 18–20) demonstrate that H3 does not sacrifice quality in these domains.

• Prefer pure Transformers over H3 variants when: You are training at scales far beyond those tested (10B+ parameters, 1T+ tokens) where H3's scaling behavior is unknown, or when your deployment involves primarily short sequences (<512 tokens) where attention's quadratic cost is negligible and the implementation maturity of Transformers (highly optimized kernels, widespread framework support, established training recipes) outweighs H3's efficiency advantages. The pure H3 model's catastrophic zero-shot failures on certain SuperGLUE tasks (Tables 14–15) also recommend caution—if your application requires reliable zero-shot classification without few-shot prompting, the hybrid or a pure Transformer may be safer until those failure modes are understood and addressed.

• Prefer existing SSMs (S4/S4D) over H3 when: Your task is purely continuous-signal modeling (audio generation, time series forecasting) with no discrete language component. H3 matches S4D on LRA accuracy (Table 9: 84.8% vs. 85.0%) but is architecturally more complex, and the shift SSM's finite memory buffer provides no benefit for signals where exact token recall is irrelevant. S4/D's simpler structure and established training recipes make it the pragmatically better choice for non-language domains until H3 demonstrates a clear advantage there.