Hyena Hierarchy: Towards Larger Convolutional Language Models

ArXiv: 2302.10866

🎯 Pitch

Attention-free convolutional models can match Transformers in language modeling quality while being 100× faster at sequence length 64K. The key is a design that combines data-controlled gating with implicit long convolutions parameterized by a neural network, rather than explicit filters or state-space models. This subquadratic operator even surpasses attention on associative recall tasks at hundreds of thousands of tokens, challenging the assumption that attention is necessary for long-range reasoning.

---

1. Executive Summary

This paper introduces Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. Evaluated on mechanistic interpretability benchmarks (associative recall, induction), autoregressive language modeling (WIKITEXT103, THE PILE), and image classification (ImageNet-1k), Hyena operators match Transformer quality with a 20% reduction in training FLOPs at sequence length 2K and achieve 100× speedups over highly optimized attention at sequence length 64K. The design is guided by three properties distilled from attention—data control (encoding an entire family of linear functions via input-conditioned diagonal matrices), sublinear parameter scaling (filter parameters decoupled from sequence length via implicit FFN-based parametrization), and unrestricted context (long convolutions with filters as long as the input)—establishing that attention-free convolutional architectures can match attention at scale, but only when long convolution parametrization uses implicit methods with exponential decay modulation rather than explicit FIR filters or state-space models.

2. Context and Motivation

The Core Problem: Attention's Quadratic Cost Is a Hard Ceiling on Context Length

The fundamental problem this paper tackles is architectural rather than algorithmic: the dominant operator in modern deep learning—self-attention—scales quadratically with sequence length, creating a strict computational boundary on how much context a model can process. For a sequence of length $L$, the self-attention mechanism computes an $L \times L$ matrix of pairwise interaction scores, requiring $\mathcal{O}(L^2)$ operations in both time and memory. As the authors note in Section 1:

"breaking the quadratic barrier is a key step towards new possibilities for deep learning, such as using entire textbooks as context, generating long-form music or processing gigapixel scale images."

This is not a marginal efficiency concern—it is a qualitative limitation on what models can do. Transformers simply cannot process sequences beyond a certain length without running out of memory, regardless of how much hardware is available. At the time of writing, state-of-the-art models typically operate on context windows of 2K–8K tokens, which is insufficient for applications requiring reasoning over entire books, long scientific papers, extended conversations, or high-resolution biological sequences (e.g., whole-genome analysis).

The problem is more subtle than it first appears. Attention's quadratic cost is not an implementation detail that can be optimized away with better engineering—it is a fundamental property of the operator's design. For every pair of positions $(i, j)$ in the sequence, attention computes a dot product between the query at position $i$ and the key at position $j$, followed by a softmax normalization over all $j$. There is no way to avoid computing every pairwise interaction without changing the nature of the operator itself. This means that any attempt to extend Transformers to longer sequences must replace or modify the attention mechanism, not merely accelerate it.

Why This Matters: The Scaling Landscape and the Need for Alternatives

The urgency of solving this problem is magnified by the scaling laws literature (Hoffmann et al., 2022), which has established that larger models trained on more data consistently improve—but also that training costs grow rapidly. If the bottleneck shifts from model size to context length, a different kind of scaling problem emerges: even if you can afford to train a massive model, you may not be able to feed it the data it needs to solve long-range reasoning tasks. The paper frames this explicitly in terms of emerging capabilities:

"Much of the success of Transformers, powered by the attention operator, relies on their scaling properties and the emergence of in-context learning, which allows them to generalize to unseen data and tasks given context as input."

In-context learning—the ability of a model to "learn" a new task from examples provided in the prompt without weight updates—is one of the most transformative properties of large language models. But in-context learning is only useful if you can fit the examples in context. A model that can only see 2K tokens of context is fundamentally limited in how many demonstrations it can absorb, how long a document it can summarize, or how complex a reasoning chain it can follow. Breaking the quadratic barrier is thus a prerequisite for unlocking the full potential of in-context learning.

The paper identifies a range of high-impact applications that are currently bottlenecked by context length: processing entire textbooks or legal documents, generating coherent long-form music or video, analyzing gigapixel-scale medical or satellite images, and modeling long-range dependencies in scientific data (e.g., protein sequences, genomic data). In all these domains, the information that determines the correct output may be separated from the current position by thousands or millions of tokens, and any operator that cannot efficiently bridge that gap will fail regardless of how many parameters it has.

The Landscape of Prior Approaches: A Taxonomy of Shortcomings

Before Hyena, a substantial body of work had attempted to reduce attention's cost, falling into several distinct families. The paper provides a concise survey in Section 1 and Section 2.2, and understanding the shortcomings of each approach is essential to appreciating why Hyena's design choices matter.

Linearized, Low-Rank, and Sparse Approximations

The first major family replaces the full $L \times L$ attention matrix with an approximation that can be computed in $\mathcal{O}(L)$ or $\mathcal{O}(L \log L)$ time. Representative methods include:

• Linformer (Wang et al., 2020): Projects keys and values to a fixed lower dimension, making the attention matrix low-rank. This reduces complexity from $\mathcal{O}(L^2)$ to $\mathcal{O}(Lk)$ where $k$ is the projection dimension. The tradeoff: the low-rank constraint limits expressivity, and for tasks requiring precise long-range retrieval, a fixed-rank approximation may miss fine-grained dependencies.

• Reformer (Kitaev et al., 2020): Uses locality-sensitive hashing (LSH) to group similar queries together, so attention is only computed within each bucket rather than across all positions. This sparsifies the attention matrix. The tradeoff: hashing introduces randomness, some pairs that should attend may be hashed into different buckets, and the method requires careful tuning of the hashing parameters.

• Sparse Transformers (Child et al., 2019): Uses fixed sparse patterns (strided and local attention) rather than computing the full matrix. The tradeoff: the sparsity pattern is hand-designed and does not adapt to the content, meaning some important long-range interactions may be missed if they don't fall within the predefined pattern.

The critical finding from this body of work is summarized in the paper:

"These approaches introduce a trade-off between expressivity and speed, requiring hybridization with standard attention layers to reach Transformer quality."

In other words, none of these approximate attention methods could fully replace attention. They all had to be combined with some standard dense attention layers to match Transformer performance (a strategy called "hybridization," used in H3 and other works). This is a telling signal: the approximations were losing something important—something that made dense attention necessary as a supplement. The paper refers to this as a "gap in capability" and makes it the central research question.

State-Space Models (SSMs) and Long Convolutions

A second family of approaches, developed contemporaneously with the sparse attention literature, uses completely different operators rather than approximating attention. State-space models (SSMs) are the most prominent example:

• S4 (Gu et al., 2021): Models sequences using structured state-space equations with diagonal plus low-rank state matrices. The input-output map can be expressed as a long convolution with a filter implicitly parametrized by the state-space parameters. S4 achieved strong results on long-range sequence modeling benchmarks like Long Range Arena, but when applied to language modeling, performance lagged behind Transformers.

• Gated State Spaces (GSS) (Mehta et al., 2022): Added multiplicative gating (element-wise multiplication) to SSM-based convolutions, creating a data-controlled variant. This improved performance but still fell short of attention on associative recall—a task the mechanistic interpretability literature had identified as fundamental to in-context learning.

• H3 (Dao et al., 2022c): Extended GSS by adding an additional gating interaction and a short "shift" convolution, creating a decomposition $\text{H3}(q, k, v) = D_q S_\psi D_k S_\varphi v$. H3 was the strongest attention-free model at the time and was explicitly designed to close the gap on associative recall—but even H3 required hybridization with attention layers to match full Transformer quality on language modeling.

The key insight from this line of work is that data control matters critically. The difference between a static convolution and an SSM-based convolution is that the SSM can learn different filters—but once trained, the filter is fixed for all inputs. GSS and H3 introduced the idea of conditioning the convolution on the input through multiplicative gating, making the operation data-dependent. But the specific mechanism (two gating interactions with SSM-parametrized filters) was still insufficient to fully close the gap with attention.

Attention-Free Transformers and Other Subquadratic Operators

A third group includes methods that completely restructure the computation:

• Attention-Free Transformer (AFT) (Zhai et al., 2021): Replaces the query-key-value interaction of attention with a procedure that applies a learned positional bias (or a learned convolution) after gating. AFT-conv uses an explicit convolution filter of limited size. The paper's experiments show AFT failing on long-range associative recall (Table 4.2: 2.3% accuracy at 30K length, 1.2% at 64K), indicating that a single short explicit convolution cannot capture the long-range dependencies needed.

• RWKV (Peng, 2021): Uses a recurrent formulation with time-mixing and channel-mixing blocks, effectively implementing a type of linear attention with exponential decay. The paper shows RWKV modestly outperforming AFT on long-range recall but still falling far short of Hyena and Transformers (Table 4.2: 12.4% at 30K, 6.5% at 64K, 2.3% at 131K).

• Linear Attention / Performer: Uses kernel approximations (random features) to linearize the softmax attention computation. The paper includes Performer in Table 4.3 (perplexity 26.8 on WIKITEXT103, compared to 18.6 for Hyena and Transformers), showing a substantial quality gap.

The consistent finding across all three families is that no pure attention-free architecture had matched Transformer quality on language modeling at scale. Every approach either required hybridization with dense attention layers or showed a clear quality gap on perplexity and downstream tasks.

The Mechanistic Interpretability Connection: Why Recall Matters

The paper's motivation is deeply informed by the mechanistic interpretability literature—particularly work on "induction heads" and "circuits" in Transformers (Elhage et al., 2021; Olsson et al., 2022). These studies reverse-engineered what specific attention heads in trained Transformers are doing, and found that a critical operation is associative recall: given a key, retrieve the associated value from earlier in the sequence. For example, if the sequence contains "The capital of France is Paris," an induction head learns to attend from the token following "France" to "Paris" in order to predict the next token.

This is not a niche capability—it is the computational primitive underlying in-context learning. When a Transformer is given a few examples of a new task in its prompt, it uses induction heads to attend from the query (the new example) to the relevant information in the prior examples. Models that cannot perform associative recall cannot learn in-context, regardless of their size.

Previous work had found that SSM-based models (including H3) struggled with associative recall because the state-space representation has difficulty preserving exact token identities over long distances—the continuous-time dynamics of SSMs naturally "smooth" information, which is beneficial for modeling physical processes but detrimental for discrete symbol manipulation. The paper takes this finding as foundational:

"a growing amount of evidence suggests that attention mechanisms only utilize a small portion of their quadratic capabilities for language processing, leading us to question its role as the gold-standard operator for deep learning at scale."

The implication is profound: if attention is mostly just doing associative recall (which can be implemented with a much simpler mechanism) and a few other operations, then the full $\mathcal{O}(L^2)$ attention matrix is wildly overparameterized for the job. The challenge is to identify the minimal set of primitives that can perform these critical operations while scaling subquadratically.

Three Guiding Properties

The paper distills the insights from mechanistic interpretability and the failures of prior approaches into three properties that any successful attention replacement must satisfy (Section 1):

a. Data control: The operator must be nonlinearly conditioned on the input—it cannot be a fixed linear transformation. Attention implements this by computing queries and keys from the input, which then define the attention matrix: $y = A(q(u), k(u)) v(u)$. A fixed convolution (even a long one) fails this property because it applies the same filter regardless of input content. GSS and H3 partially satisfy this through gating, but the paper hypothesizes their specific implementations are insufficient.

b. Sublinear parameter scaling: The number of parameters in the operator must not grow proportionally with sequence length. A standard CNN with filter size equal to the sequence length would require $\mathcal{O}(L)$ parameters, which is impractical for very long sequences. Attention satisfies this because the $Q, K, V$ projection matrices are fixed-size regardless of $L$. This property is what makes implicit convolution parametrizations (FFNs, SSMs) superior to explicit ones—they decouple filter expressivity from parameter count.

c. Unrestricted context: The operator must be able to model dependencies between any two positions in the sequence, without artificial restrictions like locality windows or fixed sparsity patterns. Attention satisfies this in the most extreme way possible (every position attends to every other position), but the paper hypothesizes that the same effect can be achieved with long convolutions that have filters as long as the input, provided the filters are properly designed. Local convolutions (FIR filters with $M \ll L$) fail this property.

How Hyena Positions Itself

The paper frames Hyena not as a single new algorithm but as a unifying framework that generalizes existing attention-free operators through a recurrence of two primitives: long convolutions and multiplicative gating. The key insight in positioning is:

"Instead of seeking an approximation to attention, we guide our design by intentionally incorporating key computational properties of attention."

This is a departure from how most prior work positioned itself. Rather than asking "how can we approximate the attention matrix with fewer operations?", the paper asks "what are the essential computational properties of attention that make it work for language, and can we realize those properties through fundamentally different mechanisms?" The answer, they propose, is that interleaving long convolutions and gating in a recurrence of sufficient depth can match attention's capabilities while remaining subquadratic.

The paper explicitly connects Hyena to the prior hierarchy of models: Hyena$_1$ recovers GSS, Hyena$_2$ recovers H3, and higher orders go beyond both. This is not just taxonomic; it implies that the previous models were underparameterized instances of a more general class, and that increasing the order (depth of the recurrence) is a principled way to increase expressivity without sacrificing efficiency.

A critical empirical claim motivates the specific design choices within Hyena: the parametrization of the long convolution matters enormously. Figure 4.1 shows that at vocabulary size 40 and sequence length 2048, explicit convolutions (Conv1d) achieve only ~8% accuracy on associative recall, while Hyena's implicit FFN-based parametrization with exponential decay achieves ~85%. This is a 77-point gap from the choice of convolution parametrization alone—even though both methods are embedded in the same Hyena$_2$ recurrence framework. The paper argues that prior work using SSMs or explicit filters was held back not just by the gating mechanism but by suboptimal filter design, and that the combination of implicit FFN parametrization, exponential decay windowing, and high-frequency activation functions is what finally closes the gap.

3. Technical Approach

This is primarily a systems and architecture design paper whose core idea is that the essential computational properties of attention—data control, sublinear parameter scaling, and unrestricted context—can be realized through a carefully designed recurrence of long convolutions and multiplicative gating, without any approximation of attention itself.

3.1 Reader Orientation

Hyena is a drop-in replacement operator for self-attention in Transformer architectures. It takes a sequence of vectors as input (same as attention would) and outputs a transformed sequence of vectors (same as attention would), but it does so using subquadratic operations based on long convolutions and element-wise gating rather than pairwise dot-product interactions. The system is a parameterized machine for constructing a data-controlled linear transformation—the transformation applied to the input is different for each input sequence because the coefficients of the transformation depend on projections of the input itself. The solution has the shape of a recurrence: repeatedly apply a learnable long convolution filter to the sequence, then multiply element-wise by a gating signal derived from the input, with the depth of the recurrence (the order $N$) controlling expressivity.

3.2 Big-Picture Architecture (Diagram in Words)

The Hyena system has four major components connected in a specific pipeline:

1. Input Projection Module: Takes the raw input sequence $u \in \mathbb{R}^{L \times D}$ (length $L$, width $D$) and linearly projects it into $N+1$ separate sequences: one value sequence $v$ and $N$ gating signals $x^1, \dots, x^N$. A short depthwise convolution is applied to each projected sequence to introduce local structure before the main operator.

2. Hyena Filter Generator: Takes a positional encoding of the time indices $t = 0, \dots, L-1$ and passes it through a shallow feed-forward neural network (FFN) with sinusoidal activations, then multiplies by a learnable window function. This generates $N$ implicit long convolution filters $h^1, \dots, h^N$, each of length $L$. The parameters of the FFN are the only parameters that define these filters, so the number of parameters is independent of sequence length.

3. Recurrent Gating-and-Convolution Core: The main computational loop iterates $n = 1$ to $N$: at each step, convolve the current value sequence with filter $h^n$ using FFT-based fast convolution, then multiply element-wise by gate $x^n$. The first iteration uses $v$ as the initial signal; each subsequent iteration feeds the result of the previous iteration into the next convolution.

4. FFTConvolution Engine: A fast convolution implementation that performs long convolutions in $\mathcal{O}(L \log L)$ time using the convolution theorem: pad input and filter to length $2L-1$, compute their FFTs, multiply pointwise, then inverse FFT. This engine is shared across all $N$ iterations.

Information flows as follows: input sequence $u$ → projection into $(v, x^1, \dots, x^N)$ → position indices → FFN generates filters $(h^1, \dots, h^N)$ → for $n = 1$: convolve $v$ with $h^1$, multiply by $x^1$, producing $z^2$ → for $n = 2$: convolve $z^2$ with $h^2$, multiply by $x^2$, producing $z^3$ → … → final output $y = z^{N+1}$.

3.3 Roadmap for the Deep Dive

• First, the formal definition of the Hyena recurrence (Definition 3.1), which establishes the mathematical structure and computational model, and its matrix form (Section 3.2) which reveals Hyena as a factorization of a data-controlled matrix—the key connection to attention.
• Second, the Hyena filter parametrization (Section 3.3), which is the design choice that empirically distinguishes Hyena from prior methods: implicit FFN-based long convolution filters with exponential decay modulation and sinusoidal activations. Understanding this choice requires examining the alternatives (explicit FIR, SSM-based, frequency-domain) and why they fail on associative recall at scale.
• Third, the fast evaluation algorithm (Section 3.4), which translates the abstract recurrence into concrete GPU operations using FFT-based convolution, making clear how the $\mathcal{O}(N L \log L)$ complexity is achieved in practice.
• Fourth, the relationship to prior operators: how Hyena$_1$ recovers Gated State Spaces, Hyena$_2$ recovers H3, and higher orders go beyond both—explaining why depth matters as a principled expressivity knob rather than an arbitrary tuning parameter.
• Fifth, the connection to butterfly matrix decompositions, which provides theoretical motivation for why interleaving diagonal (gating) and Toeplitz (convolution) matrices in a recurrence produces an expressive family of structured matrices.

3.4 Detailed, Sentence-Based Technical Breakdown

The Hyena Recurrence: Formal Definition

At the core of Hyena is a recurrence that interleaves two operations—long convolution and element-wise gating—to transform an input sequence. The formal definition establishes the precise computational model:

Definition 3.1 (Order-$N$ Hyena Operator). Let $(v, x^1, \dots, x^N)$ be projections of the input and let $h^1, \dots, h^N$ be a set of learnable filters. The Hyena$_N$ operator is defined by the recurrence:

$z_t^1 = v_t$

$z_t^{n+1} = x_t^n (h^n * z^n)_t \quad n = 1, \dots, N$

$y_t = z_t^{N+1}$

where $v_t$ is the value projection at position $t$, $x_t^n$ is the $n$-th gating signal at position $t$, $h^n$ is the $n$-th long convolution filter, $*$ denotes linear convolution, and $y_t$ is the output at position $t$.

What it computes: Starting from the value sequence $z^1 = v$, the recurrence performs $N$ steps. At each step $n$, it first convolves the current value sequence $z^n$ with the learnable filter $h^n$, producing a filtered sequence $(h^n * z^n)$. It then multiplies this filtered sequence element-wise by the gating signal $x^n$, producing the sequence $z^{n+1}$ for the next iteration. After $N$ steps, the output $y = z^{N+1}$ is the final transformed sequence. Operationally, the convolution spreads information across time (mixing information from different positions according to the filter's shape), and the gating selectively amplifies or suppresses the mixed signal at each position based on the input content.

Why this form: The alternation of convolution (time-domain mixing) and gating (content-based scaling) mirrors the commutativity structure of attention. In attention, the operator applies a data-dependent matrix $A(q, k)$ to a value vector $v$—the matrix is built from pairwise interactions, but its effect is to route information from all positions to each output position in a content-dependent way. Hyena achieves a similar routing effect through a different structural decomposition: convolutions provide the long-range mixing (like the dense connectivity of attention), and gating provides the input-dependent modulation (like the query-key interaction determining which positions matter). The depth $N$ controls how many alternating mixing-modulation steps are applied, with higher $N$ corresponding to more complex routing patterns. The paper shows that $N=2$ (matching H3) is sufficient for many language tasks, but the framework supports arbitrary depth as an expressivity knob.

The Matrix Form: Hyena as Data-Controlled Factorization

Understanding Hyena as a matrix transformation reveals its connection to attention and its computational properties. For a sequence of length $L$, any linear transformation $v \mapsto y$ can be written as a matrix-vector product $y = H v$ where $H \in \mathbb{R}^{L \times L}$. The key insight is that in Hyena, this matrix is data-controlled—its entries depend on the input $u$—but it is never materialized. Instead, it is implicitly defined through a factorization into alternating diagonal and Toeplitz matrices.

Let $D_{x^n} = \text{diag}(x^n) \in \mathbb{R}^{L \times L}$ be the diagonal matrix whose diagonal entries are the gating signals $x^n_0, \dots, x^n_{L-1}$. Let $S_{h^n} \in \mathbb{R}^{L \times L}$ be the Toeplitz matrix induced by filter $h^n$:

$S_{h^n} = \begin{bmatrix} h^n_0 & h^n_{-1} & \cdots & h^n_{-L+1} \\ h^n_1 & h^n_0 & \cdots & h^n_{-L+2} \\ \vdots & \vdots & \ddots & \vdots \\ h^n_{L-1} & h^n_{L-2} & \cdots & h^n_0 \end{bmatrix}$

where $h^n_t$ is the filter response at offset $t$, with $h^n_t = 0$ for $t < 0$ when the filter is causal (lower-triangular case).

The Hyena recurrence then has the matrix form:

$y = H(u)v = D_{x^N} S_{h^N} \cdots D_{x^2} S_{h^2} D_{x^1} S_{h^1} v$

where $H(u)$ is the data-controlled matrix, $D_{x^n}$ are diagonal matrices encoding the gating signals (derived from the input via projections), $S_{h^n}$ are Toeplitz matrices encoding the long convolutions (learned parameters, shared across inputs), and the product is read right-to-left as the composition: first convolve by $h^1$, then gate by $x^1$, then convolve by $h^2$, etc.

What it computes: The same transformation as the recurrence, but expressed as a decomposition of the effective transformation matrix. Each factor $D_{x^n} S_{h^n}$ is a data-controlled matrix: $S_{h^n}$ is a learned Toeplitz matrix (a convolution), and $D_{x^n}$ is a diagonal matrix whose entries come from the input. The product of $N$ such alternating factors is a structured dense matrix that can represent complex input-dependent transformations, but the factorization structure means it can be applied to a vector in subquadratic time by sequential convolution and gating operations.

Why this form: The decomposition into alternating diagonal and Toeplitz factors is a special case of a butterfly decomposition—a structure used in fast matrix-vector multiplication algorithms (Li et al., 2015; Dao et al., 2019). In butterfly factorizations, a dense matrix is expressed as a product of sparse structured factors, each of which can be applied quickly, enabling fast multiplication without materializing the full matrix. Hyena adapts this idea by making the diagonal factors data-controlled (they depend on $u$) while keeping the Toeplitz factors as learned parameters. This hybrid of data control and structured sparsity is what gives Hyena both expressivity (through input-dependent routing) and efficiency (through convolution's $\mathcal{O}(L \log L)$ cost). The specific structure also clarifies how Hyena$_2$ recovers H3: $H(u) = D_q S_\psi D_k S_\varphi$, which is exactly the H3 surrogate attention matrix with $x^1 = k$, $h^1 = \varphi$, $x^2 = q$, $h^2 = \psi$.

A critical property visible in the matrix form is that the matrix $H(u)$ has unrestricted context: every entry $H(u)_{ij}$ can potentially be non-zero because the Toeplitz matrices $S_{h^n}$ have non-zero entries for all $i \geq j$ (for causal filters) or all $i, j$ (for non-causal). There is no artificial restriction to a local window or fixed sparsity pattern—the model can learn filters that couple position $i$ to any position $j$ in the sequence. This is one of the three guiding properties the paper identifies as essential.

Implicit Filter Parametrization: The FFN Approach

The choice of how to parametrize the long convolution filters $h^n$ is empirically the most consequential design decision in Hyena. The paper's experiments (Figure 4.1, Table A.2) show a 77-point accuracy gap on associative recall between the best and worst parametrization choice, even though all parametrizations are embedded in the same Hyena$_2$ recurrence. Understanding why requires examining what each alternative provides and what it lacks.

Explicit (FIR) parametrization (Conv1d): In standard CNNs, a convolution filter is parametrized by directly storing the values at each offset: $h_0, h_1, \dots, h_{M-1}$ for a filter of size $M$. This has two critical limitations for long sequences. First, the number of parameters scales linearly with $M$: to capture dependencies across the full sequence ($M = L$), you need $L$ parameters per filter, which is prohibitive for very long sequences and violates the sublinear parameter scaling property. Second, learning explicit filter values independently makes it difficult for the model to discover structured patterns (e.g., exponential decay, periodic components) because it must learn each filter tap as a separate, unrelated parameter. The paper's experiments show Conv1d achieving only 8% accuracy on associative recall with vocabulary size 40 at sequence length 2048.

State-space model parametrization (H3/S4): In SSMs, the filter $h_t$ is defined as the impulse response of a linear dynamical system: $h_t = C A^t B$ for $t \geq 0$, where $A, B, C$ are learned matrices. The number of parameters depends on the state dimension of $A$, not the sequence length, satisfying sublinear parameter scaling. However, SSM filters are constrained to be sums of exponentials (if $A$ is diagonal) or more generally linear combinations of basis functions determined by the eigenvalues of $A$. This parametric form is excellent for modeling smooth, continuous dynamics (the original SSM motivation), but can struggle with the discrete, exact-matching operations needed for associative recall—the SSM naturally "blurs" information across time due to the continuous-time dynamics underlying the parametrization. The paper's experiments show H3 achieving 13% accuracy under the same 2048-length, vocabulary-40 condition.

Frequency-domain parametrization (FNO): The Fourier Neural Operator parametrizes filters directly in the frequency domain by storing a fixed number of Fourier modes. This decouples filter expressivity from length, but the constraint to a fixed number of modes limits the filter's ability to represent sharp transitions or exact delays (which require high-frequency content across many modes). FNO achieves 10% accuracy in the paper's associative recall experiments.

Hyena's implicit FFN parametrization: The key innovation is to parametrize the filter as a function of time:

$h_t = \text{Window}(t) \cdot (\text{FFN} \circ \text{PositionalEncoding})(t)$

where $t = 0, \dots, L-1$ is the time index, $\text{PositionalEncoding}$ maps the scalar index to a higher-dimensional embedding (enabling the representation of high-frequency functions), $\text{FFN}$ is a shallow feed-forward neural network (a small MLP mapping the embedding to a scalar filter value), and $\text{Window}(t)$ is a learnable multiplicative envelope applied to the filter output.

Positional encoding: The scalar time index $t$ is first mapped to a positional embedding $e_t \in \mathbb{R}^{D_e}$ where $D_e$ is the embedding dimension. The paper uses sinusoidal positional encodings (similar to the Transformer's original design) but importantly uses sinusoidal activations in the FFN itself rather than just in the initial encoding. This is a distinction from prior implicit convolution work (e.g., CKConv by Romero et al., 2021b) that uses standard ReLU or similar activations.

FFN structure: The paper uses a "shallow" FFN with depth 4 and width 64 for most language experiments (Table A.4). The architecture is:

$\text{FFN}(e_t) = W_4 \cdot \sin(W_3 \cdot \sin(W_2 \cdot \sin(W_1 e_t + b_1) + b_2) + b_3) + b_4$

where $\sin$ is applied element-wise as the activation function. The use of sinusoidal activations is critical: neural networks with standard activations (ReLU, tanh) exhibit a bias toward low-frequency functions (Basri et al., 2020), meaning they struggle to represent filters with sharp transitions or high-frequency oscillations. Sinusoidal activations, by contrast, allow the network to represent high-frequency content through the periodic structure of the activation function itself, even in early layers. The paper explicitly states this:

"We use high-frequency periodic activations (sine) in the FFN. This allows (7) to learn filters with high-frequency content, addressing the low-frequency bias of neural networks."

Window function: The window is a learnable multiplicative envelope applied to the raw FFN output. The paper highlights exponential decay as a particularly important window shape:

"we choose at least one of the convolutions in Hyena to be shaped towards exponential decay, mirroring the findings of (Li et al., 2022) in other applications."

A learnable exponential decay window takes the form $\text{Window}(t) = \exp\{-\alpha t\}$ where $\alpha$ is a learnable parameter (or is applied per-channel with different learned $\alpha$ values). This biases the filter toward longer-range influence (slow decay) or shorter-range influence (fast decay), and when combined with the learnable bias term mentioned in the paper:

"we add a bias term to our window, so that the filters are not constrained to be zeros after a length determined by the decay rate."

This bias ensures the filter doesn't decay exactly to zero, preserving the ability to represent long-range dependencies even with aggressive decay rates.

Why the FFN approach works better: The FFN parametrization satisfies sublinear parameter scaling (the FFN has fixed width and depth regardless of $L$), can represent arbitrary filter shapes (the FFN is a universal function approximator on the domain $[0, L-1]$, constrained only by the capacity of the network), and avoids the inductive bias toward smoothness that limits SSM-based filters. The sinusoidal activations address the low-frequency bias directly, enabling the representation of the sharp, discrete-matching operations needed for associative recall. Empirically, the combination of implicit FFN parametrization, exponential decay windowing, and sinusoidal activations is what enables Hyena to achieve 85% accuracy on the vocabulary-40 associative recall task—a 72-point improvement over H3 and a 77-point improvement over Conv1d, controlling for the rest of the architecture.

Cross-channel parametrization: The paper notes that the FFN generates filters for all $N$ orders and all $D$ channels simultaneously (Algorithm 2): the FFN output is reshaped to $N \times D \times L$, meaning different channels and different orders share the same FFN architecture but learn different filters through the learned weight matrices. This is efficient because it amortizes the FFN computation across all channels, rather than requiring a separate FFN per channel or per order.

Fast Convolution via FFT

The Hyena recurrence requires computing $N$ long convolutions, each of which is a linear convolution of a filter of length $L$ with a signal of length $L$. A naive direct computation of each convolution would cost $\mathcal{O}(L^2)$, defeating the purpose of replacing attention. The paper uses the convolution theorem to perform each convolution in $\mathcal{O}(L \log L)$ time via the Fast Fourier Transform (FFT).

The convolution theorem: For a linear (aperiodic) convolution $y = h * u$ of two length-$L$ sequences, the result has length $2L-1$ (due to the full overlap range). The theorem states that the convolution can be computed as:

$y = \text{iFFT}(\text{FFT}(\text{pad}(h)) \odot \text{FFT}(\text{pad}(u)))$

where $\text{pad}$ zero-pads both sequences to length $2L-1$, $\text{FFT}$ computes the discrete Fourier transform, $\odot$ denotes element-wise multiplication, and $\text{iFFT}$ computes the inverse transform. The computational cost is dominated by the FFT operations, each costing $\mathcal{O}((2L-1) \log(2L-1)) = \mathcal{O}(L \log L)$.

Why this form: The FFT convolution is asymptotically faster than direct computation for any filter length greater than the crossover point (approximately 32–128, depending on implementation). For long convolutions where the filter length equals the sequence length ($L$ can be thousands or hundreds of thousands), the speedup is dramatic—from $\mathcal{O}(L^2)$ to $\mathcal{O}(L \log L)$. This transformation is only possible because the convolution operator is diagonalized by the Fourier basis: in the frequency domain, convolution becomes pointwise multiplication, which is trivially parallelizable. The key structural property enabling this is that the Toeplitz matrix $S_h$ is diagonalized by the DFT matrix, as shown in Section 2.1:

$\hat{S}_h = W^{-1} D_H W$

where $W$ is the DFT matrix and $D_H$ is diagonal with the DFT of $h$ on its diagonal. The paper leverages a fused CUDA kernel to perform FFTConv efficiently on GPU, inheriting the implementation from H3 (Dao et al., 2022c).

Causality handling: For autoregressive language modeling, the operator must be causal—the output at position $t$ must depend only on inputs at positions $\leq t$. Proposition 3.1 states that if each filter $h^n$ is causal (i.e., $h^n_t = 0$ for $t < 0$), then the Hyena$_N$ operator is causal. In practice, causality is enforced simply by evaluating the filter only at $t = 0, \dots, L-1$ (and implicitly defining $h^n_t = 0$ for $t < 0$) and using the standard zero-padding for FFT convolution, which naturally produces a causal result when the filter is causal.

The Projection Module: From Input to Gates and Values

Before the Hyena recurrence begins, the raw input sequence must be transformed into the $N+1$ sequences ($v, x^1, \dots, x^N$) that drive the recurrence. This is handled by a projection module that mirrors the query-key-value projection in attention but with an extended number of projections.

Algorithm 1 (Projection):

1. A linear layer maps the input $u \in \mathbb{R}^{L \times D}$ to an expanded representation $\hat{z} \in \mathbb{R}^{L \times (N+1)D}$: $\hat{z} = \text{Linear}(u)$, where $\text{Linear}: \mathbb{R}^D \to \mathbb{R}^{(N+1)D}$ is a standard fully-connected layer applied independently at each of the $L$ positions.
2. A short depthwise convolution (filter size 3, as noted in Appendix A.2: "short explicit convolutions of size 3 after the dense input projections") is applied to the expanded representation: $z = \text{DepthwiseConv1d}(h_{\text{short}}, \hat{z})$, where $h_{\text{short}}$ is a short explicit convolution filter learned for each of the $(N+1)D$ channels independently. This short convolution provides local temporal mixing before the global mixing of the Hyena recurrence.
3. The result is reshaped and split into $x^1, \dots, x^N, v$, each of shape $D \times L$ (channels × length).

What it computes: For each position $t$ and each channel $d$, the projection computes a scalar value $v_{t,d}$ (the value signal) and $N$ gating signals $x^1_{t,d}, \dots, x^N_{t,d}$ from the input $u_t$. The value signal carries the information to be transformed through the recurrence, while the gating signals control how information is mixed at each step. The short depthwise convolution adds a small amount of local structure (neighboring token interactions) before the global operations.

Why this form: This design mirrors the query-key-value mechanism in attention—where attention computes $\text{SelfAttention}(u) = A(uW_q, uW_k) \cdot uW_v$—but extends it to $N$ gating signals rather than just one (query) and one (key). In Hyena$_2$, the projections map to $x^1 = k$, $x^2 = q$, and $v$, recovering the H3 structure exactly. In Hyena$_3$, there are three gates, allowing more complex input-dependent routing patterns. The learnable projections are what make Hyena data-controlled: different inputs produce different gating signals, which produce different diagonal matrices $D_{x^n}$, which produce different effective transformation matrices $H(u)$. Without these projections (i.e., if $x^n$ were fixed scalars), the operator would reduce to a fixed sequence of convolutions, losing the ability to adapt to input content.

Complete Forward Pass Algorithm

Algorithm 3 assembles all components into a single forward pass:

Algorithm 3 (Forward pass of Hyena):

1. Compute projections: $x^1, \dots, x^N, v = \text{Projection}(u)$
2. Generate filters: $h^1, \dots, h^N = \text{HyenaFilter}(L, D_e)$
3. Initialize: $v_{\text{cur}} \leftarrow v$
4. For $n = 1, \dots, N$:
   • In parallel across all $D$ channels: $v_{\text{cur}, d} \leftarrow x^n_d \odot \text{FFTConv}(h^n_d, v_{\text{cur}, d})$, where $\odot$ is element-wise multiplication, and $\text{FFTConv}$ performs the fast convolution for that channel.
5. Return $y = v_{\text{cur}}$

Parallelism note: The "in parallel across $D$" step is critical for GPU efficiency. Each of the $D$ channels has its own filter $h^n_d$ (generated by the shared FFN), and the convolution is computed independently per channel (depthwise convolution). This means the $D$ convolutions can be batched into a single FFT operation, maximizing GPU utilization.

Proposition 3.2 states the overall computational complexity:

$\mathcal{O}(NDL(\log_2 L + D))$

where $N$ is the order, $D$ is the model width, and $L$ is the sequence length. The $\log_2 L$ term comes from the FFT convolutions (performed $N$ times), and the $D$ term comes from the projections and gating multiplications (which are $\mathcal{O}(D)$ per position per step).

What it computes: The forward pass transforms an input sequence $u$ of length $L$ and width $D$ into an output sequence $y$ of the same shape, using $N$ alternating convolution-gating steps. Each convolution step mixes information across time using a learned filter, and each gating step modulates the result based on a content-dependent signal. The result is a data-controlled transformation that can route information from any position to any other position, but does so using subquadratic operations.

Why this form: The algorithmic structure is designed for both expressivity and hardware efficiency. The iteration over $n$ is sequential (each step depends on the previous), but within each step, the convolution is computed via FFT (efficient on GPU), the gating is element-wise (trivially parallelizable across positions and channels), and the entire operation never requires materializing an $L \times L$ matrix. This contrasts with attention, which must compute and store the full attention matrix (or use tiling strategies like FlashAttention to avoid full materialization), and with SSM-based models that may require sequential recurrence for filter generation or application.

The Hyena Hierarchy: Relationship to Prior Operators

The paper positions Hyena as a unifying generalization of existing attention-free operators through the order parameter $N$:

"Hyena generalizes this body of work by introducing a recurrence of gates and implicit long convolutions, evaluated efficiently."

In Remark 3.2, the specific relationships are stated:

• Hyena$_1$ recovers GSS (Gated State Spaces): With one gating interaction, the recurrence is $y_t = x_t^1 (h^1 * v)_t$, corresponding to the GSS formulation where a single gate modulates a long convolution (parametrized via SSM in GSS, but the architectural form is the same).

• Hyena$_2$ recovers H3 (Hungry Hungry Hippo): With two gating interactions, the recurrence is the H3 mechanism: $z_t = x_t^1 (h^1 * v)_t$, $y_t = x_t^2 (h^2 * z)_t$, which corresponds to $D_q S_\psi D_k S_\varphi v$ in H3 notation (after swapping the order of $k$ and $v$ to match the paper's convention). The paper notes that "for consistency with our discussion, we have swapped $k$ and $v$ compared to the notation in (Dao et al., 2022c)."

What this means: Hyena$_1$ and Hyena$_2$ are not fundamentally new operators—they are architectural generalizations that can recover existing methods. What Hyena provides is a unified framework for understanding these operators as points in a hierarchy, and the insight that increasing the order $N$ is a principled way to increase expressivity. The paper uses $N=2$ for most language experiments (Table 4.4) and $N=3$ for some WikiText103 experiments (Table 4.3), showing that higher orders can improve performance.

Why order matters: The order $N$ controls the number of alternating convolution-gating steps, which in matrix form corresponds to the number of factors $D_{x^n} S_{h^n}$ in the matrix decomposition. In butterfly factorization theory, longer decompositions (more factors) can represent wider classes of matrices—they increase the rank and expressivity of the structured matrix. The paper suggests that this connection to butterfly decompositions is not coincidental:

"the generalization of (8) to an arbitrary order is inspired by fast evaluation algorithms for structured dense matrices based on butterfly decompositions, with length of the decomposition closely tied to its expressivity."

In other words, just as butterfly factorizations with more factors can approximate more general matrices, Hyena with higher order $N$ can implement more complex data-controlled transformations. The tradeoff is computational: each additional order adds one FFT convolution and one element-wise multiplication, so the cost scales linearly with $N$: $\mathcal{O}(N L \log L + N L D)$.

Design Choice: Why Gating and Convolution Rather Than Alternatives

The paper's central design choice—interleaving long convolutions and gating—is not arbitrary. It is motivated by a careful analysis of what makes attention work and what simpler alternatives cannot do.

Why not just long convolutions alone? A fixed long convolution, no matter how expressive the filter parametrization, cannot be data-controlled. It applies the same transformation to every input sequence, which means it cannot perform content-dependent routing—the core operation of associative recall, where the model must attend to a specific earlier token based on its content (the "key") rather than its position. A fixed convolution could, in principle, learn to route information from specific positions to specific other positions, but this routing would be the same for every input, making it useless for in-context learning where the relevant key-value pairs change from sequence to sequence.

Why not just gating alone? Element-wise gating (multiplying by a data-dependent scalar at each position) can amplify or suppress information at each position, but it cannot move information across positions. A purely gating-based architecture (e.g., $y_t = x_t \cdot v_t$) has no interaction between positions—each output depends only on the input at the same position. To implement associative recall, the model needs to route information from the position where a key appears to the position where the query appears, which requires cross-position mixing.

Why interleaving works: The alternation of convolution and gating creates a composition where each step can first mix information across positions (convolution) and then selectively filter based on content (gating). In Hyena$_2$, the first convolution $\varphi$ spreads information from all previous positions to each position, the first gate $k$ selects which information is relevant (analogous to key-based routing), the second convolution $\psi$ spreads the gated information again, and the second gate $q$ selects the final output (analogous to query-based routing). The paper's analysis in Appendix B.2 makes this explicit by isolating the "surrogate attention matrix" of Hyena$_2$:

$[A_\varphi^\psi(q, k)]_{t, t'} = q_t \sum_{m=0}^{L-1} \psi_{t-m} k_m \varphi_{m-t'}$

This entry represents the effective weight from position $t'$ (the value source) to position $t$ (the output). The sum over $m$ shows that information flows from $t'$ to $t$ via the intermediate position $m$: first convolved from $t'$ to $m$ (weight $\varphi_{m-t'}$), then gated at $m$ (multiplied by $k_m$), then convolved from $m$ to $t$ (weight $\psi_{t-m}$), and finally gated at $t$ (multiplied by $q_t$). This two-step routing through an intermediate position is structurally analogous to how an induction head in a Transformer routes information through the key position to the query position.

Why not attention approximations? The paper deliberately avoids the common strategy of approximating the attention matrix (low-rank, sparse, kernelized) and instead builds a completely different mechanism. The reason is empirical: prior approximations required hybridization with dense attention to match Transformer quality, suggesting they were losing something essential. Hyena, by reconstructing the routing behavior from different primitives, aims to preserve the essential functionality without the $\mathcal{O}(L^2)$ cost.

Causality and Autoregressive Training

For language modeling, the operator must be causal—the output at position $t$ cannot depend on inputs at positions $> t$. Proposition 3.1 provides the formal guarantee:

"If each filter $h^n$, $n = 1, \dots, N$ is causal, then the corresponding Hyena$_N$ operator is causal."

A convolution filter is causal if $h^n_t = 0$ for all $t < 0$, meaning the filter only looks backward in time. When this holds for all $N$ filters, the product of the corresponding Toeplitz matrices (which are lower triangular) and diagonal matrices produces a lower triangular overall matrix $H(u)$.

In practice, causality is enforced not by constraining the filter generation, but by defining the filters only for $t = 0, \dots, L-1$ and using standard causal convolution (only convolving with past inputs). The FFT-based implementation handles this implicitly: zero-padding both sequences to length $2L-1$ and performing the circular convolution yields the same result as the linear causal convolution when the filter is causal.

The paper notes that gating signals $x_t^n$ are computed from position $t$ and earlier positions (because the projections are causal), and convolutions only mix with earlier positions (because the filters are causal), so the entire operator preserves the autoregressive property needed for language model training. This is the same causal constraint applied to self-attention through triangular masking of the attention matrix, but achieved through the structure of convolutions rather than explicit masking.

4. Key Insights and Innovations

Innovation 1: Shifting the Goal from Approximating Attention to Replicating Its Computational Properties

The most intellectually distinctive move in this paper is not architectural—it is a reframing of the research question itself. Prior work on subquadratic attention alternatives, with few exceptions, asked: "How can we approximate the attention matrix with fewer operations?" This framing led naturally to low-rank factorizations (Linformer), kernelized dot-products (Performer), hash-based sparsification (Reformer), and fixed sparsity patterns (Sparse Transformers)—all methods that start from the attention matrix and try to compute something close to it more cheaply.

Hyena asks a fundamentally different question: "What are the essential computational properties of attention that make it work, and can we realize them through completely different mechanisms?" The paper explicitly states this as a departure:

"Instead of seeking an approximation to attention, we guide our design by intentionally incorporating key computational properties of attention."

This is not a cosmetic difference in language. It is a conceptual pivot with cascading consequences. When you try to approximate attention directly, you are constrained by the structure of attention itself—you keep the query-key-value framework, the softmax normalization, the pairwise interaction pattern—and you reduce cost by making these components cheaper (lower rank, fewer pairs, approximate kernels). Any approximation error in any of these components directly degrades quality, which is why every prior method required hybridization with some dense attention layers to recover full Transformer performance. The approximations were losing something essential, and no one had isolated exactly what.

Hyena's reframing is liberating: if you can identify the computational properties that make attention effective (rather than the specific mathematical form of the attention matrix), you can realize those properties through entirely different primitives that are natively subquadratic. The paper distills three such properties from mechanistic interpretability work—data control, sublinear parameter scaling, and unrestricted context—and shows they can be satisfied by interleaving long convolutions (for unrestricted context) with element-wise gating (for data control) while using implicit FFN parametrization (for sublinear parameter scaling). None of these primitives approximates attention; collectively, they replicate what attention does without replicating what attention is.

Why this matters beyond performance: This reframing changes how researchers should approach architecture design for sequence modeling. Rather than treating attention as the gold standard to be approximated, treat it as one possible realization of a set of abstract computational properties—and then explore the space of all architectures that satisfy those properties. Hyena is one point in this space; the paper's hierarchy (Section 3.2) suggests there is a continuum of operators defined by the order $N$ and the filter parametrization, with attention sitting somewhere else in the design space entirely. This is a shift from "attention and its approximations" to "data-controlled structured matrices," which is a broader and more productive framing for future work.

Evidence for the claim: The paper's results on associative recall (Table 4.2, Figure 4.1) demonstrate that Hyena solves a task that prior subquadratic methods could not—not because it approximates attention better, but because its primitives (long convolutions with implicit FFN parametrization, exponential decay windowing, and sinusoidal activations) directly enable the routing operation that associative recall requires. GSS, H3, AFT, and RWKV all fail dramatically on long sequences (e.g., H3 achieves 4.3% accuracy at 64K length while Hyena achieves 100%), even though they all approximate attention more directly than Hyena does. The gap is structural, not quantitative—these methods lack the right computational primitives, not just enough capacity.

Innovation 2: The Hyena Hierarchy as a Unifying Framework for Attention-Free Operators

The paper introduces the concept of an order-$N$ Hyena operator that subsumes previously disparate attention-free architectures as special cases. This is not merely a taxonomic contribution—it provides a principled expressivity knob (the depth $N$ of the recurrence) and a unified mathematical lens (the matrix decomposition $H(u) = D_{x^N} S_{h^N} \cdots D_{x^1} S_{h^1}$) that reveals the structural relationships between operators and explains why some work better than others.

Before Hyena, the subquadratic attention literature was fragmented. Gated State Spaces (Mehta et al., 2022) combined SSM convolutions with one gating interaction. H3 (Dao et al., 2022c) added a second gating interaction and a short convolution. AFT (Zhai et al., 2021) used a single gate and an explicit convolution. RWKV (Peng, 2021) used a recurrent time-mixing formulation that was difficult to compare to other approaches. Each model was presented as a distinct architecture with its own notation, its own design rationale, and its own set of hyperparameters. There was no framework for understanding how they related to each other, what design choices were load-bearing versus incidental, or how to systematically improve them.

Hyena provides this framework. Remark 3.2 states the relationships directly: Hyena$_1$ recovers GSS, Hyena$_2$ recovers H3. Higher orders go beyond both. The matrix form $H(u) = D_{x^N} S_{h^N} \cdots D_{x^2} S_{h^2} D_{x^1} S_{h^1}$ provides a uniform way to analyze any operator in this class: every operator is a product of alternating diagonal (gating) and Toeplitz (convolution) matrices, with the order controlling the number of factors. This decomposition clarifies several non-obvious points:

• Why H3 outperforms GSS: GSS (Hyena$_1$) has one convolution-gating pair, H3 (Hyena$_2$) has two. The additional pair doubles the number of factors in the matrix decomposition, which butterfly factorization theory suggests increases the expressivity of the structured matrix class. In associative recall terms, two gating interactions enable query-key style routing through an intermediate position, while one gating interaction only enables a simpler modulation.

• Why additional orders might help: The matrix decomposition length directly controls the class of matrices that can be represented. With $N$ factors, Hyena can implement $N$-step routing patterns where information flows through $N$ intermediate positions, each modulated by a content-dependent gate. For tasks requiring multi-hop reasoning (e.g., "the capital of the country where the inventor of the telephone was born"), higher orders could in principle route information through multiple intermediate keys.

• Why the parametrization of the Toeplitz factors matters independently of the order: The decomposition reveals that $S_{h^n}$ (the convolution) and $D_{x^n}$ (the gating) are separate design choices. One could use SSM-parametrized convolutions (as in H3) or FFN-parametrized convolutions (as in Hyena) without changing the order. The paper's experiments (Figure 4.1) show that the filter parametrization choice creates a larger accuracy gap than the choice of order, suggesting that prior work's underperformance was partly due to suboptimal filter design rather than insufficient expressivity in the gating structure.

Why this matters beyond taxonomy: The hierarchy transforms order $N$ from an arbitrary hyperparameter into a principled dimension for architecture search. If $N=2$ is insufficient for a task, increasing to $N=3$ or $N=4$ is a well-motivated next step rather than a shot in the dark. This is analogous to how increasing the number of attention heads or Transformer layers has become a standard scaling dimension—the Hyena hierarchy provides a similar scaling dimension for the operator itself. The paper shows preliminary evidence that $N=3$ improves WikiText103 perplexity over $N=2$ (Table 4.3: Hyena-3-slim achieves 18.5 vs. 18.6 for Hyena-3, though the difference is small at this scale).

The connection to butterfly decompositions (Li et al., 2015; Dao et al., 2019) adds theoretical weight to the hierarchy concept. Butterfly factorizations are a classical technique for fast matrix-vector multiplication where a dense matrix is expressed as a product of sparse structured factors—exactly the form of the Hyena decomposition, but without data control. The fact that Hyena's decomposition mirrors this classical structure suggests that it inherits favorable properties (expressivity with subquadratic evaluation) while adding data control as a novel ingredient. This is a deeper theoretical grounding than most architecture papers provide, connecting the design to established results in numerical linear algebra.

Boundary and limitation: The hierarchy is descriptive (it unifies existing operators and suggests extensions) but not prescriptive (it does not tell you what $N$ to use for a given task or compute budget). The paper uses $N=2$ for most experiments and $N=3$ for some, but provides no systematic study of how performance scales with $N$. This is an incremental contribution—the hierarchy framework itself—that enables future fundamental work on scaling laws for operator order.

Innovation 3: Empirical Discovery That Implicit Filter Parametrization with Exponential Decay Is the Critical Design Choice

Perhaps the most surprising and consequential finding in this paper is not about the Hyena recurrence itself, but about how to parametrize the long convolution filters within it. The paper's experiments (Figure 4.1, Table A.2) reveal an enormous performance cliff between different filter parametrizations—a 77-point accuracy gap on associative recall—all within the same Hyena$_2$ recurrence framework. This finding has implications that extend beyond Hyena to the entire field of long-convolution sequence models.

The field's default assumption before this paper was that the gating mechanism (the number and arrangement of data-controlled interactions) was the primary bottleneck for attention-free models. H3 added a second gate to GSS because the first gate was insufficient for associative recall. The natural next step would be to add more gates, more complex interactions, or more attention-like features. What the paper shows instead is that even with the same gating structure (Hyena$_2$ matches H3), switching from SSM-based filters to FFN-based filters with exponential decay windows and sinusoidal activations improves associative recall accuracy from 13% to 85% (at vocabulary size 40, sequence length 2048). The filter parametrization matters more than the gating structure—a finding that was not obvious ex ante and that redirects research attention from gate design to filter design.

Why this is a fundamental insight, not an incremental tuning result: The paper provides a mechanistic explanation for why different parametrizations produce such different results. SSM-based filters are constrained to be sums of exponentials (if $A$ is diagonal) or more generally linear combinations of basis functions determined by the eigenvalues of the state matrix. This parametric form is excellent for representing smooth, continuous dynamics—exactly what SSMs were designed for in modeling physical systems—but struggles with the discrete, exact-matching operations needed for associative recall. An SSM naturally "blurs" information across time because its continuous-time dynamics act as a low-pass filter, attenuating the high-frequency content needed to represent a sharp delay (routing information from exactly position $t-k$ to position $t$). Explicit FIR filters can represent sharp delays but require $\mathcal{O}(L)$ parameters to do so. The FFN parametrization with sinusoidal activations avoids both problems: it can represent sharp, high-frequency content (through the periodic activation functions) while maintaining sublinear parameter scaling (through the fixed-size FFN). The exponential decay window adds a learnable inductive bias toward locality that can be relaxed when the task requires long-range dependencies.

This insight is diagnostic: it tells the field that the representation capacity of the filter function class is a first-order concern for long-convolution models, not a detail that can be hand-waved. Prior work (S4, H3, GSS) focused heavily on the state-space machinery—the structure of $A$, the initialization of the HiPPO matrix, the discretization step size—because those were the degrees of freedom in the SSM framework. Hyena shows that a completely different function class (FFNs with sinusoidal activations) can outperform SSMs dramatically on the specific operations that matter for language, redirecting the optimization effort from state-space engineering to filter function approximation.

Evidence for the claim: The paper's associative recall experiments (Figure 4.1, Table A.2) provide a clean ablation. All models use the same Hyena$_2$ recurrence, same width (64), same depth (2 layers), and same training data. The only difference is the filter parametrization. The ranking—Hyena (FFN + exponential decay) > CKConv (FFN only) > TransferFunc > H3 (SSM) > FNO (frequency-domain) > Conv1d (explicit FIR)—is consistent across sequence lengths and vocabulary sizes, with the gap widening as sequences get longer. At 131K sequence length (Table A.2), Hyena achieves 97.2% while H3 achieves 0.6%—a near-total collapse of the SSM-based approach at extreme lengths. This is not a subtle difference that requires careful measurement; it is a qualitative failure mode of SSM filters at long range that was not apparent at the shorter lengths (1K–4K) typically used in prior benchmarks.

Implications beyond Hyena: The finding suggests that the Long Range Arena (LRA) benchmark, on which S4 achieved strong results, may not be predictive of performance on tasks requiring exact token retrieval (like associative recall). LRA tasks (ListOps, text classification, retrieval, etc.) involve long-range dependencies but may not require the sharp, discrete matching that language modeling demands. If this hypothesis is correct, it explains why SSMs performed well on LRA but failed to match Transformers on language—and it suggests that the field needs benchmarks that specifically probe for the operations that make language modeling hard, rather than generic "long-range dependency" benchmarks.

Innovation 4: Resolving the Attention-Free Quality Gap Without Hybridization

The paper's headline empirical claim—that Hyena is "one of the first attention-free, convolutional architectures to match perplexity and downstream performance of Transformers" (Section 5)—might sound like an incremental milestone. But the significance lies in how this gap was closed and what it implies about the necessity of attention.

All prior attention-free architectures that approached Transformer quality did so through hybridization: they combined attention-free layers with some standard dense attention layers. H3 (Dao et al., 2022c), the strongest prior work, explicitly used hybrid architectures with attention layers interspersed. The implicit message was that attention-free operators alone were insufficient—you needed at least some quadratic attention to reach full quality, and the best you could do was minimize how much attention you used. This framing treats attention as an essential ingredient that can be diluted but not eliminated.

Hyena's results break this framing. By matching GPT perplexity on THE PILE with a 20% reduction in total FLOPs (Table 4.4: Hyena-2 355M at 9.2 perplexity vs. GPT 355M at 9.1, with FLOPs of 3.93 vs. 4.77 ×10¹⁹), and by matching Transformer accuracy on downstream SuperGLUE tasks (Table 4.5, 4.6), the paper demonstrates that attention is not a necessary component of high-quality language models at this scale. The specific operations that attention provides in Transformers can be equivalently provided by the interleaved convolution-gating recurrence in Hyena, and neither the quadratic cost nor the specific softmax-normalized dot-product form of attention is essential.

Why this is a conceptual shift, not just a benchmark result: The paper establishes that the "gap" between attention-free and attention-based models was not a gap in principle but a gap in design. Prior attention-free models failed not because they lacked attention, but because their specific primitives (SSM-based convolutions, single gating interactions, explicit short convolutions) could not implement the routing operations that language modeling requires. Once those primitives are replaced with more capable alternatives (FFN-based implicit convolutions with exponential decay, multiple gating interactions in a recurrence, sinusoidal activations for high-frequency content), the gap vanishes. This is an existence proof: there exists at least one attention-free architecture that matches Transformers, which means the research program of building ever-better attention-free models is viable, not a dead end.

The contrast with prior hybridization strategies is instructive. Hybrid H3 (125M) achieved 18.5 perplexity on WIKITEXT103 (Table 4.3), matching Transformers but using some attention layers. Hyena achieves the same 18.5–18.6 perplexity with no attention at all. The fact that removing the last attention layers does not degrade performance means those layers were not providing any capability that Hyena's primitives couldn't already provide—they were compensating for limitations in H3's filter parametrization and gating structure, not providing something fundamentally unavailable to attention-free designs.

Evidence and boundary conditions: The paper is appropriately cautious about the scale of this claim. All matching results are at the sub-billion parameter scale (125M–355M parameters). Section 5 notes "preliminary results for models at the 1.3B parameter scale (10.8 perplexity after 5 billion tokens)" but does not provide a full comparison at that scale. It is possible that at larger scales (tens or hundreds of billions of parameters), attention provides benefits that Hyena cannot replicate. The paper's own FLOPs comparison (Table 4.4) shows that Hyena's advantage shrinks at larger model sizes (the FLOP reduction is smaller for the 355M model than for the 125M model, because the ratio of attention FLOPs to total FLOPs decreases as the FFN width grows). This is a fundamental limitation of the claim: Hyena reduces the cost of the sequence-mixing operation, but the feed-forward layers dominate total FLOPs at large widths, so the relative benefit of cheaper mixing diminishes. This is a known property of the Transformer architecture (the FFN is already $\mathcal{O}(LD^2)$ while attention is $\mathcal{O}(L^2 D)$), and the paper does not claim to solve it.

Innovation 5: Demonstrating That Mechanistic Interpretability Benchmarks Predict Scaling Behavior

The paper makes a methodological contribution by showing that performance on small, synthetic mechanistic interpretability benchmarks—specifically associative recall with varying vocabulary sizes—correlates with language modeling perplexity at scale. Table C.1 in the Appendix provides the evidence: for six different models (Conv1d, AFT-conv, H3, Transformer, Hyena), the ranking on associative recall accuracy at vocabulary sizes 10–40 matches the ranking on THE PILE test loss after 5 billion tokens of training. Models that fail on associative recall (Conv1d: 8% at vocab 40) produce the worst perplexity (4.21); models that succeed (Hyena: 85% at vocab 40, Transformer: 82%) produce the best perplexity (2.59 for both).

This is not an obvious finding. The associative recall task uses 2-layer models with width 64, trained on 2000 synthetic sequences. The language modeling task uses models with tens of layers, hundreds of millions of parameters, trained on billions of real tokens. There is no a priori reason to expect the ranking to transfer—the synthetic task probes a single capability (can the architecture route information from a key to a query?) while language modeling requires dozens of interacting capabilities. Yet the correlation is strikingly clean.

Why this matters for research methodology: If this correlation holds more broadly, it changes how the field can iterate on architecture design. Training a 355M-parameter language model on 5 billion tokens takes days or weeks on multiple GPUs. Training a 2-layer, width-64 model on a synthetic associative recall task takes minutes on a single GPU. If the smaller experiment is predictive of the larger one, researchers can explore orders of magnitude more design choices in the same amount of compute time. This would accelerate the architecture search cycle from months to hours, enabling a more systematic exploration of the design space for attention-free operators.

The paper does not claim universal predictiveness—the correlation is shown for a single task (associative recall) correlated with a single metric (THE PILE test loss) for a specific set of models. The generalizability to other tasks (e.g., does majority voting synthetic performance predict something about language?), other metrics (e.g., downstream task accuracy), and other model classes is unknown. But the existence of even one such predictive relationship is methodologically significant because it demonstrates the principle: mechanistic understanding of what operations an architecture can and cannot perform can guide design more efficiently than end-to-end training and evaluation alone.

Evidence and caveats: Table C.1 shows the correlation, but the paper does not provide a statistical measure of the relationship (no correlation coefficient, no confidence interval). The sample size is only 5 models, and the ranking is monotonic (no inversions) but the spacing varies. H3 (92% at vocab 10) is much closer to Hyena/Transformer on easy tasks than on hard ones (13% vs. 98% at vocab 30), which is exactly the kind of difficulty-dependent gap that manifests at scale (H3's language modeling perplexity of 2.69 vs. Hyena's 2.59 looks small but represents a meaningful quality difference). This suggests the harder versions of the synthetic task are more diagnostic of scale behavior, which is itself a useful methodological insight.

The paper also notes in Appendix C that "performance on our language synthetics appears to be further linked to performance as attention replacement in other domains (Appendix A.4 for results on image classification)." This cross-domain transfer (from synthetic language tasks to image classification accuracy) is only suggestive—the image results are limited to a single architecture comparison (Hyena-ViT vs. ViT on ImageNet, Table 4.7)—but it hints that the associative recall primitive may be a universal bottleneck for sequence modeling architectures across modalities, not just for language. If confirmed, this would elevate associative recall from a mechanistic curiosity to a fundamental design criterion for any sequence model.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary language modeling evaluation uses THE PILE (Gao et al., 2020), specifically the standard training and test splits. For perplexity comparisons, WIKITEXT103 is also used with the standard GPT2 tokenizer (vocabulary size 50,257). Downstream evaluation uses the SuperGLUE benchmark (Wang et al., 2019) across eight tasks. For long-range language modeling, the PG-19 corpus (Rae et al., 2019) is used with context length 16K tokens. Image classification uses ImageNet-1k (1.3M images, 1,000 classes) and CIFAR-10.

• Base model(s). For language modeling, the paper trains Hyena models at scales of 125M, 153M, 355M, and 1.3B parameters, with corresponding Transformer (GPT) baselines at matched parameter counts (125M and 355M). The GPT models follow the standard architecture of Brown et al. (2020) and use FlashAttention (Dao et al., 2022b) for efficient training. All models within a comparison use the same GPT2 tokenizer. The choice of sub-billion parameter scale is deliberate: it is "representative of the capabilities of many contemporary LLMs" (Section 4) while remaining feasible for extensive ablation and controlled comparison. For image classification, a Vision Transformer (ViT-B, 87M parameters) is used as the attention baseline, with Hyena-ViT at 88M parameters as the attention-free replacement.

• Metrics. Perplexity is the primary metric for language modeling quality, computed as the exponentiated cross-entropy loss on held-out test data. For downstream evaluation, accuracy (%) is reported on each SuperGLUE task, with an average across all eight tasks. Zero-shot and few-shot (3-shot) settings are both evaluated. For image classification, top-1 accuracy (%) is reported. For synthetic reasoning tasks, test accuracy (%) measures whether the model produces the correct token. For runtime benchmarks, milliseconds per forward pass is measured at varying sequence lengths with batch size 64.

• Baselines. The paper compares against a comprehensive set of prior approaches. For attention-based models: standard GPT (Brown et al., 2020) using FlashAttention (Dao et al., 2022b), Transformer (standard PyTorch implementation), and GPTNeo (Black et al., 2021) for downstream evaluation. For attention-free models: AFT-conv (Zhai et al., 2021), RWKV (Peng, 2021), H3 (Dao et al., 2022c), GSS (Mehta et al., 2022), Performer, Reformer (Kitaev et al., 2020), and Linear Attention. For convolution parametrization ablations: Conv1d (explicit FIR filters), FNO (Li et al., 2020, frequency-domain parametrization), TransferFunc (rational transfer functions), CKConv (Romero et al., 2021b, FFN-based implicit convolutions without Hyena-specific enhancements). For image classification: ViT (Dosovitskiy et al., 2020), S4ND (Nguyen et al., 2022).

• Generation budget / compute accounting. Total FLOPs is the primary compute metric for language model comparisons, computed following Hoffmann et al. (2022) for Transformers and an explicit layer-by-layer accounting for Hyena (Appendix A.2). The FLOP count for Hyena replaces attention FLOPs with: (i) projection FLOPs: order × d_model² × seq_len, (ii) short convolution FLOPs: order × d_model × seq_len × filter_len, (iii) FFTConv FLOPs: 5 × (order − 1) × d_model × log(seq_len) × seq_len, and (iv) output projection FLOPs: d_model² × seq_len, all with a factor of 2 for multiply-add operations. For synthetic reasoning tasks, all models have the same width (64) and depth (2 layers) to control for parameter count. For runtime benchmarking, batch size is fixed at 64 and wall-clock time is measured in milliseconds.

• Cross-validation / statistical protocol. For language modeling, each model is trained for three independent runs of 5, 10, and 15 billion tokens to assess scaling behavior, with learning rates adjusted per model size (peak 6 × 10⁻⁴ for 125M, 4 × 10⁻⁴ for 355M, 2.2 × 10⁻⁴ for 1.3B). For synthetic reasoning tasks, models are trained for 200 epochs on 2,000 samples with a cosine decay schedule and 10-epoch linear warmup. No cross-validation is performed; results are reported on standard test splits.

Main Quantitative Results

Mechanistic Design Benchmarks: Narrowing the In-Context Learning Gap

The paper's most dramatic results are on synthetic associative recall, where Hyena achieves near-perfect accuracy even at extreme sequence lengths while all other attention-free operators collapse. The headline numbers from Table 4.2:

• At sequence length 30K (vocabulary size 30): Hyena achieves 100.0% accuracy, while FlashTransformer reaches only 32.4%, H3 reaches 8.4%, and RWKV reaches 12.4%. Standard Transformer runs out of memory and cannot be evaluated.
• At sequence length 64K: Hyena maintains 100.0% accuracy. FlashTransformer drops to 26.7%. H3 falls to 4.3%. GSS falls to 2.1%. RWKV reaches only 6.5%.
• At sequence length 131K (the most extreme setting): Hyena achieves 97.2% accuracy—a slight degradation but still near-perfect. Every other attention-free operator collapses to near-zero: H3 at 0.6%, GSS at 0.1%, AFT at 0.8%, RWKV at 2.3%. FlashTransformer cannot fit in memory ("×" in the table).

These results challenge a prevailing assumption in the field. As the paper states:

"Our results challenge the observation that only Transformers are capable of challenging in-context learning."

The convolution parametrization ablation in Figure 4.1 reveals which design choice drives this gap. At vocabulary size 40 and sequence length 2048, with all models embedded in the same Hyena₂ recurrence and matched for width and depth:

• Hyena (FFN with exponential decay + sinusoidal activations): ~85% accuracy
• CKConv (FFN without decay windowing or sinusoidal activations): ~40% accuracy—a 45-point gap
• H3 (SSM-parametrized filters): ~13% accuracy—a 72-point gap
• FNO (frequency-domain explicit): ~10% accuracy
• Conv1d (explicit FIR, filter size 64): ~8% accuracy

The gaps widen with vocabulary size (Figure 4.1, left to right). At vocabulary size 10, most implicit methods perform well (CKConv and Hyena both near 100%). At vocabulary size 40, only Hyena maintains high accuracy. This vocabulary-size scaling is significant because larger vocabularies increase the average distance between related tokens in the sequence, making the recall operation more demanding. The paper interprets this as evidence that prior work's failure on associative recall was not just a capacity issue—it was a structural limitation of the filter parametrization. SSM-based filters (H3, TransferFunc) fundamentally cannot represent the sharp, exact-delay operations needed for associative recall at long ranges, regardless of how many parameters or training samples are provided.

Table A.2 pushes the parametrization comparison to the extreme of 131K sequence length. At this scale, the collapse of non-Hyena methods is total:

"Hyena: 97.2%, CKConv: 14.3%, TransferFunc: 0.5%, H3: 0.6%, FNO: 0.3%, Conv1d: 0.5%"

The fact that CKConv (FFN-based but without Hyena's exponential decay windowing or sinusoidal activations) achieves 14.3% while H3 achieves 0.6% shows that the FFN parametrization alone provides some benefit at extreme lengths, but the full Hyena filter design (FFN + decay windowing + sinusoidal activations) is necessary for near-perfect performance.

Language Modeling: Matching Transformers with Reduced Compute

On WIKITEXT103 (Table 4.3), Hyena matches or slightly improves upon Transformer perplexity at 125M parameters:

• Transformer (125M): 18.6 perplexity
• Hyena-3 (125M): 18.6 perplexity—an exact match
• Hyena-3-slim (125M): 18.5 perplexity—slightly better, achieved by trading width for depth (18 layers instead of 12, with thinner FFNs)
• Hybrid H3 (125M): 18.5 perplexity (but uses some attention layers)
• Performer (125M): 26.8 perplexity—a dramatic gap confirming that kernelized attention fails at language modeling
• Reformer (125M): 25.6 perplexity
• AFT-conv (125M): 28.2 perplexity
• Linear Attention (125M): 25.6 perplexity

The key comparison is Hyena versus Hybrid H3: Hyena achieves the same perplexity (18.5–18.6) without any attention layers, while H3 required hybridizing with dense attention to reach this level. This directly supports the paper's claim that attention is not a necessary component for language modeling quality at this scale.

On THE PILE (Table 4.4), the paper provides a more detailed scaling picture across model sizes and training token budgets:

At 125M/153M scale:

Hyena matches GPT perplexity at every training budget while using essentially the same FLOPs (1.87 vs. 1.88 ×10¹⁹)—a negligible difference at this scale due to Hyena's slightly larger parameter count (153M vs. 125M) being offset by cheaper sequence mixing.

At 355M scale:

At this larger scale, Hyena achieves essentially identical perplexity (differing by only 0.1 at convergence) while using 18% fewer FLOPs (3.93 vs. 4.77 ×10¹⁹). This is the source of the paper's "20% reduction in training FLOPs" claim—the savings come from replacing attention's O(L²) compute with Hyena's O(L log L) convolutions, and the relative savings increase with model width because attention's cost grows with L²D while the FFN layers (shared between both architectures) dominate in the FLOP count for large D.

The preliminary scaling curves in Figure 4.2 show that both Hyena and GPT follow similar log-linear scaling trends with total training tokens, with Hyena consistently matching or slightly trailing GPT at equivalent compute. The paper describes these as "preliminary scaling laws" and does not claim statistical rigor, but the visual evidence suggests that the gap does not widen with scale—if anything, the curves are converging at 15B tokens.

Long-Range Language Modeling on PG-19

On the PG-19 benchmark with context length 16K tokens, a Hyena 153M model achieves 14.6 test perplexity after 8 epochs of training. This result is reported without a direct Transformer comparison (standard Transformers at sequence length 16K would be prohibitively expensive), but it establishes that Hyena can effectively use much longer context windows than the 2K sequence length used in the main experiments.

Downstream Evaluation: SuperGLUE

Tables 4.5 and 4.6 compare Hyena (153M, trained for 137B tokens) against GPTNeo (125M, trained for 300B tokens) and RWKV (169M, trained for 332B tokens) on SuperGLUE tasks. Key results:

Zero-shot (Table 4.5):

• Hyena average: 41.5%
• GPTNeo average: 43.8%
• RWKV average: 40.2%

Few-shot, 3 examples (Table 4.6):

• Hyena average: 49.3%
• GPTNeo average: 49.1%
• RWKV average: 43.0%

Hyena matches GPTNeo in few-shot performance (49.3 vs. 49.1) despite being trained on less than half the total tokens (137B vs. 300B). In zero-shot, Hyena slightly underperforms GPTNeo (41.5 vs. 43.8) but outperforms RWKV (41.5 vs. 40.2). The few-shot improvements are particularly notable on specific tasks:

"Hyena to display characteristic few-shot capabilities of standard Transformers, with some tasks e.g., MultiRC seeing a lift of more than 20% accuracy over zero-shot when the model is provided additional prompts as context."

MultiRC improves from 1.1% (zero-shot) to 26.7% (few-shot)—a 25.6-point gain that exceeds GPTNeo's improvement (0.0% to 22.4%). This demonstrates that Hyena not only matches Transformers on final performance but also exhibits the same qualitative in-context learning behavior that makes Transformers valuable as few-shot learners.

On the LAMBADA task (Appendix A.3), the 153M Hyena model achieves 44.64% accuracy. No direct comparison is provided, but the result establishes non-trivial performance on a challenging long-range dependency task.

Language Modeling at Larger Scale

The paper reports preliminary results at the 1.3B parameter scale: "10.8 perplexity after 5 billion tokens" (Appendix A.2). No Transformer comparison is provided at this scale, so this is a single data point rather than a comparative result. Additionally, a 153M model trained for 130B tokens reaches 9.8 perplexity, showing continued improvement with extended training.

Runtime Benchmarking

Figure 4.3 provides the efficiency results that motivate Hyena for long sequences:

• Crossover point with standard attention: Hyena is faster at sequence length 2,048—meaning even at moderate sequence lengths, Hyena's subquadratic scaling provides a wall-clock advantage.
• Crossover point with FlashAttention: Hyena becomes faster between 4,096 and 8,192 tokens—a higher threshold because FlashAttention is already 2–4× faster than standard attention through IO-aware tiling.
• At sequence length 8,192: Hyena achieves ~2× speedup over FlashAttention (5× over standard attention).
• At sequence length 64K: Hyena achieves 100× speedup over FlashAttention. Standard attention runs out of memory at this length.

The paper notes that the speedup at 64K is partly due to FlashAttention's memory requirements becoming prohibitive, not just computational cost. This is an important caveat: the 100× figure includes both algorithmic efficiency and the fact that FlashAttention simply cannot operate at this sequence length on the available hardware.

The paper is transparent about hardware utilization limitations:

"Despite the absolute reduction in FLOPs, speedups are achieved only on longer sequences when the gap grows sufficiently large. This occurs because hardware utilization of Hyena is lower than FlashAttention."

The FFT-based convolution, while asymptotically faster, does not yet achieve the same GPU utilization (50–70% for optimized attention vs. a lower unspecified value for FFTConv). The paper expects this gap to narrow with improved implementations of FFTConv and specialized hardware.

Image Classification

Table 4.7 demonstrates Hyena's generality beyond language:

ImageNet-1k (training from scratch):

Hyena-ViT matches ViT at the standard 16×16 patch size (78.5% for both) and comes within 0.2% at the finer 8×8 patch size. The paper notes that finer patches produce longer sequence lengths, suggesting Hyena's efficiency advantage would grow as vision models move toward higher-resolution inputs.

CIFAR-10 (2D convolution):

Hyena improves accuracy by 1.3 points while using 25% fewer parameters than S4ND, demonstrating that the Hyena filter design (FFN parametrization with exponential decay) transfers to 2D convolutions and outperforms SSM-based approaches in vision as well.

Ablation Studies and Robustness Checks

Convolution parametrization in Hyena₂ (Figure 4.1, Table A.2): This is the paper's central ablation. Six parametrization methods are compared within the same Hyena₂ recurrence at fixed width and depth on associative recall. The ranking is consistent across sequence lengths and vocabulary sizes: Hyena (FFN + exponential decay + sinusoidal activations) > CKConv (FFN only) > TransferFunc (rational transfer functions) > H3 (SSM) > FNO (frequency-domain) > Conv1d (explicit FIR). The gap between best and worst is 77 percentage points at vocabulary size 40, sequence length 2048. At 131K sequence length, the gap grows to 96.7 points (97.2% vs. 0.5%). Non-obvious finding: CKConv (FFN without Hyena's specific enhancements) already substantially outperforms SSM-based methods at extreme length (14.3% vs. 0.6% for H3 at 131K), suggesting the FFN parametrization class is inherently better suited than SSMs for tasks requiring exact token retrieval, even before adding Hyena's decay windowing and sinusoidal activations.

Operator comparison at extreme sequence lengths (Table 4.2): Comparing full model architectures (not just filter parametrizations within Hyena) on associative recall. All models have 2 layers and width 64. Hyena is the only operator that solves the task at 30K, 64K, and 131K sequence lengths. GSS (Hyena₁ equivalent) collapses to 5.3%, 2.1%, and 0.1% respectively. H3 collapses to 8.4%, 4.3%, and 0.6%. This shows that the filter parametrization improvements in Hyena, rather than the gating structure alone, drive the performance—H3 already has Hyena₂'s gating structure but uses SSM filters. Non-obvious finding: Transformers can solve associative recall at length 30K (32.4% accuracy) but fail to generalize from limited data (only 2,000 training samples), while Hyena achieves 100% on the same data budget. This suggests Hyena has a stronger inductive bias for the recall operation, requiring fewer examples to discover the correct solution.

Order N (Table 4.3, implicit): Comparing Hyena-2 vs. Hyena-3 on WIKITEXT103 at 125M scale: Hyena-3 achieves 18.6, Hyena-3-slim achieves 18.5. The difference between N=2 and N=3 is small (0.1 perplexity) at this scale, suggesting diminishing returns from additional gating interactions for standard language modeling. The paper does not provide a systematic sweep over N, making this a suggestive rather than conclusive ablative result.

Model depth vs. width (Table 4.3): Hyena-3-slim uses 18 layers with thinner FFNs (width multiplier 2×) compared to Hyena-3's 12 layers with wider FFNs (width multiplier 4×), at the same parameter count. The slim variant achieves slightly better perplexity (18.5 vs. 18.6), suggesting that the reduction in attention FLOPs enables trading width for depth, which is generally favorable for language modeling. This is enabled by Hyena's lower per-layer cost: with cheaper sequence mixing, more layers can be added within the same compute budget.

PRM aggregation strategy (Appendix B.2, equation 19): While not labeled as an ablation, the analysis of how the surrogate attention matrix $A_\varphi^\psi(q, k)$ composes from its factors provides an implicit ablation of the gating structure. The decomposition $A_\varphi^\psi(q, k) = D_q S_\psi D_k S_\varphi$ shows that the second gate (query) modulates after the second convolution, while the first gate (key) modulates between convolutions. If the gating matrices commuted with the convolution matrices, the entire operator would collapse to a simple convolution ($W^* D_q D_\Psi D_k D_\Phi W$), losing all data control. The paper explicitly notes this non-commutativity as essential: "The non-commutativity of the gating term acts as a non-linearity in chain of convolution operators." This explains why interleaving (alternating convolution and gating) is fundamentally different from applying all convolutions first and all gating after—the structure of the product matters, not just the set of operations.

Short convolution after projections (Algorithm 1, Appendix A.2): The projection module includes a "short explicit convolution of size 3 after the dense input projections." No ablation is provided to measure its contribution, but its presence suggests the authors found local structure important enough to include. This mirrors H3's use of a "shift SSM" for local interactions and may be load-bearing for performance even though the paper emphasizes long convolutions.

Exponential decay window (Figure 3.1, Section 3.3): The paper states that "we choose at least one of the convolutions in Hyena to be shaped towards exponential decay" but provides no ablation comparing Hyena with versus without exponential decay windowing. The comparison to CKConv (which uses FFN parametrization without decay modulation) provides an indirect ablation: CKConv achieves 40% accuracy vs. Hyena's 85% at vocabulary size 40, sequence length 2048. However, CKConv also lacks sinusoidal activations and the specific Hyena initialization, so the 45-point gap cannot be attributed to decay windowing alone.

Sinusoidal activations in filter FFN (Section 3.3): No direct ablation is provided comparing sinusoidal vs. standard ReLU/tanh activations in the filter FFN. The paper cites Basri et al. (2020) on the low-frequency bias of standard networks and states that sinusoidal activations "address the low-frequency bias," but the empirical contribution of this choice is not isolated. The gap between CKConv (ReLU-based, ~40%) and Hyena (sinusoidal, ~85%) on associative recall provides indirect evidence, but the comparison is confounded by other differences.

Downstream evaluation with fewer training tokens (Tables 4.5, 4.6): Hyena (137B training tokens) is compared against models trained for 2–3× more tokens (GPTNeo at 300B, RWKV at 332B). This is not a controlled ablation but reveals that Hyena achieves competitive downstream performance despite a significant training token disadvantage. The fact that few-shot performance matches GPTNeo (49.3 vs. 49.1 average) with less than half the training data suggests Hyena's architectural inductive biases may accelerate the emergence of in-context learning capabilities, though this hypothesis is untested.

Critical Assessment

Claim 1: "Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods."

This claim is strongly supported for associative recall at extreme sequence lengths. The specific evidence:

• At 64K length, vocabulary 30: Hyena 100% vs. H3 4.3%, GSS 2.1%, AFT 1.2% (Table 4.2)—gaps of ~96–99 points, far exceeding 50.
• At 131K length: Hyena 97.2% vs. CKConv 14.3% vs. H3 0.6%—gaps of 83–97 points (Table A.2).

However, the "more than 50 points" framing in the abstract understates the magnitude—the gaps are often much larger than 50 points at the extreme lengths the paper emphasizes. The claim is specifically about associative recall and reasoning tasks, not about language modeling perplexity (where improvements are measured in single perplexity points, not percentage accuracy). This distinction matters because a reader might assume Hyena provides 50-point accuracy gains on standard NLP benchmarks, which is not the case. The abstract's language ("In recall and reasoning tasks... Hyena improves accuracy by more than 50 points") is precise but easy to misread as applying to all evaluations.

Claim 2: "Hyena sets a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets, reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K."

Supported with important qualifications. The evidence:

• WIKITEXT103 (Table 4.3): Hyena matches Transformer perplexity (18.6) without any attention layers. Prior attention-free models (Performer: 26.8, Linear Attention: 25.6) fell far short. This supports the "state-of-the-art for dense-attention-free" claim.
• THE PILE at 355M (Table 4.4): Hyena achieves 9.2 perplexity vs. GPT's 9.1 at 15B tokens—a 0.1 gap that is probably within noise—while using 3.93 vs. 4.77 ×10¹⁹ FLOPs, an 18% reduction. This supports the "20% reduction in training compute."
• Figure 4.2: The scaling curves show Hyena and GPT converging at 15B tokens, with Hyena using fewer FLOPs to reach the same perplexity.

Qualifications and missing evidence:

1. The 20% FLOP reduction applies to one specific comparison (355M, 15B tokens). At 125M scale (Table 4.4), the FLOP reduction is negligible (1.87 vs. 1.88 ×10¹⁹) because Hyena uses more parameters (153M vs. 125M) to compensate for any architectural disadvantages. The 20% figure should not be interpreted as a universal constant—it depends on model size and the ratio of attention FLOPs to FFN FLOPs.

2. The comparison is at sequence length 2K. The paper's primary scaling experiments all use sequence length 2,048. At this length, the quadratic cost of attention is not yet dominant—attention FLOPs are O(L²D) = O(2048² × D) while FFN FLOPs are O(LD²). The 18% savings come primarily from avoiding the attention matrix computation, not from asymptotic scaling advantages. The paper's own efficiency argument (Section 4.4) shows that Hyena's speed advantage is modest at 2K (crossover around 2048 for standard attention) and only becomes dramatic at 8K+. The language modeling experiments would need to be repeated at longer sequence lengths (8K, 16K, 32K) to fully validate the claim that Hyena enables efficient training with long context—at 2K, the "20% reduction" is real but limited.

3. The largest comparison is at 355M parameters. At this scale, the FFN dominates total FLOPs (4 × 355M parameters × 12 layers × ...), so the relative savings from cheaper attention are already shrinking. The paper reports preliminary 1.3B results (10.8 perplexity at 5B tokens) but no FLOP comparison or Transformer baseline at that scale. It is possible that at 1.3B+, the FLOP advantage diminishes further because the FFN's O(LD²) compute dwarfs any savings from O(L log L) vs. O(L²) sequence mixing.

4. The matching perplexity (9.2 vs. 9.1) is a single data point at one scale and one training budget. The difference (0.1 perplexity) is small enough that statistical significance is unclear—no standard deviations or confidence intervals are reported. A rigorous claim of "matching Transformer quality" would require multiple training runs or statistical tests.

Claim 3: "Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100× faster at sequence length 64K."

Supported, with hardware caveats. Figure 4.3 directly validates these numbers:

• At 8K: Hyena achieves ~2× speedup over FlashAttention, ~5× over standard attention.
• At 64K: Hyena achieves ~100× speedup over FlashAttention. Standard attention runs out of memory.

Qualifications:

1. The 100× figure is partly a memory capacity claim, not purely a computational one. FlashAttention cannot fit the attention matrix in GPU memory at 64K sequence length with batch size 64, so the comparison is between a method that runs (Hyena) and a method that crashes (FlashAttention). This is a legitimate practical advantage but should be distinguished from an algorithmic speedup.

2. Hardware utilization is lower for Hyena. The paper acknowledges this: "hardware utilization of Hyena is lower than FlashAttention." What this means in practice is that Hyena's theoretical FLOP advantage is larger than its actual wall-clock advantage, and the 100× speedup at 64K would be larger if Hyena achieved the same GPU utilization as FlashAttention. Conversely, if FlashAttention's hardware utilization were applied to an equivalently optimized Hyena kernel, the advantage would grow. The current numbers represent a lower bound on what optimized Hyena implementations could achieve.

3. The batch size is fixed at 64. Different batch sizes would shift the crossover points—larger batches favor Hyena more because attention's memory cost scales with batch size × L² while Hyena's scales more favorably.

Claim 4: "Rankings in our reasoning benchmark suite are predictive of quality at scale."

Supported with intriguing but limited evidence. Table C.1 shows a clean monotonic relationship between associative recall accuracy (at various vocabulary sizes) and THE PILE test loss for five models. The correlation is visually apparent—Conv1d (worst recall: 8% at vocab 40, worst perplexity: 4.21), Hyena (best recall: 85% at vocab 40, best perplexity: 2.59)—but the sample size is only 5 models and no formal correlation analysis is provided.

Strengths of the evidence:

1. The correlation holds across qualitatively different architectures (convolution-only, gated convolution, attention) and different parametrizations, increasing confidence that it is not an artifact of a single model family.
2. The vocabulary size scaling (accuracies diverge as vocabulary increases from 10 to 40) mirrors the gap that appears at scale on real data, suggesting the harder versions of the synthetic task are more diagnostic.
3. The paper reports similar correlations on other synthetic tasks (not shown in detail) and for vision tasks, suggesting the principle may generalize.

Weaknesses and missing evidence:

1. Sample size. Five models is suggestive but not statistically convincing. A proper validation would require evaluating 20+ architectures on both the synthetic benchmark and at scale, then computing a Spearman rank correlation with confidence intervals.
2. Direction of causality unclear. The correlation shows that architectures with good associative recall also achieve good language modeling perplexity. But this could mean (a) associative recall is a necessary capability for language modeling, or (b) architectures that are generally more expressive happen to do well on both tasks, or (c) the specific training setup (sequence length 2048, GPT2 tokenizer) creates a data distribution where associative recall is unusually important. To validate interpretability benchmarks as predictive tools, one would need to show that improving an architecture's performance on the synthetic task causes improvements at scale, not just that they are correlated across architectures. This would require a causal intervention (e.g., ablating a specific component that is known to affect recall, and measuring the impact on language perplexity).

Missing experiments that would have strengthened the paper:

1. Systematic sweep over order N. The paper introduces the Hyena hierarchy as a key conceptual contribution but provides almost no empirical characterization of how performance scales with N beyond noting that N=2 matches H3 and N=3 slightly improves WikiText103 perplexity. A sweep over N ∈ {1, 2, 3, 4, 5} on both synthetic and language tasks would test whether higher orders provide diminishing returns, saturate, or continue to improve—directly informing the theoretical claim that deeper decompositions are more expressive.

2. Ablation of individual filter design choices. The Hyena filter combines three innovations: (a) FFN parametrization, (b) exponential decay windowing, and (c) sinusoidal activations. The paper compares against CKConv (which has (a) but not (b) or (c)) and H3 (which has none of the three, using SSMs instead). A proper ablation would independently test Hyena minus decay, Hyena minus sinusoidal activations, and Hyena minus both, to isolate the contribution of each component. Without this, the reader cannot determine whether the FFN parametrization is doing most of the work (as the CKConv comparison suggests) or whether the decay and sinusoidal enhancements are critical (as the Hyena vs. CKConv gap at 131K suggests, with 97.2% vs. 14.3%).

3. Language modeling at longer sequence lengths (8K, 16K, 32K). The paper's efficiency argument centers on long sequences, and the runtime benchmarks show dramatic speedups at 8K+, but all language modeling experiments use sequence length 2,048. Training a Hyena model at sequence length 8K on PG-19 or THE PILE and comparing perplexity to a Transformer (or showing that Transformers cannot be trained at that length) would directly validate the claim that Hyena enables efficient long-context language modeling.

4. Validation on a second language modeling dataset with different properties. WIKITEXT103 and THE PILE are both English text corpora. Evaluating on a code dataset (where long-range dependencies are structural and exact matching matters), a multilingual corpus, or a domain-specific corpus (e.g., scientific papers) would test whether Hyena's advantages generalize beyond the specific data distribution on which it was tuned.

5. Statistical significance for perplexity differences. The paper reports perplexity to one decimal place without standard deviations or confidence intervals. Given that the differences between Hyena and Transformers are often 0.1 perplexity or less (e.g., 9.2 vs. 9.1, 18.6 vs. 18.6), it is possible that some of the "matching" results are within the noise of training stochasticity.

6. Combining Hyena with other Transformer innovations. The paper tests Hyena as a drop-in replacement for attention in an otherwise standard Transformer architecture. It does not explore whether architectural innovations developed for Transformers (e.g., rotary position embeddings, parallel attention/FFN, mixture-of-experts FFNs) transfer to Hyena or whether Hyena-specific innovations (e.g., different filter designs per layer, adaptive order selection) could further improve performance. This is understandable for an initial paper but limits the strength of the "attention-free" claim—perhaps Transformers with the latest innovations still outperform Hyena, and the matching results are against a somewhat outdated Transformer baseline.

7. Memory usage comparison, not just speed. Figure 4.3 shows runtime but not peak memory consumption. Given that memory is the primary bottleneck preventing Transformers from processing very long sequences, a memory vs. sequence length plot would be more informative than runtime for practitioners deciding whether to adopt Hyena.

6. Limitations and Trade-offs

6.1 Filter Design Choices Are Empirically Validated but Theoretically Underjustified

The assumption or constraint. Hyena's filter parametrization combines three distinct design choices: FFN-based implicit representation, exponential decay windowing with a learnable bias, and sinusoidal activation functions in the FFN. The paper motivates each choice individually—sinusoidal activations address the low-frequency bias of neural networks (Section 3.3, citing Basri et al., 2020), exponential decay mirrors findings from other applications (Li et al., 2022), and FFN parametrization provides sublinear parameter scaling—but it provides no theoretical framework for why this specific combination is necessary or sufficient. The paper states:

"we choose at least one of the convolutions in Hyena to be shaped towards exponential decay, mirroring the findings of (Li et al., 2022) in other applications"

and

"We use high-frequency periodic activations (sine) in the FFN. This allows (7) to learn filters with high-frequency content, addressing the low-frequency bias of neural networks (Basri et al., 2020)."

These citations provide motivation but do not constitute a proof that these choices are load-bearing, nor do they characterize the class of filters that the combined parametrization can represent compared to alternatives.

The consequence. A practitioner attempting to reproduce or adapt Hyena to a new domain lacks principled guidance on which filter design choices matter and which are incidental. Must all three components be present, or would any two suffice? The exponential decay window biases filters toward locality, which helps for language where nearby tokens are more relevant—but for domains where long-range dependencies are uniformly distributed (e.g., genomic sequences, certain types of time-series), this bias could be actively harmful. Without understanding the function class that each component enables or restricts, adapting Hyena requires expensive empirical trial-and-error rather than principled modification. Furthermore, the theoretical justification for why SSM-based filters collapse on associative recall at extreme lengths (0.6% at 131K in Table A.2) while FFN-based filters do not (97.2%) is only partially explained—the paper appeals to SSMs' "blurring" of information due to continuous-time dynamics, but does not formalize this as a bandwidth or representational-capacity limitation of the SSM function class. This makes it unclear whether future SSM variants (with different state matrix structures, different discretization schemes, or additional mechanisms) could close the gap without abandoning the SSM framework entirely.

What evidence exists in the paper. Figure 4.1 shows the overall ranking of parametrization methods within the same Hyena₂ recurrence, and Table A.2 shows the dramatic collapse of non-Hyena methods at 131K. However, the paper does not report individual ablations of each Hyena filter component. CKConv (which uses FFN parametrization without sinusoidal activations or exponential decay) achieves 14.3% at 131K—far below Hyena's 97.2% but far above H3's 0.6%—suggesting the FFN parametrization alone provides a substantial improvement over SSMs, and the decay windowing and sinusoidal activations provide an additional large gain. But this is a two-point comparison (CKConv vs. Hyena, H3 vs. Hyena) rather than a factorial ablation that isolates each component. No experiments vary the window function (e.g., replacing exponential decay with a flat window, a Gaussian window, or no window) or the activation function (e.g., ReLU, tanh, or learned activations).

Mitigation status. Not addressed. The paper does not acknowledge the absence of component-level ablations as a limitation, nor does it propose a theoretical analysis of the filter function class as future work. The discussion section (Section 5) focuses on scaling Hyena to larger models and longer sequences rather than on understanding which design choices are empirically load-bearing versus theoretically justified.

---

6.2 Language Modeling Validation Is at Sequence Length 2K, Mismatching the Efficiency Claims

The assumption or constraint. All of the paper's primary language modeling results—WIKITEXT103 (Table 4.3), THE PILE at 125M and 355M (Table 4.4), the scaling curves (Figure 4.2), and the SuperGLUE downstream evaluation (Tables 4.5, 4.6)—use a sequence length of 2,048 tokens. This is the same sequence length used by standard GPT training pipelines and enables direct comparison to existing work. The efficiency claims, however, emphasize gains at much longer sequences: "5x speedups over dense self-attention at length 8192," "100x speedup over FlashAttention at sequence lengths of 64K," and the vision of "using entire textbooks as context." These runtime measurements (Figure 4.3, Section 4.4) measure the forward pass speed of the operator in isolation, not end-to-end training throughput or model quality at those lengths.

The consequence. The paper's headline efficiency claims and the paper's language quality claims operate at different sequence length regimes that are never bridged. A practitioner evaluating Hyena for, say, processing 16K-token documents learns from Figure 4.3 that the operator is faster, but learns nothing from the language modeling experiments about whether a Hyena model trained with 16K context actually produces good perplexity or downstream performance. The PG-19 result (14.6 perplexity at 16K, mentioned in Appendix A.2) provides a single data point but no Transformer comparison, making it impossible to assess whether Hyena maintains Transformer quality at genuinely long context lengths or whether the gap that is negligible at 2K (0.1 perplexity at 355M) widens when attention's quadratic cost becomes the dominant computational burden, forcing architectural compromises in Transformer training that Hyena avoids. The paper's own analysis in Appendix C notes that Hyena's advantage on associative recall grows with sequence length (Table 4.2: 100% vs. 4.3% for H3 at 64K, 97.2% vs. 0.6% at 131K)—suggesting that a language modeling comparison at 8K or 16K context might show Hyena pulling ahead of Transformers, but this hypothesis is untested.

Additionally, the runtime benchmarks (Figure 4.3) measure a single forward pass of the Hyena operator vs. attention, not end-to-end training iteration time. Training involves backward passes (which for FFT-based convolutions may have different memory and compute characteristics than forward passes), optimizer steps, data loading, and communication overhead. The crossover points identified in Figure 4.3 (2K for standard attention, between 4K and 8K for FlashAttention) may shift substantially when full training throughput rather than isolated operator speed is measured. The paper acknowledges the hardware utilization gap:

"Despite the absolute reduction in FLOPs, speedups are achieved only on longer sequences when the gap grows sufficiently large. This occurs because hardware utilization of Hyena is lower than FlashAttention."

This suggests that the forward-pass speedup overstates the training throughput advantage, but the magnitude of this overstatement is not quantified.

What evidence exists in the paper. Section 4.2 and Tables 4.3–4.4 specify that training uses sequence length 2048 (or 2024 in one instance; Appendix A.2 says "sequence length 2024"). Figure 4.3 benchmarks operator runtime at lengths up to 64K. The PG-19 result at 16K (Appendix A.2) is a test-time evaluation of a model trained at 2K—it is not a training-length experiment. The gap between training length and benchmark length is documented but not discussed as a limitation.

Mitigation status. The paper does not address this mismatch. The PG-19 result is mentioned in Appendix A.2 without comparison to a Transformer baseline. Training Hyena at sequence length 8K or 16K on THE PILE or another corpus and comparing to a Transformer (or showing that the Transformer cannot be trained at that length due to memory constraints) is left entirely to future work. The discussion section (Section 5) expresses excitement about "what new capabilities Hyena opens up as we scale and optimize the inference speed of these models" but does not identify training-length validation as a gap.

---

6.3 The Cost of Difficulty Estimation for Language Modeling Is Not Characterized

In Section 5, the analysis noted that the paper's associative recall experiments show strong correlation with language modeling performance (Table C.1), raising the possibility that synthetic mechanistic benchmarks can serve as cheap proxies for expensive at-scale evaluation. However, the paper does not develop this into a practical difficulty-estimation or model-selection pipeline, and the cost of the associative recall experiments themselves—while tiny compared to language model training—is not optimized. This is a methodological limitation: the synthetic benchmarks are used post-hoc to explain Hyena's success, not as a tool to guide design decisions during development.

A deeper issue is that the paper provides no framework for estimating how much "long-range mixing capacity" a given Hyena configuration (order N, filter FFN size, decay rate) provides for a given sequence length. For Transformers, the cost of attention is a simple function of sequence length: O(L²D). A practitioner knows exactly how many FLOPs increasing sequence length by 2× will cost. For Hyena, the cost scales as O(N L log L), which is analytically simpler, but the effective memory length—how far back the model can reliably route information—is determined by the learned filter shapes, which depend in complex ways on the FFN capacity, the window decay rate, the sinusoidal activation frequencies, and the interaction between successive convolution-gating steps. A practitioner who needs to process sequences of length 32K cannot determine from the paper whether a Hyena₂ with filter FFN width 64 is sufficient, or whether they need Hyena₃ with width 128 and slower decay. This makes capacity planning difficult and may lead to either underprovisioning (deploying a model that cannot handle the required context) or overprovisioning (wasting compute on unnecessary filter capacity).

What evidence exists in the paper. The associative recall experiments (Figure 4.1, Table 4.2) sweep sequence length and vocabulary size but use a fixed model configuration (2 layers, width 64, filter FFN width 32). These experiments show that Hyena maintains performance at extreme lengths where other methods fail, but they do not systematically vary Hyena's own capacity parameters to establish scaling relationships between filter capacity and effective memory length. There is no experiment showing, for example, that increasing filter FFN width from 32 to 64 improves recall at 131K but not at 30K, or that order 3 provides longer effective memory than order 2. Table A.2 reports a single configuration at 131K without sweeping filter hyperparameters.

Mitigation status. The paper does not identify this as a limitation or propose a systematic study of filter capacity vs. effective context length. The discussion section focuses on scaling model size and inference speed rather than on developing tools for capacity planning.

---

6.4 Single Model and Benchmark Family Limits Generality Claims

The assumption or constraint. The paper's language modeling results are obtained using a specific model family (Hyena models trained from scratch, following GPT-style architectures) on a specific set of benchmarks (WIKITEXT103, THE PILE, SuperGLUE) with a specific tokenizer (GPT2, vocabulary 50,257). The associative recall experiments use synthetic data with specific properties (key-value pairs sampled from a fixed vocabulary, sequences of 2K–131K tokens, 2,000 training samples). The image classification experiments transfer Hyena to a Vision Transformer and to 2D convolutions on CIFAR-10. The paper's claims about generality—"Hyena operators and filters can also be applied successfully beyond language tasks" (Section 4.1)—are supported by these vision results, but the scope of validation is narrow relative to the breadth of tasks for which attention is used.

The consequence. The paper's core empirical claim is that Hyena matches Transformer quality. But "Transformer quality" is not a monolithic property—it encompasses performance across dozens of benchmarks, tasks, and domains. The paper's evaluation, while thorough for a single paper, covers only language modeling perplexity (a proxy metric that does not always correlate perfectly with downstream utility), eight SuperGLUE tasks (at small scale), and image classification on ImageNet-1k (with a standard ViT architecture, not the best available vision architecture at the time). Several important dimensions of Transformer behavior are unexamined:

• The paper shows that Hyena matches GPT on zero-shot and few-shot SuperGLUE, but does not evaluate on generation tasks (summarization, translation, dialogue) or on knowledge-intensive benchmarks (MMLU, TriviaQA) where Transformers' ability to route information from pretraining memories is critical. It is possible that Hyena's long convolution filters, which learn to route information based on position and content within the current sequence, handle in-context routing well but are less effective at retrieving information stored in model parameters from pretraining. The associative recall tasks test retrieval within a sequence, not retrieval from model memory.

• Multilingual performance is untested. THE PILE and WIKITEXT103 are English-language corpora. Transformers have been shown to work well across languages, including those with very different word orders and morphological structures (where the locality bias of the exponential decay window might be harmful). The paper's exponential decay window explicitly biases filters toward shorter-range dependencies:

  "we choose at least one of the convolutions in Hyena to be shaped towards exponential decay"

  This bias is motivated by findings in other applications (Li et al., 2022), but those applications were also in English. For languages with freer word order or long-distance syntactic dependencies (e.g., German subordinate clauses, Japanese honorific agreement), penalizing long-range connections through exponential decay could systematically degrade performance. No experiments probe this.

• The vision results (Table 4.7) are modest. Matching ViT accuracy on ImageNet-1k (78.5% for both) demonstrates that Hyena is not actively harmful for vision, but ViT at 87M parameters is a weak baseline by 2023 standards—state-of-the-art vision models at the time used hierarchical architectures (Swin, ConvNeXt) or larger scales. The 0.2% gap at the finer 8×8 patch size (79.8% vs. 80.0%) is small but directionally unfavorable. It is unclear whether this gap would widen with larger models, more training, or more challenging vision tasks (object detection, segmentation, video).

What evidence exists in the paper. The paper evaluates on WIKITEXT103, THE PILE, SuperGLUE (8 tasks), LAMBADA, PG-19, ImageNet-1k, and CIFAR-10/2D. This is a reasonable evaluation suite for an architecture paper, but it is not sufficient to support strong claims of domain generality. The paper does not evaluate on: code generation or understanding (HumanEval, MBPP), mathematical reasoning (GSM8K, MATH), factual knowledge retrieval (MMLU, NaturalQuestions), long-form generation, or any non-English benchmark. The SuperGLUE evaluation (Tables 4.5, 4.6) shows that Hyena slightly underperforms GPTNeo in zero-shot (41.5 vs. 43.8 average) and matches in few-shot (49.3 vs. 49.1)—but GPTNeo is a 125M model trained for 300B tokens, while Hyena is a 153M model trained for 137B tokens, making it unclear whether Hyena is more efficient or merely benefiting from 22% more parameters with half the training data.

Mitigation status. The paper does not claim universality—Section 5 expresses optimism about scaling but does not assert that Hyena matches Transformers across all domains. The image classification results are presented as a demonstration of potential rather than a comprehensive vision benchmark. However, the abstract's claim that Hyena constitutes a "drop-in replacement for attention" and the paper's framing as establishing that "attention may not be all we need" (Section 5) carry an implicit generality claim that the evaluation suite does not fully support. The paper does not identify the narrow domain coverage as a limitation or propose specific additional evaluations needed to validate generality.

---

6.5 The Implicit Filter Parametrization Incurs a Per-Input Generation Cost Not Present in Attention

The assumption or constraint. Hyena's long convolution filters are generated by passing a positional encoding of all time indices through a feed-forward neural network (Algorithm 2). This filter generation step must be performed for every forward pass of every Hyena layer, because the FFN that generates the filters has learned parameters that are part of the model. While the FFN output is not data-dependent (the filters depend only on position, not on the input content), it is computed at inference time and contributes to the overall cost. The paper's complexity analysis (Proposition 3.2) includes this cost implicitly in the O(NDL(log₂ L + D)) bound, and the runtime benchmarks (Figure 4.3) measure end-to-end operator speed including filter generation. However, the filter generation cost has different scaling properties than the convolution cost and may become a bottleneck in specific regimes.

The filter generation involves: (1) computing positional encodings for all L positions (cost: O(L D_e)), (2) passing them through a 4-layer FFN of width 64 (cost: O(L × D_e × 64 × 4) approximately, where D_e is the positional embedding dimension), and (3) multiplying by the window function (cost: O(L)). For very long sequences or very deep models (large N), this cost is amortized over the N convolutions, but for shallow models or short sequences where the FFT convolution cost is small, the filter generation can represent a non-trivial overhead.

The consequence. The per-forward-pass filter generation means that Hyena's efficiency relative to attention depends on the ratio of sequence length to model depth, not just on the asymptotic complexity. For very short sequences (e.g., L = 512 or L = 1024), the filter generation cost may be comparable to or larger than the FFT convolution cost, potentially making Hyena slower than attention even though attention's asymptotic scaling is worse. Figure 4.3 shows Hyena being slower than FlashAttention at L = 2048 (the crossover is between 4K and 8K), but this is an operator-level comparison—the filter generation is part of the measured Hyena time. The paper does not decompose the operator runtime into filter generation vs. convolution vs. gating, making it impossible for a practitioner to predict how the overhead scales in different deployment scenarios.

Additionally, in autoregressive decoding (where tokens are generated one at a time and the model is run repeatedly with growing sequence length), the filter generation cost may behave differently than in the training forward pass. During training, all L positions are processed in parallel, and the filter generation cost is amortized. During autoregressive decoding with a KV-cache-style mechanism, the filters would need to be either recomputed for the full sequence at each step (costly) or cached and extended (requiring memory proportional to sequence length for the filter values). The paper does not discuss how Hyena handles autoregressive inference efficiently—the FFTConv approach described in Section 3.4 assumes processing the full sequence in parallel, which is optimal for training but may not be optimal for auto-regressive generation where each new token requires computing the convolution output at only the last position. A causal convolution can be computed incrementally without FFT (using a direct recurrence with O(L) state, similar to how SSMs handle autoregressive generation), but this would have different constant factors and may not benefit from the FFT speedups. The paper provides no analysis of autoregressive inference latency or memory.

What evidence exists in the paper. Figure 4.3 measures end-to-end operator runtime including filter generation, and shows Hyena becoming faster than FlashAttention between 4K and 8K sequence length. The complexity analysis (Proposition 3.2) provides the asymptotic cost including filter generation but does not separate the terms. The architectural hyperparameters table (Table A.4) specifies filter FFN width = 64 and filter FFN depth = 4 for all model scales, but the positional embedding dimension D_e is not specified in the main text (the sinusoidal activation frequency is given as 14, but the embedding dimension is not). There is no decomposition of runtime into sub-operations, no analysis of autoregressive inference, and no study of how filter generation cost scales with model depth or sequence length relative to the convolution cost.

Mitigation status. Not addressed. The paper focuses on training-time efficiency and long-sequence forward-pass speed, not on autoregressive generation latency. The fused CUDA kernel for FFTConv (inherited from H3, Dao et al., 2022c) is mentioned but its interaction with filter generation is not detailed. Future work on efficient autoregressive Hyena inference (e.g., using recurrent state maintenance for causal convolutions, caching filter values, or computing filters lazily for only the positions needed) is not suggested.

---

6.6 Parameter Count vs. FLOP Tradeoffs Are Not Fully Explored

The assumption or constraint. The headline efficiency claim—"Hyena matches Transformer quality with a 20% reduction in training compute"—compares a Hyena model at 153M parameters against a GPT model at 125M parameters (Table 4.4, 5B tokens: both achieve 13.3 perplexity, Hyena FLOPs: 1.87 ×10¹⁹, GPT FLOPs: 1.88 ×10¹⁹). At the 355M scale, the comparison tightens: Hyena-2 355M achieves 9.2 perplexity vs. GPT's 9.1 at 15B tokens, with FLOPs of 3.93 vs. 4.77 ×10¹⁹. The 20% FLOP reduction is computed relative to the GPT baseline, but the Hyena model at this scale has the same parameter count (355M vs. 355M).

However, the smaller-scale comparison uses unequal parameter counts: 153M for Hyena vs. 125M for GPT. The paper explains this as a result of Hyena's different architecture (more layers, different width configurations), but it introduces a confound: Hyena may be achieving its matching perplexity partly through additional parameters rather than through superior architectural efficiency. At the 125M/153M scale, the FLOP reduction is essentially zero (1.87 vs. 1.88), meaning Hyena requires the same compute to train a model with 22% more parameters—which could be interpreted as Hyena being less parameter-efficient than attention for a given perplexity.

The consequence. A practitioner choosing between Hyena and a Transformer for a fixed compute budget needs to know: if I have C FLOPs to spend, what perplexity can I expect from Hyena vs. a Transformer? The paper's results at the 355M scale suggest a modest advantage (0.1 perplexity improvement for 18% fewer FLOPs), but this advantage may be partly or entirely explained by the fact that Hyena's architecture enables different depth-width configurations that are independently beneficial. The Hyena-slim architecture (Table 4.3: 18 layers vs. 12, with thinner FFNs) improves perplexity from 18.6 to 18.5 at 125M—a 0.1 gain from architectural restructuring that is not unique to Hyena (deeper, thinner Transformers can also outperform shallower, wider ones). The paper presents this as a benefit enabled by Hyena's lower per-layer FLOPs, which is legitimate, but it means the "20% reduction" conflates two effects: (1) cheaper sequence mixing per layer, and (2) the ability to restructure the architecture for better depth-width tradeoffs. A Transformer with similar depth-width restructuring but the same attention mechanism might also improve—the paper does not provide a depth-width-tuned Transformer baseline to isolate the contribution of Hyena's cheaper mixing from the contribution of better architectural hyperparameters.

Furthermore, the FLOP reduction is achieved only for the attention layers, not for the feed-forward layers. In a standard Transformer, attention accounts for roughly 30–50% of total FLOPs at sequence length 2K, with the FFN layers dominating the remainder. Hyena's 20% total FLOP reduction implies that attention FLOPs were reduced by roughly 40–60%, with the savings diluted by the unchanged FFN cost. At longer sequence lengths (8K+), attention's share of total FLOPs grows, and Hyena's relative advantage would increase—but as discussed in Limitation 6.2, those sequence lengths are not validated for training. At larger model widths (1.3B+), the FFN dominates even more (because FFN FLOPs scale as O(LD²) while attention FLOPs scale as O(L²D)), and Hyena's advantage shrinks further. The paper's preliminary 1.3B result (10.8 perplexity at 5B tokens, Appendix A.2) is reported without a Transformer comparison or FLOP count, making it impossible to assess whether the advantage persists at that scale.

What evidence exists in the paper. Table 4.4 provides the FLOP comparison at two scales. Table A.4 shows the architectural hyperparameters—Hyena models tend to be deeper with thinner FFNs than the corresponding GPT baselines (e.g., Hyena 355M: 36 layers, FFN width 2048; GPT architectures typically have fewer layers with wider FFNs). The FLOP accounting methodology (Appendix A.2) separates attention FLOPs from FFN FLOPs only implicitly—the total FLOP count is reported, not the breakdown. Figure 4.2 shows scaling curves for Hyena and GPT, but the x-axis is training tokens, not FLOPs, so the compute-efficiency comparison requires cross-referencing Table 4.4. There is no depth-width ablation for the GPT baseline to control for architectural restructuring effects, and no experiment that varies the ratio of attention FLOPs to FFN FLOPs to establish how the advantage scales with model dimensions.

Mitigation status. The paper acknowledges that the FLOP savings depend on model configuration in a specific but narrow way: "The FLOP reduction consists in the non-parametric FLOPs of SelfAttention devoted to attention matrix computation. The ratio of parametric to non-parametric FLOPs (and hence the gains) depend on the ratio of model width D and sequence length L used in training." This footnote (Section 4.2) explains the mechanism but does not characterize how the advantage scales with model size, sequence length, or architectural hyperparameters. A more comprehensive analysis—showing FLOP-to-perplexity curves for both architectures at multiple scales, or decomposing the FLOP savings into mixing vs. FFN components—would make the efficiency claim more actionable and reveal the regimes where Hyena is most (and least) beneficial.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the landscape of efficient sequence modeling by breaking the conceptual monopoly of attention as the only viable primitive for large-scale language models. Prior to Hyena, the field operated under a tacit assumption that while attention could be approximated or supplemented, it could not be replaced—every subquadratic method that approached Transformer quality did so through hybridization with some dense attention layers. H3 (Dao et al., 2022c) exemplified this: its strongest configuration included attention layers because its attention-free components alone could not close the gap. Hyena overturns this assumption by demonstrating that a pure convolutional architecture, with no attention layers whatsoever, matches GPT perplexity on THE PILE at 355M parameters (9.2 vs. 9.1, Table 4.4) and SuperGLUE few-shot accuracy (49.3% vs. 49.1%, Table 4.6). This is not an incremental refinement of an existing attention approximation—it is an architectural counterexample that falsifies the claim "attention is necessary for high-quality language modeling at scale."

The magnitude of the shift is best understood as a reframing of the research question rather than a paradigm change in model architecture. The paper does not argue that everyone should immediately replace Transformers with Hyena, and it does not claim Hyena outperforms attention across all settings—the matching results are at sub-billion parameter scale, at sequence length 2K, and on a specific set of benchmarks. Rather, the paper changes the terms of the debate. Before: "How can we make attention cheaper?" After: "What are the essential computational properties of any sequence-mixing operator, and what is the space of operators that realize them?" This is a shift from approximation (trying to preserve attention's form while reducing its cost) to abstraction (identifying the abstract capabilities attention provides and implementing them through different primitives). The paper's three guiding properties—data control, sublinear parameter scaling, unrestricted context—are the vocabulary of this new framing. Any operator that satisfies them is provisionally a candidate; any operator that fails one of them has an explainable capability gap. This framework reinterprets the failures of prior work: GSS failed on associative recall because its single gating interaction (Hyena₁) was insufficiently data-controlled; SSM-based H3 failed at extreme lengths because its parametrization lost exact token identity; explicit convolutions failed because they violated sublinear parameter scaling. These are not mysterious failures—they are predictable consequences of missing design properties.

The paper also resolves a specific methodological contradiction. Prior mechanistic interpretability work (Olsson et al., 2022) had shown that induction heads—circuits that perform associative recall—are critical to in-context learning in Transformers. Simultaneously, SSM-based models (S4, H3) struggled with associative recall despite achieving strong results on other long-range benchmarks. This created a puzzle: if induction heads are so important, why weren't SSMs failing more dramatically on language? And if SSMs were failing on associative recall, why did they still achieve non-trivial language modeling perplexity? The paper's synthetic benchmark suite, particularly the vocabulary-size scaling experiments, resolves this: at small vocabulary sizes (10), SSM-based methods perform reasonably well on associative recall because key-value pairs are close together and the SSM's "blurring" doesn't lose the information. As vocabulary size increases (20, 30, 40), the average distance between related tokens grows, and the SSM's representational limitations become catastrophic (H3: 92% → 60% → 13% → 10% across vocabulary sizes, Table C.1). Real language modeling uses large vocabularies (~50K tokens) and has token distances that depend on corpus statistics—the synthetic benchmarks with scaled vocabulary sizes capture a dimension of difficulty that fixed-vocabulary benchmarks miss. This explains why SSMs looked promising on existing benchmarks but failed to match Transformers at scale: the benchmarks were not probing the right difficulty regime.

The paper makes certain research directions substantially more attractive:

• Implicit filter parametrization as a first-class design dimension. Prior work on long convolutions focused on the state-space machinery (the structure of A, the HiPPO initialization, the discretization scheme) because those were the degrees of freedom in existing models. Hyena demonstrates that the filter function class matters more than these details within a gated architecture—a 77-point accuracy gap emerges from switching the filter parametrization alone (Figure 4.1). This redirects attention from state-space engineering to function approximation theory: what class of filters can represent sharp delays? What is the tradeoff between FFN capacity and filter expressivity? How does the activation function determine the frequency content of the learned filter?

• Mechanistic interpretability benchmarks as architecture search tools. Table C.1 shows that associative recall accuracy at various vocabulary sizes correlates with language modeling perplexity at scale. If this correlation holds more broadly (a hypothesis the paper does not fully validate but makes plausible), it transforms architecture development from a months-long, GPU-intensive process into a minutes-long, single-GPU process. A researcher could sweep dozens of filter parametrizations, gating structures, and window functions on synthetic recall tasks and select only the most promising candidates for full-scale training. This is methodologically analogous to how the Chinchilla scaling laws (Hoffmann et al., 2022) enabled researchers to predict optimal model sizes from small-scale experiments—except here the prediction is about architectural expressivity rather than scaling exponents.

• Attention-free architectures for long-context applications. The paper establishes that subquadratic operators can match attention at moderate sequence lengths (2K), which means the current generation of long-context models (which use attention with approximations or sparse patterns) could potentially be replaced entirely by convolutional architectures that are natively fast at any length. The 100× speedup at 64K (Figure 4.3) is not hypothetical—it is measured on real hardware for the same operator that achieves matching perplexity at 2K. This lowers the barrier for researchers and practitioners who want to work with very long sequences but are currently bottlenecked by attention's memory requirements.

Conversely, some research directions become less attractive:

• Hybridization as a necessary strategy. H3 and similar works proposed that the optimal architecture combines attention-free layers with some attention. Hyena shows this is not required—a pure attention-free architecture can match Transformers at the scales tested. Hybrid architectures may still be beneficial (the paper notes this in Appendix A.1: "if final downstream performance is the only metric of interest, improved results can be obtained by hybridizing our models similarly to H3"), but hybridization is no longer the only known way to close the quality gap. Research effort previously allocated to optimizing attention-sparsity ratios or layer placement in hybrid models can now focus on improving the attention-free components themselves.

• Kernelized or low-rank attention approximations for language modeling. The paper's results with Performer (26.8 perplexity on WIKITEXT103, Table 4.3) and Linear Attention (25.6 perplexity) reinforce the finding that methods which directly approximate the attention matrix through kernelization or low-rank factorization produce large quality gaps on language. Hyena achieves 18.6 perplexity on the same benchmark—an 8-point improvement that isn't bridged by scaling kernel parameters or rank. This suggests the gap is architectural, not quantitative, and that the kernel-approximation approach is fundamentally limited for language because it preserves a form (pairwise dot-product interactions) that is not the essential property of attention.

• State-space models as the default long-convolution parametrization. Before Hyena, S4 and its variants were the dominant approach to long convolutions in deep learning. Hyena shows that FFN-based implicit parametrization with sinusoidal activations and exponential decay windowing can dramatically outperform SSMs on tasks requiring exact token retrieval (97.2% vs. 0.6% at 131K, Table A.2). This does not make SSMs obsolete—they remain excellent for modeling physical dynamics and smooth signals—but it does mean that SSMs should not be the default choice for language or other discrete symbol-manipulation tasks. The burden of proof shifts: a new long-convolution model must justify its filter parametrization choice relative to Hyena's FFN approach, not relative to S4.

Follow-Up Research This Work Enables

Characterizing the filter function class: what can Hyena filters represent that SSM filters cannot? The paper provides an empirical demonstration that Hyena's FFN-based filters outperform SSM-based filters on associative recall, but the theoretical explanation is incomplete. The paper appeals to SSMs' "blurring" of information and the low-frequency bias of standard neural networks, but does not formalize the representational capacity of each filter class. A strong follow-up would: (1) analytically characterize the space of filters representable by an SSM with diagonal state matrix of dimension K (sums of K exponentials with learnable decay rates and magnitudes) versus the space representable by a 4-layer FFN with width 64 and sinusoidal activations, (2) measure the approximation error of each class for a target impulse (an exact delay) as a function of the delay distance, and (3) connect this approximation error to the observed associative recall degradation at extreme sequence lengths. The hypothesis is that SSM filters can approximate a delay at distance d with error that grows polynomially or exponentially in d, while FFN filters with sinusoidal activations can represent exact delays (via Fourier reconstruction of a delta function) with error that depends on the FFN capacity but not on d. A negative result—finding that a carefully designed SSM with sufficient state dimension can match Hyena's recall performance—would be equally informative, because it would clarify whether the gap is fundamental (SSMs cannot represent sharp delays at long range) or contingent (current SSM training procedures don't find the right parameters).

Scaling laws for Hyena order N and filter capacity. The paper introduces the Hyena hierarchy as a principled expressivity knob (increasing order N increases the length of the matrix decomposition and thus the class of representable data-controlled matrices), but provides minimal empirical characterization of how performance scales with N. The only comparison is between Hyena₂ and Hyena₃ on WIKITEXT103 (Table 4.3: 18.6 vs. 18.5 perplexity with slim variant), a 0.1 perplexity gain. A systematic study would train Hyena models at orders N ∈ {1, 2, 3, 4, 5} while controlling for total parameters (by adjusting width to compensate for the additional filter and projection parameters at higher N) and measure: (a) associative recall accuracy as a function of N and sequence length, (b) language modeling perplexity at a fixed training budget as a function of N, and (c) the effective memory length (the maximum distance at which information can be reliably routed) for each N. The hypothesis is that higher N provides diminishing returns for standard language modeling (where most dependencies are local and N=2 suffices) but becomes increasingly important for tasks requiring multi-hop reasoning or very long-range dependencies. A parallel study would sweep filter FFN width (currently fixed at 64, Table A.4) and depth (currently fixed at 4) to establish how filter capacity trades off against compute for a fixed N. This would produce the equivalent of "Chinchilla laws for Hyena"—guidelines for how to allocate parameters between the operator (projections, filter FFN) and the feed-forward layers.

Large-scale validation at sequence lengths where attention is infeasible. The paper's efficiency argument centers on long sequences, with 100× speedups demonstrated at 64K (Figure 4.3), but all language modeling quality experiments use sequence length 2K. A definitive follow-up would train Hyena and Transformer models at sequence length 8K or 16K on THE PILE (or a long-document corpus like PG-19 or Books3), measure training throughput and perplexity, and determine whether Hyena's perplexity advantage grows, shrinks, or stays constant as sequence length increases. The prediction from the associative recall results: at 2K, Hyena and Transformers perform similarly because 2K is within the range where both can route information effectively. At 8K, Transformers may begin to struggle because attention's quadratic cost forces compromises (smaller batch size, gradient accumulation, fewer layers) that degrade quality, while Hyena's subquadratic cost allows training with the same architecture at 8K as at 2K. If this prediction holds, it would shift the practical recommendation from "Hyena matches Transformers at 2K with 20% less compute" to "Hyena scales to context lengths where Transformers cannot be trained," which is a qualitatively stronger claim. The PG-19 result in Appendix A.2 (14.6 perplexity at 16K for Hyena 153M) is a starting point but lacks a Transformer baseline—completing this comparison is a high-value experiment.

Isolating the causal role of associative recall in language modeling performance. Table C.1 demonstrates a correlation between associative recall accuracy on synthetic tasks and language modeling perplexity for five architectures, but correlation is not causation. A rigorous causal test would: (1) take a single architecture (e.g., a specific Hyena configuration) and create variants that are identical except for a targeted modification known to affect recall but not other capabilities—for example, manipulating the exponential decay window parameter α to control the effective memory length, or ablating specific gating interactions by fixing some projection matrices to identity, (2) verify that the modification changes recall accuracy on the synthetic benchmark in the expected direction, (3) measure the impact on language modeling perplexity and on specific linguistic phenomena that require long-range recall (e.g., pronoun resolution, subject-verb agreement across clauses, document-level coherence). If reducing recall capability consistently degrades performance on long-range linguistic phenomena but not on local phenomena (bigram statistics, local syntax), it validates the paper's implicit claim that associative recall is a bottleneck capability that mechanistic benchmarks can diagnose. If reducing recall capability degrades all linguistic phenomena uniformly (including local ones), it may indicate that the manipulation affected general model quality rather than recall specifically. This experiment would transform the heuristic correlation in Table C.1 into a causal design principle.

Autoregressive inference efficiency and the recurrent formulation. The paper's training and forward-pass algorithms use FFT-based convolution, which processes all sequence positions in parallel—optimal for training but potentially suboptimal for autoregressive generation where each new token requires computing the convolution output at only the last position. A natural follow-up would develop and benchmark a recurrent formulation of Hyena for inference: since each causal convolution $y_t = (h * u)_t = \sum_{n=0}^t h_{t-n} u_n$ can be updated incrementally as $y_{t+1} = y_t + \sum_{n=0}^{t+1} h_{t+1-n} u_n$ (or more efficiently maintained via a state buffer if the filter has a parametric form that supports recurrence, like an SSM or a rational function), an optimized autoregressive Hyena could maintain a state of size proportional to the filter's effective length rather than recomputing the full FFT at each step. The specific experiment: measure the latency and memory of autoregressive generation for Hyena vs. a Transformer with KV-cache at sequence lengths up to 64K, decomposing the cost into filter maintenance, convolution updates, and gating. This would determine whether Hyena's efficiency advantage extends from training to deployment—critical for applications like long-form text generation, streaming transcription, or dialog systems with extended context. The negative result (Hyena is fast at training but no faster than a KV-cached Transformer at inference) would still be informative because it would scope the practical applicability of the method.

Combining Hyena with the revision and test-time compute strategies developed for Transformers. This paper establishes Hyena as a drop-in replacement for attention in standard Transformer architectures, but it does not explore whether techniques developed for improving Transformer output quality at inference time—sequential revisions, best-of-N sampling with verifiers, beam search against process reward models—transfer to Hyena or require modification. A natural extension would replicate the test-time compute scaling analysis for Hyena models on reasoning benchmarks (MATH, GSM8K): train a process reward model on Hyena outputs, compare beam search vs. best-of-N vs. sequential revision at varying compute budgets, and measure whether the difficulty-dependent optimal strategy differs from what is observed for Transformers. The hypothesis: because Hyena's data-controlled matrix has a different structure from attention's (factorized vs. dense pairwise), the verifier may need to be trained differently or may exhibit different over-optimization patterns. A negative result—finding that Hyena benefits less from test-time compute than Transformers—would suggest that the specific form of attention's data control (explicit pairwise similarity scoring) provides advantages for verifiability that Hyena's gated convolution structure does not replicate, refining our understanding of what attention provides beyond raw sequence-mixing capability.

Practical Applications and Downstream Use Cases

Long-document processing and retrieval-augmented generation. The most immediate practical application is preprocessing or encoding very long documents—entire legal contracts, scientific papers, books, or codebases—where the context length exceeds what Transformers can handle efficiently. At sequence length 64K, Hyena achieves 100× speedup over FlashAttention (Figure 4.3), meaning a document that would require minutes to encode with attention can be encoded in seconds. For a retrieval-augmented generation pipeline where thousands of documents must be embedded for similarity search, this speedup directly reduces infrastructure costs. The paper's PG-19 result (14.6 perplexity at 16K context) provides preliminary evidence that Hyena models can effectively use long context; a deployment architecture could use a Hyena encoder to chunk and embed documents at very long chunk sizes (reducing the number of chunks and thus the retrieval index size), with a standard Transformer decoder for generation. This hybrid approach would leverage Hyena's efficiency where it matters most (encoding) while preserving whatever benefits attention provides for generation quality, and the paper's matching perplexity at 2K suggests the encoder would not sacrifice representation quality.

On-device language models with large context windows. The paper's sublinear parameter scaling property means that model size is decoupled from context length, unlike Transformers where memory consumption grows quadratically with sequence length. A Hyena model deployed on a mobile device or edge accelerator could maintain a very long conversation history or document context without hitting memory limits. The 2× speedup at 8K over FlashAttention and 100× at 64K translates to lower latency and lower energy consumption per query, which are critical constraints for on-device deployment. The paper's SuperGLUE results (Tables 4.5, 4.6) show that Hyena models retain the few-shot learning capabilities that make large context windows useful—the model can be given many examples in its prompt and learn from them in-context, even with an attention-free architecture. A concrete scenario: an on-device email assistant that processes a user's entire inbox (potentially tens of thousands of emails) as context for summarization, prioritization, or drafting responses. A Transformer would be prohibitively slow and memory-intensive at this context length; Hyena makes it feasible.

Training data generation through long-context self-distillation. The paper's observation that Hyena's associative recall capability enables in-context learning on very long sequences (100% at 30K, Table 4.2) suggests a bootstrapping pipeline: train a Hyena model on a large corpus with moderate context length (e.g., 2K), then use it to generate high-quality training data by providing very long context windows (e.g., 16K of relevant preceding text) as conditioning for next-token prediction. The model's ability to route information across long distances means the generated tokens will be more coherent with distant context than what a shorter-context model could produce. These generated long-context sequences can then be used to train subsequent models (Hyena or otherwise) on longer-range dependencies. This is a form of self-distillation where the teacher model's long-context capability (enabled by Hyena's efficiency) is transferred to a student model that may have a simpler architecture. The 100% recall accuracy at 30K provides confidence that the teacher model will actually use the provided context rather than ignore it, which is a common failure mode of attention-approximated models at long range.

Genomic and biological sequence analysis. The paper's results on associative recall at extreme lengths and the preliminary 2D convolution results (CIFAR-10: 91.2% accuracy with 25% fewer parameters than S4ND, Table 4.7) suggest Hyena is well-suited for biological sequence data where context windows of 100K+ are routine. Whole-genome analysis, protein structure prediction from long sequences, and single-cell sequencing data all require modeling dependencies across very long sequences. The implicit filter parametrization's ability to learn filters with specific decay characteristics (exponential, but with learnable bias and sinusoidal components) maps naturally to biological systems that exhibit multi-scale dynamics—fast local interactions and slow long-range correlations. A domain-specific deployment could train Hyena on genomic data with sequence lengths of 100K–1M tokens, using the exponential decay window to provide an inductive bias toward the 1/f power spectrum commonly observed in biological sequences, while the sinusoidal activations capture periodic motifs (e.g., nucleosome positioning, CpG islands). The paper's demonstration that Hyena matches Transformer quality on language at 2K and maintains it on associative recall at 131K suggests that the architecture would transfer well to these extreme-length biological tasks without the catastrophic quality degradation observed in SSM-based models (H3: 0.6% at 131K, Table A.2).

When to Prefer This Method

The paper explicitly positions Hyena against dense attention (the standard Transformer operator) and against prior attention-free operators (GSS, H3, AFT, RWKV), and its results support clear decision rules:

• Prefer Hyena over dense attention (GPT/Transformer) when: (1) sequence length exceeds 2K tokens, where Hyena's subquadratic scaling provides a wall-clock speed advantage (Figure 4.3: crossover at 2K for standard attention, 4K–8K for FlashAttention); (2) training compute is constrained and the model is in the 125M–355M parameter range, where Hyena achieves matching perplexity with 0–18% fewer FLOPs (Table 4.4); (3) memory is the binding constraint rather than FLOPs, as at 64K where standard attention runs out of memory entirely; (4) the task requires in-context learning or associative recall over sequences longer than a few thousand tokens, where prior attention-free operators collapse (Table 4.2: H3 4.3% vs. Hyena 100% at 64K).

• Prefer Hyena over prior attention-free operators (H3, GSS, AFT, RWKV) when: (1) the task involves exact token retrieval over long distances (associative recall, induction, document QA with long context), as Hyena's FFN-based filter parametrization with exponential decay and sinusoidal activations provides a 50+ point accuracy improvement (Figure 4.1, Table 4.2); (2) vocabulary size is large (≥30), where SSM-based and explicit convolution methods degrade sharply (Table C.1); (3) the model must be purely attention-free without hybridization, as Hyena matches hybrid H3's perplexity (18.5–18.6 on WIKITEXT103, Table 4.3) without any attention layers; (4) parameter efficiency matters at long sequence lengths—Hyena's sublinear parameter scaling via implicit filter parametrization avoids the linear parameter growth of explicit FIR convolutions.

• Prefer dense attention over Hyena when: (1) the model scale exceeds 1.3B parameters, where the FFN dominates total FLOPs and Hyena's operator-level savings become negligible relative to total compute—the paper does not demonstrate a FLOP advantage at this scale; (2) sequence lengths are consistently under 2K, where optimized attention implementations (FlashAttention) are already faster due to higher GPU utilization, and Hyena's filter generation overhead makes it slightly slower (Figure 4.3: FlashAttention faster at 2K); (3) the application requires autoregressive generation latency to be minimized at short sequence lengths, where the FFT-based convolution approach is not optimized for incremental decoding and a recurrent formulation of Hyena for inference has not been developed; (4) maximum hardware utilization is critical regardless of asymptotic complexity—the paper acknowledges Hyena's GPU utilization is lower than FlashAttention's, and in throughput-bound settings with moderate sequence lengths, attention's higher utilization may outweigh Hyena's theoretical FLOP advantage.

• Prefer SSM-based approaches (S4, H3) over Hyena when: (1) the data exhibits smooth, continuous dynamics (e.g., physical simulations, audio, time-series forecasting), where SSMs' inductive bias toward exponential basis functions is beneficial rather than limiting, and where exact token retrieval is not required; (2) the sequence length is very long but the required memory is short and state-based recurrence is preferred over FFT convolution for autoregressive decoding—SSMs naturally support fast recurrent inference through their state-space formulation, while Hyena's autoregressive inference has not been optimized; (3) the SSM framework provides theoretical guarantees (e.g., the HiPPO theory for online function approximation) that are important for the application domain, and the empirical gap on associative recall is not relevant to the task.