Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

ArXiv: 2402.19427

🎯 Pitch

Griffin, a new hybrid architecture, matches the quality of the Llama-2 model despite being trained on over 6x fewer tokens. It achieves this by interleaving novel gated linear recurrent layers with efficient local attention, delivering over 4x higher inference throughput than a comparable Transformer. This demonstrates that recurrent models can now rival Transformers in both performance and hardware efficiency at scale.

1. Executive Summary

This paper introduces Hawk, a pure RNN model built around a novel Real-Gated Linear Recurrent Unit (RG-LRU), and Griffin, a hybrid model that interleaves RG-LRU-based recurrent blocks with local sliding-window attention. The models are evaluated on language modeling benchmarks using up to 14B-parameter variants trained on the MassiveText dataset, comparing against MQA Transformer baselines, Mamba, and Llama-2. The RG-LRU incorporates a recurrence gate that modulates the influence of the recurrent state—enabling the layer to approximately interpolate between discarding the input and preserving prior history (via the gating term $a^{cr_t}$ in Equation 4)—while remaining diagonal and element-wise for efficient execution. Griffin matches Llama-2's downstream performance despite being trained on over 6× fewer tokens, achieves over 4× higher throughput than an MQA Transformer during inference at 1B scale (Figure 1b), and extrapolates to sequence lengths significantly beyond those seen during training, establishing that gated linear recurrences combined with sparse local attention can rival or exceed Transformers on both quality and efficiency—though fixed-state recurrent models alone underperform Transformers on exact copying and retrieval tasks without fine-tuning (Section 6.2).

2. Context and Motivation

The Core Problem: Transformers Have a Quadratic Cost Bottleneck That Scaling Cannot Fix

The Transformer architecture (Vaswani et al., 2017) has become the dominant backbone for large language models, but it carries a fundamental computational burden that becomes increasingly severe as models are deployed on longer sequences. The problem is twofold.

First, during training, global self-attention requires every token to attend to every other token in the sequence. For a sequence of length $T$ , this costs $O(T^2 D)$ operations, where $D$ is the model dimension. While this quadratic cost can be absorbed for moderate sequence lengths (e.g., 2048 tokens) by parallelizing across many accelerators, it becomes prohibitive when scaling to the long-context regimes that many applications demand—document summarization, codebase understanding, multi-turn dialogue, and scientific literature analysis all require reasoning over tens or hundreds of thousands of tokens.

Second, during inference, the problem shifts from computational FLOPs to memory bandwidth. Transformers cache the key and value vectors for every previously generated token in a Key-Value (KV) cache. This cache grows linearly with sequence length: $2 \times N \times T \times H_k \times d_{head}$ , where $N$ is the number of layers, $T$ is the sequence length, $H_k$ is the number of key-value heads, and $d_{head}$ is the head dimension. For Multi-Head Attention (MHA), $H_k = H$ (the total number of heads) and $H \times d_{head} = D$ , so the cache size at layer $N$ , sequence length $T$ , and batch size 1 is $2 N T D$ . For a 7B-parameter model with 32 layers, a 2048-token sequence, and $D = 4096$ , this cache is already approximately 1 GB—comparable to the model parameters themselves. At 100K tokens, the cache dwarfs the model, and generating each new token requires loading the entire cache from high-bandwidth memory (HBM). Since the per-token computation is $O(LD)$ —tiny relative to the cache size—the decode step becomes memory-bandwidth bound, and latency is dominated by the time to transfer the KV cache from HBM to the vector memory (VMEM) where computation occurs.

Multi-Query Attention (MQA) (Shazeer, 2019) mitigates this by reducing $H_k$ to 1, shrinking the KV cache by a factor of $H$ (typically 32–64). However, the cache still scales linearly with $T$ . The asymptotic bottleneck remains: for sufficiently long sequences, the KV cache will always dominate. The authors frame this explicitly in Section 1:

"Additionally, the linear growth of the Key-Value (KV) cache with the sequence length makes Transformers slow during inference. Although Multi-Query Attention (MQA) partially mitigates this issue by reducing the cache size by a constant factor, the cache still grows linearly in sequence length."

This is not a mere implementation detail—it is a structural inefficiency inherent to any architecture that stores per-token state for all past positions. As language models are deployed in increasingly long-context settings, this inefficiency compounds.

Why This Problem Matters: The Growing Gap Between Training and Inference Efficiency

The practical implications of the Transformer's inference bottleneck are substantial and growing. Several trends make this gap increasingly urgent:

Long-context applications are proliferating. Systems like Gemini (Gemini Team Google, 2023) and GPT-4 (Achiam et al., 2023) are being deployed with context windows of 32K, 128K, or even 1M tokens. These enable novel capabilities—analyzing entire codebases, reasoning over full-length books, maintaining coherent multi-hour conversations—but they amplify the KV cache bottleneck. A model deployed with a 128K context window and generating tokens auto-regressively must, at each decode step, load a KV cache that is 64× larger than the cache for a 2K context. Even with MQA, this dominates inference cost.

Throughput directly impacts cost and feasibility. For batch inference pipelines—scoring candidate outputs for reinforcement learning from human feedback (RLHF), generating training data via rejection sampling, or serving production traffic—throughput (tokens per second) determines economic viability. The paper shows in Figure 1(b) that at 1B parameter scale with long sequences, Hawk and Griffin achieve over 4× higher throughput than an MQA Transformer. This difference translates directly to cost: a 4× throughput improvement means 4× fewer accelerators to serve the same workload, or 4× faster turnaround for batch jobs.

Latency matters for interactive applications. In real-time settings (chatbots, coding assistants, voice interfaces), the time between a user's query and the model's response—the end-to-end latency—determines user experience. During the decode phase, loading a large KV cache introduces unavoidable latency that scales with sequence length. The paper's latency analysis (Figure 4) shows that recurrent models maintain near-constant per-token decode time regardless of sequence length, while Transformers slow down proportionally as the cache grows.

The training-inference efficiency asymmetry is widening. During training, Transformers benefit from massive parallelism: the quadratic attention computation is distributed across hundreds or thousands of accelerators, and the sequence dimension is processed in parallel. During inference, the decode step is inherently sequential—each token depends on all previous tokens, so the computation cannot be parallelized across time steps. This creates an asymmetry where a model that trains efficiently can be surprisingly expensive to deploy, especially at scale. The authors note (Section 4.3, Figure 3) that Griffin's training speed is comparable to a Transformer at short sequences, but the Transformer slows down as sequence length grows while Griffin's runtime remains constant. This training-time advantage compounds with the inference-time advantage: recurrent models are faster to train on long sequences and faster to serve.

Prior Approaches and Where They Fall Short

The field has not been idle in addressing the Transformer's efficiency limitations. The paper situates itself within a rich landscape of alternatives, each with identifiable shortcomings.

State-Space Models (SSMs): Efficient but Underperforming at Language Modeling

The SSM family—including S4 (Gu et al., 2021a), S4D (Gu et al., 2022), S5 (Smith et al., 2022), and H3 (Dao et al., 2022b)—replaced attention with linear recurrences derived from continuous-time state-space representations. These models achieved strong results on the Long Range Arena benchmark (Tay et al., 2020), which tests long-sequence reasoning tasks, and showed particular promise for audio generation (Goel et al., 2022). The key insight was that linear recurrences can be computed efficiently either via convolution (exploiting the associative property of linear time-invariant systems) or via parallel prefix-sum algorithms (the associative scan).

However, SSMs faced a critical limitation when scaled to language modeling: their linear, time-invariant dynamics meant that the recurrence weights were fixed for all inputs. Every token in the sequence was processed identically regardless of its content. This is fundamentally at odds with the selective, context-dependent processing that makes attention so effective—in attention, the model can choose which past tokens to attend to based on their relevance to the current query. SSMs lacked this input-dependent selectivity, which limited their ability to perform the kind of precise token manipulation (copying, retrieval, in-context learning) that Transformers excel at. The authors acknowledge this implicitly when they note in Section 2.4 that their RG-LRU "does not use initialization inspired by the theory of orthogonal polynomials"—distancing themselves from the SSM tradition of carefully designed initialization schemes.

Mamba: Adding Selectivity but with a Different Gating Philosophy

Mamba (Gu and Dao, 2023), developed concurrently with this work, addressed the time-invariance limitation by introducing an input-dependent selection mechanism. In Mamba, the recurrence parameters (the A, B, and C matrices in the SSM notation) become functions of the input $x_t$ . This allows the model to selectively propagate or discard information based on content, bringing the recurrent architecture closer to the flexibility of attention.

The paper acknowledges Mamba as the strongest recurrent baseline at the time of writing (Section 3.2): "Mamba-3B... the strongest small recurrent model reported in the literature to date." However, the authors identify a specific architectural difference in the gating mechanism. Mamba's selection mechanism is described as comparable to the update gate of GRUs or the forget gate of LSTMs—it can interpolate between the previous hidden state and the current observation, allowing the model to "reset its state and forget any information it holds from the past" (Section 2.4, Gate behaviour). The RG-LRU's recurrence gate ( $r_t$ in Equation 1) operates differently: it modulates the effective recurrent weight $a_t = a^{c r_t}$ , where $a$ is a fixed learned parameter between 0 and 1, and $c = 8$ is a scalar constant. When $r_t \approx 0$ , $a_t \approx 1$ , and the recurrence becomes $h_t \approx h_{t-1}$ —the model preserves the previous state and ignores the input. When $r_t \approx 1$ , $a_t \approx a^c$ , and the recurrence operates like a standard LRU update—the model incorporates the new input while decaying the old state. The key distinction: the RG-LRU's gate is biased toward retention, making it easier for the model to preserve information across long gaps when inputs are uninformative. The authors state this explicitly:

"We believe the key role of this gate is to enable the model to achieve super-exponential memory by reducing the influence of uninformative inputs."

This is a subtly different design philosophy from Mamba's reset-oriented gating, and the paper's empirical results suggest it is effective: Hawk-3B outperforms Mamba-3B on downstream tasks despite being trained on half as many tokens (300B vs. 600B).

The Linear Recurrent Unit (LRU): A Clean Baseline Without Gating

The LRU (Orvieto et al., 2023b) was a systematic ablation study that stripped down SSMs to their essentials: a diagonal linear recurrence with carefully designed initialization (eigenvalues distributed uniformly on a circle to control the effective memory length). The LRU demonstrated that much of the complexity in SSMs (HiPPO initialization, normal-plus-low-rank parameterization) was unnecessary—a simple diagonal recurrence with the right initialization could match SSM performance on long-range tasks.

The RG-LRU can be seen as an extension of the LRU with two key additions: (1) gating on both the input and the recurrent weight, making the layer content-dependent, and (2) the use of real-valued rather than complex-valued recurrences. The authors note (Section 2.4) that "while using complex recurrences would lead to a more expressive layer... we found that complex recurrences were not beneficial for language modelling in practice, as also observed by Gu and Dao (2023)." This is a practically important finding: complex numbers increase the state size and computational cost (since complex multiplication requires 4 real multiplications and 2 real additions, versus 1 multiplication and 1 addition for a real diagonal recurrence), but the additional expressiveness does not translate to better language modeling performance.

Hybrid Models and Local Attention: Partial Solutions

The observation that recurrent models struggle with precise token-level retrieval tasks (copying, induction heads) while attention excels at them has motivated several hybrid approaches. The H3 model (Dao et al., 2022b) used SSMs for long-range context and a "shift-SSM" (a small convolution) to capture local patterns. RetNet (Sun et al., 2023) combined a linear recurrence with a multi-head attention-like gating mechanism. Mistral 7B (Jiang et al., 2023) used sliding-window attention as a drop-in replacement for full attention, reducing the KV cache to a fixed size.

Griffin's hybrid design—alternating two recurrent blocks with one local attention block in a fixed layered pattern—builds on this line of work but with a crucial empirical finding: a single local attention layer per three temporal-mixing blocks is sufficient to recover the copying and retrieval capabilities that pure RNNs lack. The paper shows (Section 6.2, Figure 6) that on the Selective Copying task, a 5-layer Griffin model with one local attention layer matches the learning speed of a full Transformer, while a pure Hawk model is "significantly slower." On the Induction Heads task, Griffin with local attention can perfectly extrapolate to sequences far beyond its training length, while a Transformer with RoPE cannot extrapolate at all. This suggests that the local attention in Griffin serves a specific functional role—precise short-range retrieval—that complements the recurrent blocks' role in long-range information transport, and that a small fraction of attention layers is sufficient to fulfill this role.

How This Paper Positions Itself

The paper positions Griffin and Hawk not as radical departures from existing architectures but as synthesis and systematic scaling of ideas that had been demonstrated only at small scale or on specialized benchmarks. The authors make several deliberate positioning choices:

They emphasize scaling, not novelty of individual components. The RG-LRU is presented as a novel layer, but the paper's contribution is primarily empirical: demonstrating that gated linear recurrences (Hawk) and recurrent-attention hybrids (Griffin) maintain power-law scaling between held-out loss and training FLOPs up to 14B parameters (Figure 1a), and that this scaling matches or exceeds Transformers. Prior work on SSMs and linear RNNs had shown promising results at small scale (100M–1B parameters) or on synthetic long-range tasks, but had not demonstrated that these architectures scale competitively with Transformers on standard language modeling benchmarks and downstream tasks at the multi-billion-parameter scale. The paper explicitly aligns itself with the scaling laws tradition (Kaplan et al., 2020; Hoffmann et al., 2022), showing that Hawk and Griffin follow the same log-linear relationship as Transformers—this is a claim about the viability of recurrent models as a replacement for Transformers at scale, not just about a clever new layer.

They prioritize hardware efficiency as a first-class concern. The paper devotes an entire section (Section 4) to the engineering challenges of training diagonal RNNs efficiently on TPUs, and another (Section 5) to inference latency and throughput. This reflects the reality that an architecture's theoretical properties matter only insofar as they can be realized on real hardware. The custom Pallas kernel for the RG-LRU scan, the block-diagonal gate parameterization for Megatron-style sharding, and the analysis of memory-boundedness during decoding are all motivated by the observation that "diagonal RNN layers are memory bound" (Section 4.2) and require hardware-aware optimization to be competitive. The authors are explicit that their efficiency claims are hardware-specific ("we focus on developing an efficient implementation tailored to this device"—i.e., TPU-v3) but the principles (minimizing memory transfers, keeping the hidden state in VMEM) are general.

They acknowledge limitations in copying and retrieval. Rather than claiming that recurrent models solve all problems, the paper is candid about weaknesses. In Section 6.2, they show that pre-trained Hawk models fail on the phonebook lookup task for longer phonebooks (similar to Mamba's failure reported by Jelassi et al., 2024), while pre-trained Griffin models succeed up to the local attention window size. The authors state plainly: "more work is needed to improve these capabilities for such models." This intellectual honesty strengthens the paper's credibility and frames the hybrid design as a pragmatic compromise rather than a final solution.

They benchmark against strong, contemporary baselines. Rather than comparing against weak or outdated Transformers, the paper uses an MQA Transformer baseline with the same gated MLP block, same residual structure, same training data, and same hyperparameter tuning budget as Hawk and Griffin. Comparisons to Mamba-3B and Llama-2 (Table 1) are done with the caveat that these external models were trained on different data and with different tuning strategies, but the paper still claims—and the results support—that Hawk and Griffin are competitive or superior despite being trained on substantially fewer tokens. This positions the work as directly relevant to practitioners choosing architectures for their next training run, not just as an academic exploration of an alternative paradigm.

3. Technical Approach

3.1 Reader Orientation

This is primarily an architecture design and empirical scaling paper. The core idea is to build a language model family where the temporal-mixing component—the part that aggregates information across token positions—is a gated linear recurrence with a fixed-size hidden state rather than self-attention with a growing KV cache, and to show that a hybrid version that adds a small number of local attention layers can match or exceed Transformer quality while eliminating the quadratic and linear scaling bottlenecks at inference time.

3.2 Big-Picture Architecture (Diagram in Words)

All models in the paper share the same outer skeleton. The system has five major components:

Token Embedding Layer — maps input token IDs to vectors of dimension $D$ , shared with the final output projection (weight tying).
Residual Block (stacked $N$ times) — the fundamental repeating unit. Each block contains two sub-components applied sequentially: a temporal-mixing block (attention, recurrence, or local attention) followed by an MLP block, with RMSNorm before each sub-component and skip connections around both.
Temporal-Mixing Block — the component that differentiates the three model families. It can be global MQA (Transformer baseline), a recurrent block (Hawk), or a mixture of recurrent and local MQA blocks (Griffin).
MLP Block — a gated GEGLU-style feed-forward network with expansion factor $M = 3$ , identically used across all model variants.
Final RMSNorm + Output Projection — normalizes the final hidden states and projects to vocabulary logits via the tied embedding matrix.

Information flows as follows: token IDs → embedding lookup → $N$ residual blocks (each: RMSNorm → temporal-mixing → add skip → RMSNorm → MLP → add skip) → final RMSNorm → linear projection → softmax → token probabilities.

3.3 Roadmap for the Deep Dive

First, the residual block and MLP block — shared across all models — to establish the common backbone before introducing the differentiating components.
Second, the RG-LRU layer — the core mathematical innovation — because both Hawk and Griffin depend on it as their primary recurrent mechanism.
Third, the recurrent block that wraps the RG-LRU, including the Conv1D and gating structure, since it is the temporal-mixing component for Hawk and half of Griffin's temporal-mixing.
Fourth, the MQA and local attention blocks — the Transformer baseline and Griffin's auxiliary mechanism — since understanding Griffin requires seeing how recurrence and attention are interleaved.
Fifth, the complete model architectures (Hawk, Griffin, MQA Transformer), including the layered structure, hyperparameter choices, and parameter-count matching strategy.
Sixth, the hardware-efficient implementation — the custom Pallas kernel, the sharding strategy, and the rationale for a linear scan over associative scan — since these engineering decisions determine whether the theoretical efficiency advantages materialize in practice.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architecture design and empirical scaling paper whose core idea is that gated linear recurrences with fixed-size hidden states can serve as a drop-in replacement for self-attention in large language models, and that a hybrid design incorporating a small fraction of local attention layers recovers the precise token-manipulation capabilities that pure recurrent models lack, all while maintaining or exceeding Transformer quality at scale.

The Residual Block (Shared Backbone)

The residual block is the repeating unit from which all models are constructed. It defines the macro-level architecture: how the temporal-mixing and feed-forward components are ordered, normalized, and connected.

Structure. Each residual block contains two sequential sub-components, both wrapped with pre-normalization and skip connections:

Temporal-mixing sub-component: Input $x$ → RMSNorm → TemporalMixingBlock → output added to $x$ via skip connection.
MLP sub-component: Result from step 1 → RMSNorm → MLPBlock → output added to the result from step 1 via skip connection.

This is a standard pre-norm Transformer pattern (Xiong et al., 2020), adapted to accommodate recurrent temporal-mixing blocks instead of attention. The choice of pre-norm (RMSNorm before each sub-component, rather than after) is motivated by training stability at scale—it prevents the residual stream from accumulating unbounded variance across layers.

Stacking. $N$ such blocks are stacked sequentially. After the final residual block, a terminal RMSNorm is applied to produce the final activations. These are projected to vocabulary logits via a linear layer whose weights are tied to the input embedding matrix (weight tying), which reduces the parameter count and acts as a regularizer.

Design rationale. The shared residual structure ensures that all differences between model families are attributable solely to the temporal-mixing block. The MLP, normalization, skip connections, and output projection are identical across the MQA Transformer baseline, Hawk, and Griffin. This is a deliberate experimental design choice that enables clean ablation: any performance difference between models can be traced to the temporal-mixing mechanism.

The Gated MLP Block

The MLP block uses a gated architecture inspired by GEGLU (Shazeer, 2020), which the paper generalizes by adding a final output projection.

Structure. Given an input of dimension $D$ :

Two parallel linear projections: The input is passed through two independent linear layers, each producing an output of dimension $M \cdot D$ , where $M$ is the expansion factor. Throughout this work, $M = 3$ . So each branch projects from dimension $D$ to dimension $3D$ .
Non-linearity on one branch: A GeLU activation (Hendrycks and Gimpel, 2016) is applied to one of the two branches.
Element-wise multiplication (gating): The two branches are merged by element-wise multiplication. The GeLU-activated branch acts as a gate—it can selectively suppress or amplify the output of the non-activated branch at each dimension. This is the "GeGeLU" operation: two linear projections, one gated through GeLU, multiplied element-wise.
Output projection: The result (dimension $3D$ ) is projected back to dimension $D$ via a final linear layer.

Why gating? The element-wise multiplication allows the network to learn input-dependent filtering within the MLP. Unlike a standard two-layer MLP (linear → activation → linear), where the activation function is fixed and position-independent, the gated variant lets the model dynamically control which features propagate forward. The GeLU branch can output values near zero to suppress a feature dimension, near one to pass it through unchanged, or intermediate values for partial suppression. This is conceptually similar to the gating mechanisms in LSTMs and GRUs, but applied in the feed-forward rather than the recurrent pathway.

Expansion factor $M = 3$ . The paper uses a fixed expansion factor of 3 throughout all experiments and model scales. This is lower than the typical value of 4 used in many Transformer implementations (e.g., GPT-3, Llama). The authors do not explicitly justify this choice in the main text, but it is a hyperparameter that trades off MLP capacity against parameter count and computational cost. Using $M = 3$ rather than $M = 4$ reduces the MLP's parameter count by approximately 25% relative to the standard expansion, which partially compensates for the recurrent block's slightly higher parameter count (discussed below).

Parameter count. A standard two-layer MLP with expansion factor $M$ and no gating has $2 \cdot M \cdot D^2$ parameters (input projection $D \rightarrow MD$ plus output projection $MD \rightarrow D$ ). The gated MLP adds a third projection (the second branch of the input), bringing the total to $3 \cdot M \cdot D^2$ parameters, or equivalently $2 \cdot M \cdot D^2$ for the two input projections plus $M \cdot D \cdot D = M \cdot D^2$ for the output projection. With $M = 3$ , this is $6 D^2 + 3 D^2 = 9 D^2$ parameters, compared to $8 D^2$ for a standard MLP with $M = 4$ (though the gated variant does not exactly match either standard configuration since the computation is different).

The RG-LRU Layer: Core Mathematical Innovation

The Real-Gated Linear Recurrent Unit (RG-LRU) is the central contribution of the paper. It is a diagonal, real-valued linear recurrence augmented with two learned gates: an input gate that controls how much of the current input enters the state, and a recurrence gate that modulates the effective decay rate of the previous state.

Equation-by-Equation Breakdown

Recurrence gate:

$r_t = \sigma(W_a x_t + b_a)$

where $x_t \in \mathbb{R}^{D_{RNN}}$ is the input at timestep $t$ (already projected from the model dimension $D$ to the recurrent dimension $D_{RNN}$ by an upstream linear layer), $W_a \in \mathbb{R}^{D_{RNN} \times D_{RNN}}$ is a learned weight matrix (in practice, block-diagonal with 16 blocks—see Section 4.1), $b_a \in \mathbb{R}^{D_{RNN}}$ is a learned bias vector, and $\sigma(\cdot)$ is the element-wise sigmoid function, squashing each component to $(0, 1)$ .

What it computes: For each of the $D_{RNN}$ dimensions of the recurrence, the gate $r_t$ produces a scalar between 0 and 1. When $r_t \approx 1$ , the gate is "open"—the recurrence will behave closer to the standard LRU update. When $r_t \approx 0$ , the gate is "closed"—the recurrence will preserve the previous state and ignore the input. The gate is content-dependent: it is a learned function of the input $x_t$ , meaning the model can learn to close the gate on uninformative tokens (e.g., punctuation, filler words) and open it on information-bearing tokens.

Why this form: The sigmoid nonlinearity guarantees $r_t \in (0, 1)$ , which ensures that the effective recurrent weight $a_t = a^{c r_t}$ (defined below) remains well-behaved. A hard threshold or a ReLU gate could push $a_t$ to exactly 1 or exactly 0, which would make the recurrence either non-decaying (unstable) or memoryless (losing all past information). The smooth sigmoid allows the model to learn intermediate retention levels.

Input gate:

$i_t = \sigma(W_x x_t + b_x)$

where $W_x \in \mathbb{R}^{D_{RNN} \times D_{RNN}}$ is a second learned weight matrix (also block-diagonal with 16 blocks), $b_x \in \mathbb{R}^{D_{RNN}}$ is a learned bias, and $\sigma(\cdot)$ is again the sigmoid function.

What it computes: For each dimension, $i_t$ produces a scalar between 0 and 1 that gates the contribution of the input $x_t$ to the state update. When $i_t \approx 1$ , the input passes through at full strength. When $i_t \approx 0$ , the input is effectively zeroed out—the state update will be dominated by the decayed previous state. This is directly analogous to the input gate in LSTMs (Hochreiter and Schmidhuber, 1997).

Why this form: The input gate provides a second, independent mechanism for the model to control information flow. The recurrence gate $r_t$ controls how quickly the state decays; the input gate $i_t$ controls how much new information enters. Having both gates gives the model fine-grained control: it can, for example, close the input gate on a token it wants to ignore while still maintaining a moderate decay rate, or it can open the input gate fully while also opening the recurrence gate to overwrite old state with new information.

Effective recurrent weight:

$a_t = a^{c r_t}$

where $a \in \mathbb{R}^{D_{RNN}}$ is a learned parameter vector with each component in $(0, 1)$ (parameterized as $a = \sigma(\Lambda)$ with $\Lambda \in \mathbb{R}^{D_{RNN}}$ learnable), $c = 8$ is a fixed scalar constant, and $r_t$ is the per-dimension recurrence gate value from Equation 1. The exponentiation $a^{c r_t}$ is computed element-wise.

What it computes: This equation modulates the base decay rate $a$ by the content-dependent gate $r_t$ . When $r_t \approx 0$ , $a_t = a^0 = 1$ —the state does not decay at all, and the previous hidden state $h_{t-1}$ is preserved unchanged. When $r_t \approx 1$ , $a_t = a^c$ , which is the base decay rate raised to a fixed power—the state decays more aggressively. The constant $c = 8$ amplifies the effect of the gate: since $a \in (0.9, 0.999)$ at initialization (see below), $a^c \in (0.9^8, 0.999^8) \approx (0.43, 0.992)$ , giving a wide range of possible decay rates.

Why this form: The exponentiation by $c r_t$ gives the gate an exponential rather than linear effect on the decay rate. To see why this matters, consider that the effective memory timescale of a linear recurrence is $1 / (1 - a_t)$ . A small change in $a_t$ near 1 produces a large change in the timescale. The factor $c = 8$ stretches the sigmoid output $r_t \in (0, 1)$ to $c r_t \in (0, 8)$ , giving the gate access to decay rates spanning several orders of magnitude in effective memory length. In practice, the recurrence gate is computed in log-space for numerical stability (Appendix A):

$\log a_t = \log \sigma(\Lambda)^{c r_t} = -c \cdot \text{softplus}(\Lambda) \odot r_t$

where $\text{softplus}(\Lambda) = \log(1 + e^\Lambda)$ , and the identity $\log \sigma(\Lambda) = -\text{softplus}(-\Lambda)$ is used (with a sign flip due to the parameterization). Computing in log-space avoids the numerical issue of raising a number near 0 or 1 to a potentially large power in floating-point arithmetic.

State update:

$h_t = a_t \odot h_{t-1} + \sqrt{1 - a_t^2} \odot (i_t \odot x_t)$

where $h_t \in \mathbb{R}^{D_{RNN}}$ is the hidden state at timestep $t$ , $h_{t-1}$ is the previous hidden state, $a_t$ is the effective recurrent weight from Equation 3, $i_t$ is the input gate from Equation 2, $x_t$ is the current input, and $\odot$ denotes element-wise multiplication. All operations are element-wise—the recurrence is diagonal, meaning each dimension of the state evolves independently based only on its own past value and its own input.

What it computes: This is a gated linear recurrence that updates the hidden state by interpolating between the decayed previous state $h_{t-1}$ and the gated input $i_t \odot x_t$ . The interpolation is controlled by $a_t$ : when $a_t$ is close to 1, the previous state dominates and the input is nearly ignored (the coefficient $\sqrt{1 - a_t^2} \approx 0$ ). When $a_t$ is close to 0, the previous state is nearly erased and the input dominates (the coefficient $\sqrt{1 - a_t^2} \approx 1$ ). The factor $\sqrt{1 - a_t^2}$ is a norm-preserving weight: if the previous state and the gated input are orthogonal and unit-norm, the output will also be unit-norm. This is not strictly required for correctness but provides a useful inductive bias that prevents the state norm from exploding or vanishing.

Why this form: The coupling between the decay coefficient and the input coefficient via $\sqrt{1 - a_t^2}$ is a deliberate design choice inherited from the LRU (Orvieto et al., 2023b). It enforces an energy-conservation property: the sum of squared coefficients is $a_t^2 + (\sqrt{1 - a_t^2})^2 = a_t^2 + (1 - a_t^2) = 1$ . This means the state update is a rotation-like operation in the 2D plane spanned by $(h_{t-1}, i_t \odot x_t)$ for each dimension, which prevents the state from accumulating unbounded magnitude over long sequences. An alternative formulation $h_t = a_t h_{t-1} + (1 - a_t)(i_t \odot x_t)$ (linear interpolation) would not have this property and could allow the state norm to drift.

Contrast with Mamba's gating. Mamba's selection mechanism (Gu and Dao, 2023) uses an update-gate-like formulation: the effective recurrence is $\bar{A}(x_t) h_{t-1} + \bar{B}(x_t) x_t$ , where both $\bar{A}$ and $\bar{B}$ are input-dependent and there is no coupling between the decay and input coefficients. The RG-LRU's coupling via $\sqrt{1 - a_t^2}$ is more constrained—it prevents the model from simultaneously decaying the state heavily AND injecting a large input, which could cause instability. The paper argues (Appendix A, Figure 7) that this constraint is beneficial for language modeling because it biases the model toward preserving information rather than aggressively overwriting it.

Initialization. The weight matrices $W_a$ and $W_x$ are initialized using LeCun init (LeCun et al., 2002), which scales weights inversely with fan-in to maintain unit variance in activations. The parameter $\Lambda$ is initialized such that $a^c = \sigma(\Lambda)^c$ is uniformly distributed between 0.9 and 0.999 at the start of training. This is a deliberate choice: at initialization, the recurrence has a long memory (effective timescales ranging from $1 / (1 - 0.9) = 10$ steps to $1 / (1 - 0.999) = 1000$ steps), and the gate $r_t$ (which starts near 0.5 for all inputs due to random initialization of $W_a$ ) can modulate this timescale up or down. The paper explicitly distances itself from the SSM tradition of orthogonal polynomial initialization (Gu et al., 2020):

"Unlike many recent works in the SSM literature, the RG-LRU does not use initialization inspired by the theory of orthogonal polynomials, and it also is not defined as the discretization of an underlying continuous system."

This is an important architectural statement: the RG-LRU is designed as a discrete-time, real-valued gated recurrence from first principles, not as a discretized continuous-time SSM. The authors found this simpler formulation to be sufficient for language modeling.

Why real-valued rather than complex-valued. The LRU (Orvieto et al., 2023b) used complex diagonal recurrences because complex eigenvalues can represent oscillatory dynamics (via the phase $\theta$ ), which are useful for certain long-range tasks. However, the paper notes:

"We found that complex recurrences were not beneficial for language modelling in practice, as also observed by Gu and Dao (2023)."

The complex-valued variant of the RG-LRU (the CG-LRU, defined in Appendix B) splits the input into real and imaginary parts, uses complex exponential recurrences $a_t = \sigma(\Lambda) e^{i \theta}$ , and applies a complex version of the norm-preserving update. The decision to use real recurrences is pragmatic: it halves the effective state size at the same $D_{RNN}$ , reduces the parameter count (no $\theta$ parameters), and simplifies the computation (real multiplication vs. complex multiplication). For language modeling, the additional expressiveness of oscillatory dynamics does not appear to be worth the added cost.

The Recurrent Block (Temporal-Mixing for Hawk)

The recurrent block wraps the RG-LRU into a complete temporal-mixing component that replaces the attention block in Hawk. It follows a structure similar to the GSS block (Mehta et al., 2022) and Mamba's block (Gu and Dao, 2023), but with specific design choices detailed below.

Structure. Given an input of dimension $D$ (the model dimension):

Two parallel linear projections: The input is passed through two independent linear layers, each projecting from dimension $D$ to dimension $D_{RNN}$ . This creates two branches: the "recurrent branch" and the "gate branch."
Recurrent branch: On the first branch, a small separable Conv1D layer is applied, followed by the RG-LRU.
- The Conv1D has a temporal filter width of 4, meaning it convolves each channel with a learned kernel of length 4 across the time dimension. It is "separable": each of the $D_{RNN}$ channels is convolved independently (no cross-channel mixing), so the total parameter count is $4 \cdot D_{RNN}$ . This is very small—the authors emphasize: "this Conv1D layer is very small, with just $4 D_{RNN}$ parameters."
- The Conv1D is inspired by the Shift-SSM in H3 (Dao et al., 2022b). Its role is to capture very local temporal patterns (within a window of 4 tokens) before the recurrence, providing a short-term context that the purely diagonal RG-LRU cannot model (since the RG-LRU has no cross-dimension or cross-time interaction beyond the element-wise recurrence).
Gate branch: On the second branch, a GeLU nonlinearity is applied. No additional learnable parameters—this is just an activation function applied element-wise to the linearly projected input.
Element-wise multiplication (gating): The output of the recurrent branch (after Conv1D and RG-LRU) is multiplied element-wise by the output of the gate branch (after GeLU). This is the same gating pattern as the MLP block: the GeLU-activated branch can selectively amplify or suppress each dimension of the recurrent output.
Output projection: The result (dimension $D_{RNN}$ ) is projected back to dimension $D$ via a final linear layer.

Why this structure? The dual-branch design with element-wise gating gives the block the ability to learn which dimensions of the recurrent state are relevant for the current token. The gate branch receives the same input as the recurrent branch but processes it through a simple nonlinearity rather than through the recurrence—it acts as a "readout gate" that can suppress dimensions of the recurrent output that are not useful for the current prediction. This is analogous to the output gate in LSTMs, but applied after the recurrence rather than as part of the state update.

Why the small Conv1D? The diagonal RG-LRU processes each dimension independently—dimension $i$ of $h_t$ depends only on dimension $i$ of $h_{t-1}$ and $x_t$ . This means the recurrence has no mechanism to detect patterns that span multiple dimensions at short timescales. The Conv1D with temporal width 4 partially alleviates this by mixing information across 4 consecutive timesteps before the recurrence. However, because the Conv1D is separable (no cross-channel mixing), it only captures temporal patterns within each channel, not cross-channel patterns. The paper's design philosophy appears to be that the recurrence handles long-range cross-channel interactions (via the linear projections before and after the RG-LRU, which are dense), while the Conv1D handles short-range temporal smoothing.

Recurrent dimension $D_{RNN}$ . The paper sets $D_{RNN} \approx 4D / 3$ , where $D$ is the model dimension. This expansion factor is chosen to approximately match the parameter count of a Multi-Head Attention (MHA) block when both use the same model dimension $D$ . The authors note (Section 3, Hawk):

"We expand the width of the recurrent block by a factor of approximately $\frac{4}{3}$ (i.e. $D_{RNN} \approx 4D/3$ ) in order to roughly match the parameter count of a MHA block when both use the same model dimension $D$ ."

A footnote clarifies that this parameter matching is against MHA, not MQA, even though the Transformer baseline uses MQA for inference efficiency. This means the recurrent block has slightly more parameters than the MQA block it replaces, since MQA has fewer key-value head parameters than MHA. The choice to match against MHA rather than MQA is conservative: it ensures that any performance advantage of Hawk over the Transformer baseline is not simply due to having more parameters.

Parameter count of the recurrent block. The two input linear layers contribute $2 \cdot D \cdot D_{RNN}$ parameters each. The Conv1D contributes $4 \cdot D_{RNN}$ . The RG-LRU gates ( $W_a$ , $W_x$ ) contribute $2 \cdot D_{RNN}^2 / 16$ each (since they are block-diagonal with 16 blocks—see Section 4.1). The output linear layer contributes $D_{RNN} \cdot D$ . With $D_{RNN} \approx 4D/3$ , the dominant terms are $2 \cdot (2 \cdot D \cdot 4D/3) + (4D/3) \cdot D = 16D^2/3 + 4D^2/3 = 20D^2/3 \approx 6.67 D^2$ , compared to $4D^2$ for a standard MHA block (query, key, value, output projections) and $2D^2 + 2 \cdot D \cdot (D/H) \cdot H = 2D^2 + 2D^2 = 4D^2$ for an MQA block (query projection + key/value projections with $H_k = 1$ ). So the recurrent block is indeed closer in parameter count to MHA than to MQA.

Multi-Query Attention and Local Attention (Temporal-Mixing for Transformers and Griffin)

The paper uses two attention variants: global Multi-Query Attention (MQA) for the Transformer baseline, and local (sliding-window) MQA for the attention layers in Griffin.

Global Multi-Query Attention

Structure. MQA (Shazeer, 2019) is a variant of Multi-Head Attention (MHA) where all query heads share a single set of key and value heads. Specifically:

The input of dimension $D$ is projected to $H$ query heads, each of dimension $d_{head}$ , where $H \cdot d_{head} = D$ . Throughout the paper, $d_{head} = 128$ is fixed, so $H = D / 128$ , requiring $D$ to be a multiple of 128.
The same input is projected to a single key head and a single value head, each also of dimension $d_{head} = 128$ . This means $H_k = 1$ (number of key-value heads) instead of $H_k = H$ as in standard MHA.
Each query head computes attention scores against the shared key head, producing $H$ attention weight distributions. Each query head then uses its attention weights to aggregate the shared value head, producing $H$ output vectors of dimension $d_{head}$ .
The $H$ outputs are concatenated and projected back to dimension $D$ via a final linear layer.

Parameter count. The query projection has $D \cdot H \cdot d_{head} = D^2$ parameters. The key and value projections each have $D \cdot d_{head} = D \cdot 128$ parameters. The output projection has $D \cdot D = D^2$ parameters. Total: $D^2 + 2 \cdot 128 D + D^2 = 2D^2 + 256D$ . For large $D$ , this is approximately $2D^2$ , compared to $4D^2$ for standard MHA. The reduction by roughly a factor of 2 in attention parameters is what makes MQA attractive for inference: the KV cache size is $2 \cdot N \cdot T \cdot d_{head}$ instead of $2 \cdot N \cdot T \cdot H \cdot d_{head}$ , a factor of $H$ reduction.

Positional encoding. The paper uses Rotary Position Embeddings (RoPE) (Su et al., 2021) applied to the queries and keys before the attention computation. RoPE encodes relative position by rotating the query and key vectors by an angle proportional to their absolute positions, such that the dot product between a query at position $i$ and a key at position $j$ depends only on the relative position $i - j$ . This is a relative positional encoding that does not require any learned parameters. The paper explicitly states: "We do not use any absolute positional embeddings."

Why MQA over MHA? The primary motivation is inference speed. As analyzed in Section 5.1 and Appendix F, the KV cache is the dominant memory overhead during autoregressive decoding, and MQA reduces this overhead by a factor of $H$ (typically 32–64 for models at the 1B–14B scale). The paper acknowledges that this comes at a potential quality cost—MHA can learn more diverse attention patterns because each head has its own keys and values—but the quality difference is relatively small, especially at scale, and the inference efficiency gains are substantial. The Transformer baseline uses MQA specifically to provide a strong baseline for the inference comparisons: if Hawk and Griffin outperform an MQA Transformer (which is already optimized for fast inference), the advantage is more meaningful than outperforming a slower MHA Transformer.

Local (Sliding-Window) MQA

Local attention (Beltagy et al., 2020) restricts each query to attend only to keys within a fixed window of past tokens, rather than to all past tokens as in global attention.

Structure. The mechanism is identical to global MQA except for the attention mask. For a query at position $i$ , the attention computation is restricted to key positions $j \in [i - W + 1, i]$ , where $W$ is the window size (set to 1024 in all Griffin experiments unless otherwise stated). Positions outside this window are masked (their attention logits are set to $-\infty$ before the softmax).

KV cache. Because each query only needs the most recent $W$ keys and values, the KV cache can be bounded to size $2 \cdot N_{local} \cdot W \cdot d_{head}$ , where $N_{local}$ is the number of local attention layers. This is constant in the sequence length $T$ , unlike the global MQA cache which grows proportionally to $T$ . For Griffin, $N_{local} = N / 3$ (one local attention layer per three residual blocks, as described below), making the local attention cache small and fixed.

Computational complexity. Each local attention layer costs $O(T \cdot W \cdot D)$ operations, compared to $O(T^2 \cdot D)$ for global attention. Since $W$ is a constant (1024), this is linear in $T$ rather than quadratic.

Why local attention in Griffin? The paper's key insight is that the combination of recurrent blocks (which transport information across arbitrarily long distances via the fixed-size hidden state) and local attention (which enables precise token-level retrieval within a recent window) covers the full spectrum of temporal dependencies. Recurrence handles long-range information transport; local attention handles precise short-range copying and retrieval. The authors show (Section 6.2) that this combination is remarkably effective: a single local attention layer in the middle of a 5-layer network allows Griffin to match Transformers on the Selective Copying task, while Hawk (without any attention) is significantly slower to learn.

Complete Model Architectures: Hawk, Griffin, and MQA Transformer

All three model families share the same residual backbone and gated MLP block. They differ only in the temporal-mixing blocks that are interleaved with the MLPs.

MQA Transformer Baseline

Temporal-mixing: Every residual block uses a global MQA block as its temporal-mixing component. There are $N$ such blocks, each containing one MQA layer and one gated MLP.

Design philosophy: This is the standard Transformer architecture adapted with MQA for efficient inference, serving as the primary point of comparison. By sharing the MLP and residual structure with Hawk and Griffin, the paper isolates the effect of replacing attention with recurrence.

Hawk (Pure RNN)

Temporal-mixing: Every residual block uses the recurrent block (Section 2.3) as its temporal-mixing component. There are $N$ such blocks, each containing one recurrent block (Conv1D + RG-LRU + gating) and one gated MLP.

Design philosophy: Hawk is the purest test of the RG-LRU's capability as an attention replacement. If Hawk performs well, it demonstrates that gated linear recurrences can, on their own, match the quality of attention-based models at scale. The paper's results show that Hawk is competitive with the MQA Transformer baseline (slightly higher loss at small FLOPs budgets, closing the gap as scale increases) and outperforms Mamba-3B on downstream tasks, establishing the RG-LRU as an effective recurrent primitive.

Griffin (Hybrid)

Temporal-mixing: Griffin uses a layered pattern of recurrent and local attention blocks. Specifically, for every three residual blocks:

Blocks 1 and 2: Use recurrent blocks as the temporal-mixing component.
Block 3: Uses a local (sliding-window) MQA block as the temporal-mixing component.

This pattern of "two recurrent, one local attention" repeats throughout the depth of the model. For $N$ residual blocks, there are $2N/3$ recurrent blocks and $N/3$ local attention blocks.

Design philosophy. The hybrid design is motivated by the complementary strengths observed in the synthetic task experiments (Section 6.2). Recurrent blocks are efficient at compressing long-range information into a fixed-size state but struggle with precise token-level retrieval (as shown by Hawk's poor performance on the phonebook lookup task). Local attention is excellent at precise retrieval within its window but has no mechanism for information transport beyond the window size. Griffin combines them: the recurrent layers carry information across the full sequence, while the local attention layers allow the model to precisely access recent tokens when needed. The fixed window size (1024) is chosen as a balance between retrieval capability and cache size.

Why the 2:1 ratio? The paper does not provide an explicit ablation of the recurrent-to-attention ratio, but the choice of two recurrent blocks per one attention block reflects the empirical finding that only a small amount of attention is needed to recover the copying and retrieval capabilities that pure RNNs lack. The synthetic experiments (Section 6.2, Figure 6) used a single local attention layer in the middle of a 5-layer network and showed that this was sufficient to match Transformers on Selective Copying. The 2:1 ratio in Griffin extends this finding to the full-scale architecture.

Window size ablations (Appendix E, Figure 9). The paper investigates the effect of local attention window size for 400M-parameter models trained on different sequence lengths. The key findings:

Griffin with a fixed window size of 1024 outperforms global attention MQA Transformers at all training sequence lengths tested (2048, 4096, 8192).
MQA Transformers using purely local attention (window size < training sequence length) perform significantly worse than both global attention MQA Transformers and Griffin.
As the training sequence length grows, the gap between Griffin (fixed window 1024) and global attention MQA shrinks, suggesting that the window size should eventually be scaled with sequence length for optimal performance.

Griffin as the primary model. The paper positions Griffin—not Hawk—as the main contribution. Griffin's results (lower loss than Transformers in Figure 1a, performance competitive with Llama-2 in Table 1, strong long-context extrapolation in Figure 5) demonstrate that a hybrid recurrent-attention architecture is not just a theoretical possibility but a practical alternative to Transformers at scale. The authors are careful to note that Hawk, the pure RNN, has some limitations (poor copying and retrieval without fine-tuning), which Griffin partially addresses through its local attention layers.

Hyperparameters Across Model Scales

The paper defines model configurations at scales from approximately 100M to 14B parameters (Appendix C, Table 2). The key hyperparameters are:

Model dimension $D$ : Ranges from 768 (at ~100M) to 5120 (at 14B). Griffin and Hawk use the same $D$ as the MQA Transformer at each scale.
Number of layers $N$ : Ranges from 12 (at ~100M) to 40 (at 14B). For Griffin, this means there are $N/3$ local attention layers (e.g., ~4 at 1B scale, ~13 at 14B scale).
MLP expansion factor $M$ : Fixed at 3 for all models.
Head dimension $d_{head}$ : Fixed at 128 for all attention layers, requiring $D$ to be a multiple of 128 at all scales.
Recurrent dimension $D_{RNN}$ : Set to approximately $4D/3$ at each scale, as described above.
Local attention window size: Fixed at 1024 for all Griffin models, regardless of scale or training sequence length (though the paper suggests scaling it for very long sequences).
Vocabulary size: Not explicitly stated for the main scaling runs, but the embedding and output projection are tied.

Training hyperparameters. All models use the AdamW optimizer (Loshchilov and Hutter, 2017). The learning rate, weight decay, and $\beta_2$ parameter are tuned on small models (presumably 100M–400M scale) and then scaled to the larger models using fitted scaling rules. The paper states (Section 3.1):

"We tune the learning rate, weight decay and $\beta_2$ parameters for small models, and use these runs to identify scaling rules for these hyper-parameters which predict their optimal values for the 7B and 14B models."

The specific scaling rules are not detailed in the paper, but the approach follows the standard practice of extrapolating hyperparameters from smaller proxy models to larger target models, as introduced by Kaplan et al. (2020). Models are trained on the MassiveText dataset (Hoffmann et al., 2022) with a sequence length of 2048 tokens (except for the long-context experiments in Section 6.1, which use 8192 tokens). The number of training tokens follows the Chinchilla scaling law: roughly proportional to the number of model parameters. The largest models (7B and 14B) are trained on 300B tokens for the downstream evaluation in Table 1, which is below the Chinchilla-optimal amount (a 7B model would be Chinchilla-optimal at approximately 140B tokens, so 300B is "overtraining" relative to the Chinchilla prescription; the paper explicitly describes this as "overtraining" in Section 3.2).

Hardware-Efficient Implementation

The paper's efficiency claims depend critically on implementation details that address the fundamental challenge of executing diagonal recurrent layers on hardware designed for matrix multiplications. This section describes the key engineering decisions.

The Memory-Bandwidth Bottleneck for Diagonal RNNs

The RG-LRU's state update (Equation 4) is an element-wise operation: for each of the $D_{RNN}$ dimensions, it loads $h_{t-1}$ , $a_t$ , and $i_t \odot x_t$ (3 values in bfloat16 = 6 bytes), performs approximately 6 FLOPs (1 multiply for $a_t \odot h_{t-1}$ , 1 multiply for $i_t \odot x_t$ , 1 multiply for $\sqrt{1 - a_t^2} \odot (\ldots)$ , 1 addition, plus the computation of $\sqrt{1 - a_t^2}$ and $a_t$ itself), and writes $h_t$ (2 bytes). The FLOPs-to-byte ratio is:

$\frac{6 \text{ FLOPs}}{6 \text{ bytes read} + 2 \text{ bytes written}} = \frac{6}{8} = 0.75 \text{ FLOPs/byte}$

This is far below the TPU-v3's capacity for element-wise operations, which the paper reports as 4.2 FLOPs/byte (Appendix D, Table 3). This means the RG-LRU update is memory-bandwidth bound: the computation units spend most of their time waiting for data to be transferred from High-Bandwidth Memory (HBM) to the Vector Memory (VMEM) where computation occurs. The execution time is dominated by memory transfers, not by arithmetic.

Custom Pallas Kernel: The Linear Scan

To mitigate the memory-bandwidth bottleneck, the authors implement a custom kernel for the RG-LRU update using Pallas, a JAX-based domain-specific language for writing TPU kernels (Bradbury et al., 2018). The kernel uses a linear scan—a sequential loop over the time dimension—rather than a parallel algorithm (convolution or associative scan).

Why a linear scan? The key insight is that a linear scan keeps the hidden state $h_t$ in VMEM throughout the entire sequence, avoiding repeated transfers to and from HBM. At each timestep $t$ :

Load the current input $x_t$ (and computed gates $a_t$ , $i_t$ ) from HBM in a chunk (not one element at a time).
Update the hidden state in VMEM using the already-resident $h_{t-1}$ .
Write the output $h_t$ back to HBM (again in chunks).

The critical optimization is step 2: $h_t$ never leaves VMEM during the scan, eliminating the repeated HBM ↔ VMEM transfers that would occur in a naive implementation where the entire sequence of states is read and written at each timestep. The paper states (Section 4.2):

"In practice, this translates to almost 3× speed up over the native Jax implementation of the linear scan. Additionally, we observe 10–20% lower training times per step of the full Hawk model."

Why not a convolution? Prior work on linear SSMs (e.g., S4, H3) exploited the fact that time-invariant linear recurrences can be computed as a convolution of the input sequence with the system's impulse response. This is highly parallelizable and maps efficiently to matrix-multiply hardware. However, the RG-LRU is not time-invariant: the gates $r_t$ and $i_t$ depend on the input $x_t$ , which breaks the convolutional equivalence. The input-dependent gating means that the effective recurrence weights $a_t$ vary across timesteps, and there is no fixed impulse response to convolve with. The paper explicitly notes:

"The RG-LRU's gating mechanism on $a_t$ is not compatible with the convolutional view."

Why not an associative scan? The associative scan (also known as parallel prefix sum) is a standard algorithm for computing linear recurrences in $O(\log T)$ parallel steps. It works by observing that the recurrence $h_t = A_t h_{t-1} + B_t x_t$ can be expressed as a first-order linear recurrence where pairs of steps can be composed associatively: $(A_2, B_2) \circ (A_1, B_1) = (A_2 A_1, A_2 B_1 + B_2)$ . This composition operator is associative, enabling a binary-tree-style parallel reduction.

However, the paper finds that the associative scan is slower than the naive Jax linear scan on TPU-v3 (Appendix D.2, Figure 8a), not faster. The authors speculate:

"We speculate that the random access nature of the tree recombination of the parallel prefix-sum algorithm makes it poorly suited for the TPU architecture, leading to even slower memory transfers."

The associative scan requires non-contiguous memory access patterns (accessing elements at power-of-two strides during tree reduction), which is inefficient on TPU's vector memory architecture, which is optimized for contiguous, strided access patterns typical of matrix multiplications and convolutions. The linear scan, while sequential, accesses memory in a single, predictable, contiguous pattern that the memory controller can prefetch efficiently.

Model Parallelism: Sharding the Recurrent Block

For large models (7B+), the parameters and optimizer states cannot fit on a single TPU device. The paper uses a combination of Megatron-style tensor parallelism (Shoeybi et al., 2019) and ZeRO data parallelism (Rajbhandari et al., 2020).

MLP and MQA sharding. The gated MLP and attention blocks are sharded using the standard Megatron approach: each linear layer's weight matrix is split column-wise across devices, requiring an all-reduce operation in the forward pass (to sum the partial outputs) and another in the backward pass (to sum the partial gradients). For MQA, the attention heads are additionally sharded across devices (Narayanan et al., 2021).

Recurrent block sharding. The recurrent block presents unique challenges because it contains operations that are not dense matrix multiplications. The paper's sharding strategy is:

Linear layers: The two input projections ( $D \rightarrow D_{RNN}$ ) and the output projection ( $D_{RNN} \rightarrow D$ ) are sharded using Megatron-style column-wise splitting, same as the MLP and attention blocks. This requires one all-reduce per linear layer.
Conv1D: The separable Conv1D operates independently across channels—each of the $D_{RNN}$ channels has its own 4-tap filter, with no cross-channel interactions. This allows the channels to be split across devices with no communication overhead: each device handles its subset of channels independently, and the outputs are simply concatenated (which is free in the sharded layout since the channels are already partitioned).
RG-LRU gates: The weight matrices $W_a$ and $W_x$ for the recurrence and input gates would normally be dense $D_{RNN} \times D_{RNN}$ matrices, which would require cross-device communication if sharded. To avoid this, the paper makes these matrices block-diagonal with 16 blocks. This means $W_a$ and $W_x$ are $D_{RNN} \times D_{RNN}$ matrices that are zero except for 16 diagonal blocks. The output of the gate at dimension $i$ depends only on inputs in the same block (a contiguous range of $D_{RNN} / 16$ dimensions). This allows the 16 blocks to be assigned to different devices, again with no communication overhead—each device computes its gates independently.
Diagonal recurrence: The RG-LRU state update (Equation 4) is element-wise, meaning dimension $i$ of $h_t$ depends only on dimension $i$ of $h_{t-1}$ , $x_t$ , $a_t$ , and $i_t$ . This naturally shards without communication: each device maintains and updates its slice of the hidden state independently.

Overall communication. With this strategy, the recurrent block has the same communication requirements as the MLP block: one all-reduce per linear layer. The block-diagonal gates and the channel-independent Conv1D and recurrence add no additional communication. This is critical for scaling to large models, because communication overhead is often the dominant cost in distributed training.

Other optimizations. The paper also uses:

bfloat16 for model parameters and activations, reducing memory usage by 2× compared to float32 with minimal impact on training stability.
ZeRO (Rajbhandari et al., 2020) to distribute optimizer states across batch shards, further reducing per-device memory.

Training Speed Scaling

Figure 3 shows the training time per step for Griffin relative to the MQA Transformer baseline as a function of model size and sequence length. The key findings:

At short sequences (2K tokens): Griffin and the MQA Transformer have similar training times, with the MQA Transformer being slightly faster at 7B scale due to having fewer parameters (since MQA has fewer attention parameters than the recurrent block, which was matched to MHA).
As sequence length increases: The MQA Transformer's training time grows, while Griffin's remains approximately constant. This is because the Transformer's attention cost is $O(T^2 D)$ (for global attention) or $O(T W D)$ (for local attention, once $T > W$ ), while Griffin's recurrent cost is $O(T D^2)$ for the linear layers plus $O(T D)$ for the RG-LRU scan. The linear layers dominate at all sequence lengths, scaling as $O(T D^2)$ , which is the same as the MLP cost shared by both models.
The benefit is largest at small model sizes: At 400M parameters, Griffin is substantially faster than the Transformer at long sequences. At 7B parameters, the difference is smaller because the $O(T D^2)$ linear layers dominate the computation, and the attention's quadratic cost is a smaller fraction of total FLOPs. This is because $D$ grows faster than $T$ in typical scaling configurations—a 7B model has $D \approx 4096$ , making the $D^2$ factor in the linear layers much larger than the $T$ factor in the attention.

Practical implication: The training speed advantage of recurrent models is most pronounced when the sequence length is large relative to the model dimension. For very wide models (large $D$ ), the linear layers become the bottleneck regardless of architecture, and the advantage of avoiding attention is diminished.

Summary of Design Choices and Their Justifications

Diagonal real-valued recurrence over complex or dense recurrence: Real-valued diagonal recurrences halve the state size and parameter count compared to complex recurrences, and the paper found no quality improvement from complex numbers for language modeling. Dense recurrences (like classical RNNs) would be compute-bound during the state update but would have $O(D^2)$ per-timestep cost, making them too expensive for large $D$ .
Norm-preserving update ( $\sqrt{1 - a_t^2}$ coupling) over linear interpolation: The norm-preserving form prevents state norm explosion/vanishing over long sequences without requiring additional normalization layers. Linear interpolation ( $h_t = a_t h_{t-1} + (1 - a_t) x_t$ ) would be simpler but allows the state norm to drift.
Recurrence gate modulating $a_t$ rather than directly interpolating: Modulating the decay rate gives the gate an exponential effect on memory timescale, allowing the model to span several orders of magnitude in effective memory length with a single learned parameter $a$ per dimension. Direct interpolation (like GRU's update gate) would give only linear control over the mixing ratio.
Block-diagonal gates (16 blocks) over dense gates: The block-diagonal structure eliminates cross-device communication for the gate computation in distributed training, at a small cost in expressiveness (each block of dimensions can only gate based on inputs within that block). The number 16 is a pragmatic choice trading off communication overhead against gate capacity.
Separable Conv1D with temporal width 4: Provides minimal local temporal context (4 tokens) at negligible parameter cost ( $4 D_{RNN}$ ). Wider filters or non-separable convolutions would add more parameters without clear benefit—the recurrence is designed to handle longer-range dependencies, and the MLP handles cross-channel interactions.
2:1 recurrent-to-attention ratio in Griffin over other ratios: A small amount of local attention (one layer per three residual blocks) is sufficient to recover precise retrieval capabilities while keeping the KV cache small and bounded. More attention layers would improve retrieval further but at the cost of a larger cache and slower inference.
Fixed local attention window (1024) over scaled windows: A fixed window keeps the cache size constant and independent of training sequence length, simplifying deployment. The paper acknowledges that scaling the window with sequence length may be beneficial for very long contexts, but a fixed window of 1024 already outperforms global attention at the tested scales.
Pallas linear scan over associative scan or convolution: The linear scan keeps the hidden state in VMEM, minimizing HBM transfers, which is the primary bottleneck for diagonal RNNs on TPU-v3. The associative scan's tree reduction pattern causes inefficient non-contiguous memory access. The convolution approach is incompatible with input-dependent gating.
Tied input-output embeddings over separate embeddings: Weight tying reduces the total parameter count and provides a regularization effect, which is particularly valuable for models trained below the Chinchilla-optimal token count (as is the case for the 300B-token runs, where models are "overtrained" relative to their parameter count).

4. Key Insights and Innovations

Innovation 1: Gated Linear Recurrences Don't Need Complex Numbers or Continuous-Time Discretization for Language Modeling at Scale

This is fundamentally a negative result that reorients the design space for recurrent language models. The dominant tradition in the SSM literature—from S4 (Gu et al., 2021a) through S4D (Gu et al., 2022) to the LRU (Orvieto et al., 2023b)—treated complex-valued diagonal recurrences as the default expressive primitive. The theoretical motivation was clear: complex eigenvalues can represent oscillatory dynamics via the phase θ, enabling the recurrence to capture patterns that a purely real-valued decay cannot. The LRU's initialization scheme (eigenvalues uniformly distributed on the unit circle) was specifically designed to leverage this complex structure, and the Hippo framework (Gu et al., 2020) provided a continuous-time foundation for deriving optimal recurrence parameterizations.

Griffin and Hawk dispense with both, and show it doesn't matter for language modeling. The RG-LRU is real-valued, discrete-time from first principles (no continuous-to-discrete mapping), and initialized with a simple uniform distribution of effective decay rates between 0.9 and 0.999—no orthogonal polynomials, no HiPPO matrices, no normal-plus-low-rank parameterizations. The paper states this explicitly as a design choice (Section 2.4):

"Unlike many recent works in the SSM literature, the RG-LRU does not use initialization inspired by the theory of orthogonal polynomials, and it also is not defined as the discretization of an underlying continuous system."

This is not a theoretical claim about expressiveness—complex recurrences are provably more expressive (Orvieto et al., 2023a). Rather, it's an empirical claim about which dimensions of expressiveness matter for the task at hand. For language modeling, the additional capacity that complex numbers provide (oscillatory dynamics, twice the effective state dimension at the same parameter count) appears to be irrelevant—or at least not worth the added computational cost. The paper's pragmatic choice to ablate complex numbers and confirm "they were not beneficial for language modelling in practice" (Section 2.4) shifts the burden of proof: the default for language modeling recurrent architectures should be real-valued unless complex numbers demonstrate a clear advantage, rather than the reverse.

The significance extends beyond language modeling to the broader methodology of architecture design. The SSM literature had accumulated considerable theoretical machinery (continuous-time state spaces, orthogonal polynomial projections, measure-preserving flows) that was elegant but introduced complexity in initialization schemes, parameterization choices, and implementation. Griffin demonstrates that this machinery may be unnecessary for the specific regime of large-scale language modeling on modern hardware. This is an instance of a broader pattern in deep learning: as models scale, inductive biases that matter at small scale (careful initialization, theoretical guarantees about long-range memory) can be subsumed by learned behavior from data. The RG-LRU's success with a simpler parameterization suggests that the field has been over-indexing on theoretical desiderata from the long-range arena (Tay et al., 2020) that may not transfer to the distribution of dependencies present in natural language.

Innovation 2: The Recurrence Gate Introduces a Qualitatively New Gating Primitive—Content-Dependent Decay Modulation Rather Than State-Input Interpolation

This is a conceptual distinction with architectural consequences. To understand why this is innovative rather than incremental, we need to map the landscape of gating primitives that existed before this work.

The standard gating repertoire inherited from the LSTM and GRU literature operates on a single principle: interpolation between the previous state and a candidate new state. The LSTM's forget gate controls how much of $h_{t-1}$ to retain; its input gate controls how much of the candidate update to add. The GRU's update gate interpolates between $h_{t-1}$ and a candidate $\tilde{h}_t$ . Mamba's selection mechanism (Gu and Dao, 2023) follows the same pattern: the effective recurrence parameters $\bar{A}$ and $\bar{B}$ are input-dependent, but the update equation $h_t = \bar{A}(x_t) h_{t-1} + \bar{B}(x_t) x_t$ is structurally an interpolation between old state and new input. In all of these, the model can, at each timestep and for each dimension, choose to overwrite the state with new information or retain the old state.

The RG-LRU's recurrence gate $r_t$ operates on a fundamentally different axis: it modulates the decay rate of the state, not the interpolation ratio. The update equation $h_t = a^{c r_t} \odot h_{t-1} + \sqrt{1 - a^{2 c r_t}} \odot (i_t \odot x_t)$ couples the state retention and input incorporation through the norm-preserving weight $\sqrt{1 - a_t^2}$ . When $r_t \to 0$ , $a_t \to 1$ , and the model fully preserves $h_{t-1}$ while nearly zeroing out the input contribution. When $r_t \to 1$ , $a_t \to a^c$ (the base decay rate), and the model operates like a standard LRU. What the recurrence gate cannot do—and this is the key difference from interpolation-based gates—is simultaneously discard the old state AND inject a large new input. The coupling $\sqrt{1 - a_t^2}$ means that strong decay (small $a_t$ ) forces strong input incorporation, and strong retention (large $a_t$ ) forces weak input incorporation.

The paper argues (Appendix A, Figure 7) that this asymmetry is a feature, not a bug: it biases the model toward information preservation across uninformative tokens. In language, many tokens carry little new information (function words, punctuation, boilerplate phrases). An interpolation-based gate must actively learn to close both the forget and input gates on such tokens to avoid corrupting the state—a failure mode where the model overwrites useful context with noise. The recurrence gate, by design, makes it easier to preserve the state than to overwrite it: closing the gate ( $r_t \approx 0$ ) preserves the state perfectly, while opening it fully ( $r_t \approx 1$ ) only brings the model to the base decay rate, which is still relatively high ( $a^c \approx 0.43$ – $0.992$ depending on $a$ ). The authors frame this as enabling "super-exponential memory by reducing the influence of uninformative inputs" (Section 2.4).

Whether this specific inductive bias is the causal mechanism behind Hawk's strong performance is difficult to isolate—the paper doesn't provide ablation studies comparing different gating strategies in otherwise identical architectures. But the conceptual contribution is clear: the recurrence gate identifies a new axis in the gating design space (decay-rate modulation vs. state-input interpolation) and provides a concrete, well-motivated instantiation that performs competitively at scale.

Innovation 3: A Small Fraction of Local Attention Layers (1 in 3) Is Sufficient to Recover the Precise Retrieval Capabilities That Pure Recurrent Models Lack—and This Is an Empirical Finding, Not an Obvious Design Choice

This is the paper's most diagnostically significant finding, and it operates at a different level than the architectural contributions. Before this work, the tension between recurrent efficiency and retrieval precision was understood qualitatively—SSMs and linear RNNs were known to struggle with copying tasks (Jelassi et al., 2024), while Transformers excelled at them but suffered from quadratic cost. The natural engineering response would be to treat this as a spectrum: add as much attention as you can afford given your latency/throughput budget, with quality improving monotonically with the amount of attention.

Griffin's results challenge this monotonicity assumption. The paper demonstrates three things simultaneously:

First, on synthetic tasks (Section 6.2, Figure 6), a 5-layer Griffin with one local attention layer in the middle (block 3 of 5) matches the learning speed of a full 5-layer Transformer on Selective Copying, while a 5-layer Hawk is "significantly slower." One attention layer out of five—20% of the temporal-mixing blocks—is enough to close the gap. This is not an obvious result: one might reasonably expect that copying requires attention at multiple layers to propagate information up and down the network.

Second, on the Induction Heads task, the same 1-out-of-5 Griffin not only learns as fast as the Transformer but also extrapolates to sequences far longer than training length—something the Transformer with RoPE cannot do at all (Figure 6b). Hawk can extrapolate perfectly without any attention, but Griffin with local attention also extrapolates perfectly. This suggests that the local attention doesn't interfere with the recurrent layers' extrapolation capability—a non-trivial finding, since one might worry that attention layers with fixed positional encodings would bottleneck the model's ability to generalize to longer sequences.

Third, on the pre-trained model evaluation (phonebook lookup, Figure 6c), Griffin perfectly solves the task up to its local attention window size (1024 tokens), then degrades gradually. The Transformer solves it up to its training sequence length (2048) and then fails catastrophically. Hawk fails at phonebook lengths beyond a few entries. This pattern reveals a functional specialization: the recurrent layers carry information across the full sequence, while the local attention layers enable precise retrieval within their window. The model learns to route information appropriately without explicit architectural constraints beyond the layered structure.

The significance of this finding is that it establishes a lower bound on how much attention is necessary for competitive language modeling at scale. The answer appears to be "surprisingly little"—one local attention layer per three temporal-mixing blocks, with a fixed 1024-token window, is sufficient for the model to match Llama-2's downstream performance (Table 1) and outperform global-attention Transformers on long-context extrapolation (Figure 5). This is a practical design insight with immediate implications: if you're building an efficient language model, you don't need to choose between "all attention" and "no attention." A small, fixed-cost amount of local attention provides most of the retrieval benefit, and additional attention layers yield diminishing returns relative to their cost in KV cache size and inference latency. The paper doesn't provide a systematic sweep of attention ratios (different fractions of attention layers, different placement strategies), which is a limitation—the specific 2:1 recurrent-to-attention ratio may not be optimal. But the qualitative finding that a small fraction suffices is robust and actionable.

Innovation 4: Hardware-Aware Implementation Choices for Diagonal RNNs—Specifically, Linear Scan Over Associative Scan on TPUs—Are Non-Obvious and Architecture-Defining

This is an engineering insight with architectural consequences, and it's worth distinguishing from standard "we wrote a custom kernel" claims. The paper is making a specific argument: for diagonal RNNs with input-dependent gating on TPU-class hardware, the asymptotically superior parallel algorithm (associative scan) is worse in practice than the sequential linear scan, and this fact should influence how we design recurrent layers.

The associative scan (parallel prefix sum) is the algorithm that makes linear recurrences theoretically competitive with convolutions for training efficiency. It reduces the sequential dependency from $O(T)$ to $O(\log T)$ parallel steps by exploiting the associativity of the recurrence operator: $h_t = A_t h_{t-1} + B_t x_t$ can be composed as $(A_2, B_2) \circ (A_1, B_1) = (A_2 A_1, A_2 B_1 + B_2)$ , and this composition is associative, enabling a binary-tree reduction. Prior work (S5 by Smith et al., 2022; Mamba by Gu and Dao, 2023) relied on the associative scan as the primary efficient training mechanism for diagonal SSMs. It is, on paper, the "right" algorithm—it parallelizes the recurrence, reduces the critical path, and maps naturally to the sort of parallel computation that GPUs and TPUs are designed for.

The paper's finding that the associative scan is slower than a naive sequential loop on TPU-v3 (Appendix D.2, Figure 8a) is surprising because it contradicts the theoretical expectation. The authors attribute this to memory access patterns:

"We speculate that the random access nature of the tree recombination of the parallel prefix-sum algorithm makes it poorly suited for the TPU architecture, leading to even slower memory transfers."

The associative scan's binary tree reduction requires accessing elements at power-of-two strides during the upward sweep and the downward sweep. On a TPU's vector memory architecture—which is heavily optimized for contiguous, strided access patterns typical of matrix multiplications—these non-contiguous accesses cause cache thrashing and prevent the memory controller from prefetching effectively. The linear scan, despite being sequential, accesses memory in a single contiguous sweep that the hardware can prefetch perfectly. Combined with the Pallas kernel's ability to keep the hidden state in VMEM (eliminating repeated HBM transfers), the linear scan achieves ~3× speedup over the naive Jax implementation and is substantially faster than the associative scan.

This finding is not just an implementation note—it has architectural implications that shaped the RG-LRU design. The fact that the associative scan is slow on TPUs means that the RG-LRU does not need to be expressible as an associative composition to be efficient. This frees the design from constraints that would otherwise bind: the recurrence gate $r_t$ can modulate $a_t$ in non-associative ways (the exponentiation $a^{c r_t}$ is not easily composable), because the sequential scan doesn't require associativity. If the associative scan were the only efficient option, the gating mechanism might need to be constrained to preserve the associative property—which would limit expressiveness. The paper's hardware-aware perspective is thus not just about making the architecture faster, but about enabling architectural choices that would be ruled out if one assumed the associative scan was necessary.

It's important to note that this finding is hardware-specific. The authors are careful to state: "The conclusions drawn here do not necessarily apply to other accelerators" (Section 4.2, footnote). On GPUs with different memory hierarchies and warp-level primitives, the associative scan may be competitive or superior. But the principle—that hardware-specific memory access patterns should inform recurrence design—is general: the right algorithm for training diagonal RNNs depends on the target hardware, and architecture designers should not assume that theoretically parallel algorithms are faster in practice without empirical validation.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the MassiveText dataset (Hoffmann et al., 2022), previously used to train Gopher (Rae et al., 2021) and Chinchilla (Hoffmann et al., 2022), though with "a slightly different data subset distribution" (Section 3.1). A sequence length of 2048 tokens is used for the main scaling runs, with additional training at 8192 tokens for the long-context experiments. For downstream evaluation, the paper uses standard benchmarks including MMLU, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, and WinoGrande (Section 3.2). Long-context extrapolation is evaluated on held-out books and arXiv article datasets (Section 6.1, Appendix G). The synthetic copy and retrieval tasks (Selective Copying, Induction Heads, Phonebook Lookup) are described in Section 6.2.
Base model(s). Three model families are the primary subjects: (1) an MQA Transformer baseline — a gated-MLP pre-norm Transformer using Multi-Query Attention with RoPE; (2) Hawk — the pure RNN model using the recurrent block (RG-LRU + Conv1D + gating) in all temporal-mixing positions; and (3) Griffin — the hybrid model alternating two recurrent blocks with one local MQA block (window size 1024). Models are scaled from ~100M to 14B parameters (Appendix C, Table 2). External comparisons are made to Mamba-3B (Gu and Dao, 2023) and Llama-2 (Touvron et al., 2023). The choice of PaLM 2-S* is not relevant here — this paper trains models from scratch rather than building on a pre-existing base model.
Metrics. The primary metric throughout is held-out validation loss on the MassiveText evaluation set, plotted against training FLOPs to establish scaling laws (Figure 1a). For downstream tasks, character-normalized accuracy is reported on MMLU, HellaSwag, PIQA, ARC-E, and ARC-C, with unnormalized accuracy with partial scoring for WinoGrande (Section 3.2). For inference speed, latency (time to generate a specified number of tokens at batch size 16) and maximum throughput (largest tokens/second achievable on a single device) are the key metrics (Section 5). For long-context modeling, evaluation loss as a function of context length is the primary metric (Figure 5). For synthetic tasks, accuracy on the copying/retrieval target is used (Figure 6).
Baselines. The paper uses several baselines: (1) MQA Transformer (global attention) — the in-house baseline sharing the exact same MLP, residual structure, training data, and hyperparameter tuning budget as Hawk and Griffin, making it the fairest comparison. (2) MQA Transformer (local attention only) — Transformers where all attention layers use sliding-window attention rather than global attention, tested in Appendix E. (3) Mamba-3B — the strongest prior recurrent model at the time, trained on 600B tokens of different data. (4) Llama-2 — a widely-used open Transformer model trained on 2T tokens. The paper acknowledges that comparisons to external baselines are confounded by different training data and hyperparameter tuning strategies.
Generation budget / compute accounting. For scaling law experiments, compute is measured in total training FLOPs, and models are trained with Chinchilla-proportional token budgets (tokens roughly proportional to parameter count). All three model families are trained at matched FLOPs budgets for the scaling curves in Figure 1a. For the downstream evaluation, all in-house models are trained on a fixed 300B tokens — described as "overtraining" relative to Chinchilla-optimal, since a 7B model is Chinchilla-optimal at ~140B tokens. For inference comparisons, the generation budget is implicitly the time or memory to generate a fixed number of tokens (512, 1024, 2048, or 4096) at a given batch size. For the long-context experiments, models trained at 2048 sequence length and 8192 sequence length are compared at equal total training tokens (the 8K models use 4× smaller batch size for 4× fewer steps to keep total tokens fixed).
Cross-validation / statistical protocol. The paper does not describe a cross-validation protocol for the scaling law experiments — held-out loss is reported on a fixed evaluation set. For the downstream task evaluation (Table 1), standard benchmark evaluation protocols are followed (character-normalized accuracy, partial scoring for WinoGrande). The scaling rules for learning rate, weight decay, and β2 are tuned on small models and extrapolated to larger models (Section 3.1), but the specific extrapolation methodology is not detailed. No statistical significance testing or confidence intervals are reported — the paper relies on the consistency of trends across model scales and the large size of the evaluation datasets to establish reliability.

Main Quantitative Results

Scaling Laws: Training FLOPs vs. Held-Out Loss

The central scaling result is presented in Figure 1a. All three model families — MQA Transformer, Hawk, and Griffin — exhibit power-law scaling between held-out loss and training FLOPs across scales from ~100M to 14B parameters, as previously observed for Transformers (Kaplan et al., 2020; Hoffmann et al., 2022). The log-log relationship is approximately linear for all three model families.

Griffin achieves the lowest held-out loss at all FLOPs budgets. The Griffin curve lies below both the MQA Transformer and Hawk curves at every point on the FLOPs axis. At the largest scale (14B parameters, corresponding to the rightmost point on Griffin's curve), the gap between Griffin and the MQA Transformer is visually apparent in Figure 1a, though the paper does not quote an exact numerical difference in loss. The authors state (Section 1, point 2): "Griffin achieves slightly lower held-out loss than strong Transformer baselines at all model scales."

Hawk is competitive with Transformers, with the gap closing at scale. At small FLOPs budgets (left side of Figure 1a), Hawk shows slightly higher held-out loss than the MQA Transformer. However, as training FLOPs increase, the Hawk and Transformer curves converge. At the largest scales (7B parameters and above), the gap is visually minimal. The paper characterizes this as (Section 3.1): "Hawk on the other hand achieves slightly higher validation loss, but this gap appears to close as the training budget increases."

The scaling behavior extends beyond 7B parameters. The Griffin curve includes a data point at 14B parameters, demonstrating that the power-law relationship holds through at least this scale. This is significant because prior work on SSMs and linear RNNs had not demonstrated maintained scaling to this size — Mamba, for instance, reported results up to 2.8B parameters.

What this means: These scaling curves are the paper's core empirical contribution. They establish that recurrent and hybrid architectures are not just competitive with Transformers at small scale (where many architectural alternatives work) but maintain their competitiveness into the multi-billion-parameter regime where Transformers have been dominant. The fact that Griffin's curve is below the Transformer curve at all points — rather than crossing or converging — suggests that the hybrid design's advantage is not a small-scale artifact that disappears with more compute.

Downstream Task Performance: Comparison to Mamba and Llama-2

Table 1 reports character-normalized accuracy across six benchmarks for models trained on 300B tokens. The key comparisons are:

Hawk-3B vs. Mamba-3B. Hawk-3B achieves stronger performance on downstream tasks than Mamba-3B despite being trained on half as many tokens (300B vs. 600B). The paper states (Section 3.2): "Hawk-3B achieves stronger performance on downstream tasks than Mamba-3B, despite being trained on half as many tokens." Specific numbers are not quoted in the main text for this comparison, but Table 1 provides the per-benchmark breakdown.

Griffin-3B vs. Mamba-3B. Griffin-3B "significantly outperforms" Mamba-3B (Section 3.2), again despite the 2× token disadvantage. This is the strongest evidence that the RG-LRU-based architecture is not just competitive with but superior to the prior state-of-the-art recurrent model.

Griffin-7B and Griffin-14B vs. Llama-2. Griffin-7B and Griffin-14B "achieve performance competitive with Llama-2, despite being trained on nearly 7 times fewer tokens" (Section 3.2). Llama-2 was trained on 2T tokens across its variants; Griffin at 7B and 14B was trained on only 300B tokens. The paper does not specify exactly which Llama-2 variant is being compared (Llama-2 comes in 7B, 13B, and 70B sizes), but context suggests it is Llama-2-7B and Llama-2-13B. The claim is about token efficiency: Griffin reaches similar downstream accuracy with ~7× fewer training tokens.

Griffin vs. in-house MQA Transformer baseline. Griffin outperforms the MQA Transformer baseline across the board (Section 3.2: "Griffin outperforms this baseline"), while Hawk is described as "competitive" with it. This is the cleanest comparison because both models share the same data, hyperparameter tuning, and training budget.

WinoGrande scoring. The paper notes that for WinoGrande specifically, "unnormalized accuracy with partial scoring" is reported, whereas the other benchmarks use character-normalized accuracy. This is consistent with standard practice for WinoGrande evaluation.

Inference Speed: Latency and Throughput

The inference results focus on 1B-parameter models, comparing Hawk, Griffin, and an MQA Transformer baseline.

Latency (Figure 4). With an empty prefill (Figure 4a), all three models show increasing latency as the number of sampled tokens grows, but the MQA Transformer's latency grows faster than Hawk or Griffin. With a 4096-token prefill (Figure 4b), the difference is more dramatic: the MQA Transformer's latency per token is significantly higher because it must load a larger KV cache (4096 prefill tokens + generated tokens). Hawk and Griffin maintain near-constant per-token latency regardless of prefill length because their recurrent state size is fixed. The paper states (Section 5.2): "Hawk and Griffin achieve faster sampling latency than MQA Transformers for long sequences. This is particularly noticeable as the sequence length and the prefill length... are increased."

Throughput (Figure 1b). Throughput is measured as the maximum tokens per second achievable on a single device when sampling 512, 1024, 2048, or 4096 tokens from an empty prompt. At all sequence lengths, both Hawk and Griffin achieve substantially higher throughput than the MQA Transformer — "significantly higher throughput" (Section 5.2). The advantage grows with sequence length: at 4096 tokens, Hawk and Griffin achieve approximately 4× higher throughput than the Transformer baseline. The paper explains this (Section 5.2): "This is partially due to recurrent models having lower latency but also primarily occurs because Griffin and Hawk can fit larger batch sizes than the MQA Transformer on a single device, since their cache size is smaller."

Hawk vs. Griffin throughput. Hawk achieves the highest throughput of all three models because it has no attention cache at all — its only state is the fixed-size recurrent hidden state. Griffin's throughput is slightly lower than Hawk's because the local attention layers still maintain a KV cache, albeit a small one bounded by the window size (1024). The paper notes (Section 5.2): "Hawk achieves higher throughputs than Griffin, since the size of the local attention cache eventually becomes comparable to the size of the parameters when the batch size is large."

Scaling with model size (not explicitly plotted for inference). The inference experiments are conducted at 1B parameter scale. The paper does not report latency or throughput measurements at larger model sizes (e.g., 7B or 14B). The theoretical analysis in Section 5.1 (Equation 5) suggests that the advantage of recurrent models' small cache size should become less pronounced as model size grows, because the parameter loading time eventually dominates the cache loading time. At 1B parameters with $D = 2048$ (from Table 2), the recurrent state per layer is $D_{RNN} \approx 4D/3 \approx 2730$ elements. For an N-layer model, the total recurrent state is $N \times D_{RNN} \approx 24 \times 2730 \approx 65,520$ elements for a 1B Hawk, which is tiny compared to ~1B parameters. For a 14B model, the parameter count grows by ~14× while the recurrent state grows roughly proportionally to $N \times D$ , keeping the ratio roughly constant. So the qualitative advantage should persist at larger scales, but the paper does not empirically verify this.

Training Efficiency at Long Sequences

Figure 3 compares the training time per step of Griffin and the MQA Transformer baseline as a function of sequence length and model size. Key findings:

At 2K sequence length (the base configuration): Griffin and the MQA Transformer have similar training times, with Griffin being slightly slower at 7B scale. The paper explains this (Section 4.3): "our MQA baseline has slightly fewer parameters than Griffin at the same model scale (and performs fewer FLOPs). This explains why Griffin trains slightly slower than our MQA baseline at 7B for short sequences."

As sequence length increases: The Transformer baseline slows down while Griffin's training time per step remains approximately constant across sequence lengths from 2K to 16K tokens (Figure 3, all three panels). This is because the Transformer's global attention cost is O(T²D), while Griffin's recurrent cost is O(TD) for the RG-LRU plus O(TD²) for the linear layers. Since the linear layers dominate, and they scale as O(TD²) regardless of architecture, the attention's quadratic cost becomes the differentiating factor as T grows.

The advantage is largest at small model sizes: At 400M parameters (Figure 3a), Griffin is substantially faster than the Transformer at long sequences — the ratio of Transformer time to Griffin time grows visibly with sequence length. At 7B parameters (Figure 3c), the advantage is smaller because the linear layers ( $O(T D^2)$ ) represent a larger fraction of total FLOPs. The paper notes (Section 4.3): "as we increase the model width D compared to the sequence length T, the linear layers become the primary computational bottleneck, minimizing the efficiency gains from the RNN block."

At 16K sequence length (7B model): The Transformer is only modestly slower than Griffin — the curves in Figure 3c are close. This suggests that for very large models at very long sequences, the training speed advantage of recurrent architectures diminishes, though the inference advantage (throughput, latency) persists because it depends on cache size rather than FLOPs.

Long Context Modeling: Extrapolation and Learning from Longer Contexts

Figure 5 (left) evaluates models trained on 2048-token sequences on a held-out books dataset with evaluation context lengths from 512 to 32,768 tokens.

Hawk and Griffin extrapolate; Transformers do not. Both Hawk and Griffin show decreasing evaluation loss as the context length increases beyond their training length of 2048, up to approximately 8K–16K tokens. Beyond that, loss levels off or slightly increases but remains far below the Transformer's loss. The MQA Transformer, in contrast, shows loss that increases sharply beyond the training sequence length. The paper states (Section 6.1): "Hawk and Griffin improve next token prediction given longer contexts, and they are overall able to extrapolate to significantly longer sequences (at least 4× longer) than they were trained on."

Griffin extrapolates despite using RoPE. This is a non-trivial result because RoPE-based Transformers are known to struggle with length extrapolation — the rotary embeddings are trained for positions 0–2047, and extending to positions beyond 2048 produces out-of-distribution position encodings. Griffin uses RoPE for its local attention layers, yet the model as a whole extrapolates well. The recurrent layers apparently provide enough long-range context that the local attention's RoPE limitation does not bottleneck the model.

Learning from longer contexts (Figure 5, right). When models are trained on 8192-token sequences (Griffin-8k, Hawk-8k) and compared to models trained on 2048-token sequences (Griffin-2k, Hawk-2k) at equal total tokens:

At evaluation lengths of 8192 and beyond, the 8K-trained models achieve lower loss, demonstrating that they can effectively learn from longer contexts.
At short evaluation lengths (512–2048), the 2K-trained models achieve slightly lower loss. The paper interprets this as (Section 6.1): "This suggests that the training sequence length should be carefully chosen according to the intended downstream use of the model."

arXiv results (Appendix G, Figure 10). The qualitative pattern is replicated on an arXiv evaluation set — Hawk and Griffin extrapolate, the Transformer does not, and the 8K-trained models outperform at long contexts while underperforming slightly at short contexts.

Synthetic Copy and Retrieval Tasks

Figure 6 presents results on three synthetic tasks designed to probe copying and retrieval capabilities.

Selective Copying (Figure 6a). Five-layer models (~250K parameters) are trained to copy 16 data tokens embedded in 1024-token sequences of noise. Results:

Transformer: Solves the task quickly — accuracy reaches near 1.0 early in training.
Hawk: Also solves the task eventually but is "significantly slower" to learn (Section 6.2), requiring substantially more training steps to reach the same accuracy.
Griffin: Effectively matches the Transformer's learning speed, despite having only one local attention layer (in block 3 of 5). The paper states this "demonstrated exceptional ability" (Section 6.2).

Induction Heads (Figure 6b). Five-layer models are trained on sequences of length 256 to recall the token following a special token. At evaluation:

Transformer: Solves the task within the training length but fails to extrapolate to longer sequences. Accuracy drops sharply beyond 256 tokens.
Hawk: Solves the task and perfectly extrapolates to sequences "several orders of magnitude longer than the training sequence length" (Section 6.2).
Griffin: Also perfectly extrapolates, despite using local attention with RoPE.

Phonebook Lookup (Figure 6c). Pre-trained 7B Hawk, 7B Griffin, and 6B MQA Transformer models are evaluated zero-shot (no fine-tuning on the lookup task):

MQA Transformer: "Almost perfectly solves this task up to the training sequence length" (~2048 tokens), but fails catastrophically beyond that — accuracy drops to near zero for longer phonebooks.
Hawk: Performs "reasonably well" for very short phonebooks (a few entries) but accuracy degrades sharply as the phonebook grows, similar to Mamba's behavior reported by Jelassi et al. (2024).
Griffin: "Perfectly solves this task up to a context length that matches its local attention window size of 1024." Beyond 1024, accuracy degrades gradually rather than catastrophically, continuing to perform above chance at context lengths where the Transformer has zero accuracy. The paper notes (Section 6.2): "Once the context length is long enough such that the local attention window does not cover the whole phonebook, performance starts to degrade."

Ablation Studies and Robustness Checks

Local attention window size in Griffin (Appendix E, Figure 9): 400M-parameter Griffin models are trained on sequence lengths of 2048, 4096, and 8192 tokens with local attention window sizes ranging from 256 to the full training sequence length. The key findings:

Griffin with a fixed window of 1024 outperforms the global attention MQA Transformer at all training sequence lengths tested. This is remarkable — a model with no global attention and only local attention in 1/3 of its layers beats a full-global-attention Transformer.
MQA Transformers using purely local attention (window < training sequence length) perform significantly worse than both global attention MQA Transformers and Griffin across all settings. This confirms that local attention alone is not sufficient; the recurrent layers in Griffin are essential for long-range information transport.
The performance gap between Griffin (fixed window 1024) and the global attention MQA Transformer shrinks as the training sequence length grows (from 2048 to 8192). The paper suggests (Appendix E): "if the sequence length grows further, it is likely important to slowly also grow the local attention window size."

Recurrence gate behavior (Appendix A, Figure 7): The paper analyzes the effect of different gating mechanisms on the recurrent weight by plotting the interpolation coefficients α(rt) (weight on ht−1) and β(rt) (weight on xt) as functions of the gate value rt. The RG-LRU's recurrence gate is compared against GRU-style gating, Mamba-style selection, and the LRU's no-gating baseline. The analysis shows:

The RG-LRU's gating is qualitatively different from GRU and Mamba: it biases toward high α values (state retention) across the full range of rt, while GRU/Mamba gates interpolate more evenly between state retention and input incorporation.
When rt ≈ 0 (gate closed), the RG-LRU fully retains the previous state (α ≈ 1, β ≈ 0). When rt ≈ 1 (gate open), it approaches the LRU's update (α = a^c, β = sqrt(1−a^(2c))). The GRU gate, in contrast, exactly interpolates between the previous state and the input.
This is presented as a feature — the bias toward retention makes it easier for the model to preserve information across long sequences of uninformative tokens.

Complex vs. real recurrence (Section 2.4, Appendix B): The paper states that a complex-valued variant of the RG-LRU was developed but found not beneficial for language modeling:

"We found that complex recurrences were not beneficial for language modelling in practice, as also observed by Gu and Dao (2023)."

No quantitative ablation is presented in the main text or appendix — this is reported as an informal experimental finding. The complex CG-LRU is defined in Appendix B for reference but no training runs using it are reported.

Pallas kernel vs. native Jax vs. associative scan (Appendix D.2, Figure 8): The custom Pallas linear scan kernel achieves nearly 3× speedup over the naive Jax implementation of the RG-LRU scan, as shown in Figure 8a. The associative scan is substantially slower than even the naive Jax implementation — up to 50% slower when measured end-to-end on the full Hawk model (Figure 8b). The paper states (Appendix D.2): "for completeness we have also added the runtime of the associative scan, which can be up to 50% slower." This justifies the architectural choice to use a linear scan rather than building on associativity.

Training sequence length (2K vs. 8K, Figure 5 and Figure 10): Models trained at 8K sequence length (with total tokens held constant by reducing batch size) outperform 2K-trained models at long evaluation contexts but underperform slightly at short contexts. This is replicated on both books (Figure 5) and arXiv (Figure 10), suggesting it is a general pattern rather than dataset-specific.

Model scale sweep (Figure 1a): The power-law relationship is validated across model scales from ~100M to 14B parameters (Griffin), 7B (Hawk and Transformer). This is itself a robustness check — the architecture's behavior is consistent across more than two orders of magnitude in parameter count.

Critical Assessment

Claim: "Hawk exceeds the reported performance of Mamba on downstream tasks" (Abstract, Section 3.2)

This claim is supported by Table 1 but comes with significant caveats that the paper acknowledges. Hawk-3B is trained on 300B tokens of MassiveText; Mamba-3B is trained on 600B tokens of a different dataset (the Pile, as described in Gu and Dao, 2023). The training data distribution, data quality, and tokenizer are all different. The paper is transparent about this (Section 3.2): "We note however that both Mamba and Llama-2 were trained on different datasets and with different hyper-parameter tuning strategies, which may partially explain our strong performance."

However, the strength of the result — outperforming Mamba despite training on half as many tokens — makes it unlikely that the difference is entirely attributable to data. If data quality were the sole factor, one would need MassiveText to be more than 2× as "efficient" per token as the Pile for language model training, which is possible but would represent a substantial finding in itself. The more parsimonious interpretation is that the RG-LRU architecture combined with the gated MLP and residual structure is genuinely more token-efficient than Mamba's selective SSM architecture.

A stronger test would have been to train a Mamba model on MassiveText with the same hyperparameter tuning budget as Hawk, eliminating the data confound entirely. The paper does not report such a comparison.

Claim: "Griffin matches the performance of Llama-2 despite being trained on over 6 times fewer tokens" (Abstract, Section 3.2)

The claim states "matches the performance of Llama-2" but the text in Section 3.2 says "achieve performance competitive with Llama-2." The word "matches" implies equality within noise; "competitive" implies being in the same ballpark. Table 1 is needed to assess which characterization is accurate, and the paper does not quote specific comparative numbers in the text.

The ~7× token efficiency claim is striking but requires careful interpretation. Llama-2 was trained for 2T tokens, which is Chinchilla-optimal or beyond for the 7B and 13B model sizes. Griffin at 7B and 14B trained on 300B tokens is substantially undertrained by Chinchilla standards (a 7B model would be Chinchilla-optimal at ~140B tokens, a 14B model at ~280B). So Griffin is achieving competitive performance at roughly the Chinchilla-optimal token count (or slightly beyond it), while Llama-2 is operating in the "overtrained" regime where additional tokens yield diminishing returns. The fair comparison would be: if you had 2T tokens to train on, would Griffin or Llama-2 achieve better performance? The paper's scaling curves (Figure 1a) suggest Griffin's loss would continue to decrease with more tokens, but there is no guarantee the curves remain parallel — they could converge or cross. The paper does not answer this question because it does not train Griffin to 2T tokens.

Claim: "Griffin achieves slightly lower held-out loss than strong Transformer baselines at all model scales" (Section 1, point 2)

Figure 1a supports this claim directly. The Griffin curve is below the MQA Transformer curve at every FLOPs point. The gap appears consistent rather than narrowing at scale, suggesting it is not a small-model artifact. The caveat is that "slightly" is not quantified — without numerical loss values, we cannot assess whether the gap is practically meaningful (e.g., 0.01 nats) or near the noise floor. The log-scale axes make it difficult to visually estimate the absolute difference.

An important design confound: the MQA Transformer baseline uses Multi-Query Attention (fewer parameters in the attention block) while the recurrent block in Griffin/Hawk was parameter-matched to Multi-Head Attention (more parameters). The paper acknowledges this (Section 3, Hawk: "Note that we match parameters with MHA attention block, though our Transformer baseline and Griffin ended up relying on MQA attention"). This means Griffin has slightly more parameters than the MQA Transformer at the same model dimension D. Whether the loss advantage is attributable to architecture or simply to having more parameters is unclear. An MHA Transformer baseline would have been a fairer comparison for parameter-matched evaluation, but MHA is slower at inference, making it a weaker baseline for the efficiency claims.

Claim: "Hawk and Griffin achieve significantly higher throughput than MQA Transformers" during inference (Section 1, point 5)

Figure 1b shows this clearly for 1B-parameter models. The throughput advantage is substantial — roughly 4× at 4096 tokens. The latency results in Figure 4 further support the claim, showing that recurrent models avoid the growing per-token cost that Transformers incur as the KV cache expands.

However, the inference results are limited to a single model scale (1B) and do not systematically explore how the advantage scales with model size. The paper's own analysis (Section 5.1) suggests the advantage comes from smaller cache size, and the cache-to-parameters ratio remains favorable for recurrent models at all scales (since the recurrent state size is proportional to ND, same as the parameter count), but this is not empirically verified beyond 1B. It is also hardware-specific — the throughput numbers are measured on a single device (presumably a TPU-v3, consistent with the training experiments), and the absolute throughput values would differ on other hardware, though the relative advantage should persist on any memory-bandwidth-bound accelerator.

Claim: "Griffin performs better than Transformers when evaluated on sequences longer than those seen during training" (Section 1, point 6)

Figure 5 strongly supports the extrapolation claim. The Transformer's loss increases sharply beyond 2048 tokens; Hawk and Griffin's loss continues to decrease or remains flat through ~8K–16K tokens. The arXiv replication (Figure 10) provides consistency across datasets.

A limitation: the evaluation is on next-token prediction loss, not on task performance at long contexts. Lower loss on long books sequences is a reasonable proxy for long-context understanding, but it's not the same as demonstrating that the model can actually use information from 10K tokens ago to answer a question or perform a task. The phonebook lookup task (Figure 6c) provides a more direct test of retrieval from long contexts, and there Griffin only performs well up to its attention window size (1024) before degrading — the recurrent layers alone are not sufficient for precise long-range retrieval. So "performs better on long sequences" needs qualification: it performs better at next-token prediction (which benefits from broad context) but not at precise token-level retrieval (which requires attention).

Missing Experiments and Baselines

Several experiments would have strengthened the paper's claims:

Ablation of the recurrence gate: The paper presents the recurrence gate as a key innovation but provides no ablation comparing the RG-LRU to a version without the gate (i.e., a real-valued LRU with the same residual structure and MLP block) at any scale. Without this ablation, it is impossible to attribute Hawk's strong performance to the gating mechanism specifically versus the overall architecture (residual structure, gated MLP, Conv1D, etc.).
Training a Mamba model on MassiveText: A head-to-head comparison with Mamba on the same data and tuning budget would eliminate the data confound and provide a cleaner measure of architectural differences.
Scaling the local attention window with sequence length: The paper shows in Appendix E that the gap between Griffin (fixed window 1024) and global attention shrinks as training sequence length grows. Training a Griffin model with a proportionally scaled window (e.g., window = sequence length / 2) at longer sequence lengths would test whether the fixed window is a bottleneck.
Inference speed at 7B and 14B scales: The inference results are all at 1B. The paper's theoretical model (Equation 5) predicts that the advantage should persist but weaken as model size grows (because parameter loading dominates cache loading). Empirical verification would make the efficiency claims more concrete for practitioners deploying at scale.
Ablation of the recurrent-to-attention ratio: Griffin uses a 2:1 ratio. Would 3:1 be better? 1:1? The synthetic experiments used 1 attention layer out of 5 (a 4:1 ratio), and it worked, but this was at tiny scale (250K parameters). A sweep of ratios at, say, 400M parameters would establish whether the 2:1 choice is near-optimal or arbitrary.
Comparison to hybrid Transformer variants: Mistral 7B (Jiang et al., 2023) uses sliding-window attention with a small amount of global attention. Comparing Griffin to such architectures, which also bound the KV cache, would contextualize the recurrent-attention hybrid against attention-only hybrids.
Synthetic task results at larger scale: The synthetic copy/retrieval experiments use 5-layer, 250K-parameter models. Whether the "one local attention layer is sufficient" finding holds at billion-parameter scale is not tested. The phonebook lookup on pre-trained 7B models (Figure 6c) partially addresses this but is limited to one task.

Where the Claims Hold Conditionally

The paper's central claims hold under these conditions, which are not always explicitly stated:

Training data: MassiveText (or comparable high-quality web text). Performance comparisons to Mamba and Llama-2 are confounded by different training data.
Model scale: Up to 14B parameters for Griffin, 7B for Hawk and Transformer. Claims about scaling should be understood as applying within this range; extrapolation to 70B or 175B is plausible based on the power-law trends but not guaranteed.
Sequence length: Training at 2K–8K tokens. The inference advantage grows with sequence length; the training advantage is modest at 2K and grows at 8K+.
Hardware: TPU-v3. The Pallas kernel optimization and the linear-scan-over-associative-scan finding are hardware-specific. The architecture itself is hardware-agnostic, but the training efficiency claims depend on an efficient implementation for the target hardware.
Task domain: Language modeling (next-token prediction) and standard NLU benchmarks. The synthetic copy/retrieval results probe specific mechanistic capabilities. No experiments are reported on code generation, mathematical reasoning, multi-lingual tasks, or multi-modal tasks.
Inference batch size: The throughput results are measured at maximum batch size that fits on a single device. At very small batch sizes (e.g., batch size 1 for interactive use), the advantage from fitting larger batches disappears, and only the latency advantage from smaller cache remains.

6. Limitations and Trade-offs

The Recurrence Gate Is Not Ablated — We Cannot Attribute Performance Gains to the Proposed Gating Mechanism

The assumption or constraint. The paper presents the recurrence gate $r_t = \sigma(W_a x_t + b_a)$ as a core innovation, positioning it as "different from other gating mechanisms in the literature" (Section 2.4) and arguing that its bias toward state retention enables "super-exponential memory by reducing the influence of uninformative inputs." Appendix A and Figure 7 provide an analytical comparison of the RG-LRU's gating behavior against GRU and Mamba-style gating, showing that the interpolation coefficients are qualitatively different. However, the paper never trains a version of Hawk or Griffin without the recurrence gate — that is, a model where $a_t = a^c$ (a fixed decay rate per dimension, equivalent to a real-valued LRU with the same residual backbone, gated MLP, Conv1D, and input gate) — to isolate the gate's contribution.

The consequence. Without this ablation, it is impossible to determine how much of Hawk's strong performance is attributable to the recurrence gate specifically versus the broader architectural context (the pre-norm residual structure, the gated MLP with GeGeLU, the separable Conv1D, and the input gate $i_t$ ). The paper's claim that the recurrence gate plays a "key role" (Section 2.4) in enabling long-range memory is a reasonable hypothesis but is not empirically tested. A simpler alternative — a real-valued LRU with only the input gate and no recurrence gate, using the same residual backbone and hyperparameter tuning — might perform equivalently. If so, the recurrence gate would be unnecessary complexity rather than an essential innovation. Given that the paper makes strong claims about the gate's novelty ("to our knowledge, our recurrence gate is different from other gating mechanisms"), the absence of this ablation is a significant gap in the evidence chain.

What evidence exists in the paper. The only evidence for the recurrence gate's importance is indirect: (1) Appendix A, Figure 7 provides an analytical comparison showing the gate's behavior differs from GRU/Mamba, but this is a mathematical characterization, not a performance comparison; (2) Hawk (which includes the gate) outperforms Mamba-3B on downstream tasks (Table 1), but this is confounded by different training data, token counts, and many other architectural differences between the two models. The paper does not report any head-to-head comparison of gated vs. ungated RG-LRU variants.

Mitigation status. Not addressed. The paper does not acknowledge this as a missing ablation. The closest the paper comes to isolating the gate is the comparison of Hawk to Mamba (which uses a different gating philosophy), but this comparison involves too many confounds to attribute the performance difference to the gating mechanism.

Difficulty Estimation for Compute-Optimal Scaling Is Prohibitively Expensive — Making the Headline Efficiency Gains Unrealized in Deployment

The assumption or constraint. The paper's efficiency claims for Hawk and Griffin — lower latency, higher throughput, competitive training speed — are based on the architecture itself and are validated with end-to-end measurements (Figures 1b, 3, 4, 8b). However, the paper also makes a broader claim about token efficiency: Griffin matches Llama-2's performance despite being trained on ~7× fewer tokens (Section 3.2, Table 1). This claim implicitly assumes that the training data, hyperparameter tuning strategy, and evaluation protocol are comparable — an assumption the paper partially acknowledges. But there is a deeper, unaddressed assumption: that the architecture's token efficiency advantage over Llama-2 would persist if both were trained to the same total token budget on the same data with matched tuning effort.

The consequence. The ~7× token efficiency claim is the paper's most headline-grabbing result, but it rests on a comparison that is not controlled. Llama-2 was trained on a different data mixture (publicly available web data, not MassiveText), used a different tokenizer, and was optimized under a different hyperparameter search budget by a different team with different infrastructure. The paper trains Griffin on 300B tokens; Llama-2 was trained on 2T tokens. If Llama-2's data quality is lower than MassiveText — or if its hyperparameters were suboptimal for the 300B-token regime (since it was tuned for 2T-token training) — then part of Griffin's apparent token efficiency advantage could be attributable to these confounds rather than to architectural superiority. Conversely, if Griffin were trained on 2T tokens of MassiveText, it is not guaranteed that its loss curve would maintain the same advantage over Llama-2's curve — the scaling trends in Figure 1a show Griffin maintaining a consistent gap below the MQA Transformer, but this gap could narrow, widen, or cross at token counts an order of magnitude beyond what was tested.

What evidence exists in the paper. The comparison to the in-house MQA Transformer baseline (Table 1) partially addresses this — both are trained on the same data with the same hyperparameter tuning budget. Griffin outperforms this baseline, which supports the architectural advantage claim without data confounds. However, the comparison to Llama-2 specifically is what generates the headline numbers (~7× fewer tokens), and that comparison is confounded. The paper is transparent about this (Section 3.2: "both Mamba and Llama-2 were trained on different datasets and with different hyper-parameter tuning strategies, which may partially explain our strong performance"), but the transparency does not remove the confound.

Mitigation status. Partially addressed through the in-house MQA Transformer comparison, which isolates the architectural effect from the data effect. However, this comparison does not produce a "token efficiency ratio" relative to Llama-2 — it only shows that Griffin > MQA Transformer on matched data. The paper does not train a Llama-2-architecture model on MassiveText to quantify how much of the ~7× gap is architectural vs. data/tuning. The authors acknowledge this as a limitation in the text but do not treat it as a barrier to the central claim.

Pre-Trained Pure RNNs (Hawk) Cannot Perform Exact Retrieval or Copying at Long Range Without Fine-Tuning

The assumption or constraint. The paper positions recurrent models as efficient alternatives to Transformers, emphasizing their fixed-size state and fast inference. However, Section 6.2 reveals a capability boundary: on the phonebook lookup task — a direct test of whether the model can retrieve a specific piece of information from a long context — pre-trained Hawk-7B (the pure RNN) fails for phonebooks longer than a few entries. The paper states (Section 6.2):

"while Hawk can do reasonably well on the task for very short phonebook lengths, it fails to memorize and retrieve the correct phone number when the phonebook length grows."

This is consistent with theoretical expectations: a fixed-size recurrent state of dimension $D_{RNN}$ must compress an arbitrarily long sequence into a vector of fixed dimension, and precise retrieval of arbitrary tokens is fundamentally bottlenecked by this compression. The paper explicitly acknowledges this limitation:

"This is not particularly surprising since Hawk uses a small fixed-size state."

The consequence. Hawk cannot serve as a drop-in replacement for Transformers in applications that require precise long-range token retrieval without additional mechanisms (fine-tuning, retrieval-augmented generation, or the addition of attention layers as in Griffin). This includes tasks like: answering questions about specific facts mentioned earlier in a long document, following multi-step instructions where each step references earlier context, or code generation tasks that require recalling variable names or function signatures defined thousands of tokens ago. The phonebook results (Figure 6c) are a synthetic instantiation of this class of problems, and Hawk's near-zero accuracy for long phonebooks suggests it would fail on real-world analogs. This is a capability ceiling, not just an efficiency trade-off — additional test-time compute or larger model scale would not fix it, because the bottleneck is the fixed state size, not insufficient training.

What evidence exists in the paper. Figure 6c directly demonstrates the failure: Hawk-7B accuracy drops sharply as the number of phonebook entries increases. The Transformer baseline solves the task up to its training sequence length (2048 tokens), while Hawk fails much earlier. The synthetic Selective Copying task (Figure 6a) provides additional evidence: even when explicitly trained on the copying task (not pre-trained and evaluated zero-shot), Hawk requires substantially more training steps to reach perfect accuracy compared to Transformers or Griffin. The Induction Heads task (Figure 6b) shows a more optimistic picture — Hawk can learn to extrapolate perfectly — but this task requires recalling a single token associated with a learned pattern, not arbitrary token-level retrieval.

Mitigation status. Partially addressed through Griffin, which adds local attention layers specifically to handle this limitation. Griffin "perfectly solves this task up to a context length that matches its local attention window size of 1024" (Section 6.2, Figure 6c), demonstrating that a small amount of attention is sufficient to recover the retrieval capability. However, this mitigation has a ceiling: when the context length exceeds the local attention window (1024 tokens), Griffin's performance degrades, though more gracefully than Hawk's. The paper explicitly states that limitations remain: "more work is needed to improve these capabilities for such models" (Section 6.2). The mitigation is therefore partial — Griffin closes the gap for contexts up to the window size but does not solve the general long-range retrieval problem that Transformers handle natively (up to their training length).

Inference Speed Measurements Are Limited to 1B Scale — The Advantage at Deployment Scale Is Unvalidated

The assumption or constraint. All inference latency and throughput measurements in Section 5 are conducted at the 1B-parameter scale (Figures 1b, 4). The paper's theoretical model of decoding speed (Section 5.1, Equation 5) predicts that inference time is dominated by the sum of parameter loading time and cache loading time:

$\text{Time} \approx \frac{\text{param size} + \text{batch size} \times \text{cache size}}{\text{memory bandwidth}}$

For recurrent models, the cache size is tiny (the fixed hidden state, ~ $N \cdot D_{RNN}$ elements); for Transformers, it grows linearly with sequence length. At 1B parameters, the recurrent state is negligible compared to the parameters, so the advantage is clear. However, as model size grows, the parameter loading time grows proportionally, while the relative importance of the cache size advantage depends on the ratio of batch size × cache size to parameter size. The paper acknowledges this dependency implicitly in Section 5.1: "as the model size grows the sequence length at which we see latency benefits (where the KV cache size is comparable to parameter size) also increases."

The consequence. At 7B, 14B, or 70B parameters — the scales at which production language models are deployed — the inference speed advantage of Hawk and Griffin is not empirically measured. The theoretical model suggests the advantage should persist (since the recurrent state remains proportional to $N \cdot D$ , same scaling as parameter count, while the KV cache grows with $T$ independently), but there are second-order effects that could change the picture:

At larger model sizes, the batch size that fits on a single device may shrink for all architectures, reducing the throughput advantage from being able to fit larger batches.
Multi-device inference (model parallelism) introduces communication overhead that could disproportionately affect recurrent models if the sharding strategy for the RG-LRU scan requires cross-device synchronization at each timestep (though the paper's sharding design in Section 4.1 minimizes communication).
The absolute memory bandwidth and FLOPs characteristics of inference hardware (which may differ from TPU-v3 training hardware) could shift the memory-boundedness threshold, potentially making attention relatively less memory-bound at large $D$ and small $T$ .

What evidence exists in the paper. Only the 1B-scale measurements (Figures 1b, 4) and the analytical model (Section 5.1). The paper does not report latency or throughput for 7B or 14B models during inference. The training speed measurements (Figure 3) cover up to 7B, but training and inference have different computational profiles (training involves parallel processing of full sequences; decoding involves sequential token generation with a KV cache), so training speed trends do not directly predict inference speed trends.

Mitigation status. Not addressed. The paper does not acknowledge the lack of larger-scale inference measurements as a limitation, nor does it suggest such measurements as future work. The analytical model provides a theoretical justification for the advantage persisting, but the empirical validation is absent.

The Architecture's Copying and Retrieval Performance Is Evaluated Only at Tiny Scale (250K Parameters) — Generalization to Pre-Training Scale Is Assumed

The assumption or constraint. The synthetic task experiments in Section 6.2 — Selective Copying (Figure 6a) and Induction Heads (Figure 6b) — are conducted on 5-layer models with model dimension 64, totaling approximately 250K parameters. This is four orders of magnitude smaller than the 7B models evaluated on the phonebook lookup task (Figure 6c). The paper uses these tiny-scale experiments to draw conclusions about mechanistic capabilities: Griffin matches Transformer learning speed on Selective Copying and extrapolates on Induction Heads. These conclusions are then implicitly assumed to explain why pre-trained Griffin-7B performs well on the phonebook lookup (up to the attention window size).

The consequence. Mechanistic capabilities observed at 250K parameters may not transfer to billion-parameter scale. At 250K parameters, the model dimension is 64, the number of attention heads is small (presumably 1 or 2 given $d_{head} = 128$ in the main experiments would require $D \geq 128$ ), and the total representational capacity is severely constrained. The finding that "one local attention layer in a 5-layer network is sufficient to match Transformers on Selective Copying" (Section 6.2) tells us that the architecture can in principle learn this capability, but it does not tell us whether it does learn it in the pre-training regime — with orders of magnitude more parameters, a different data distribution (natural language rather than synthetic tokens), and a different training objective (causal language modeling on diverse text rather than a single synthetic task). The phonebook lookup (Figure 6c) partially bridges this gap by testing pre-trained 7B models, but it tests only one task and only evaluates zero-shot performance, not learning efficiency.

What evidence exists in the paper. The tiny-scale experiments (250K parameters, Figure 6a-b) and the pre-trained evaluation (7B, Figure 6c) are presented as complementary evidence. The paper does not train models at intermediate scales (e.g., 100M or 400M) on synthetic copying/retrieval tasks to test whether the efficiency advantage scales with model size. The phonebook results do show Griffin outperforming Hawk at 7B, which is consistent with the tiny-scale finding that local attention helps with retrieval, but the "learning speed" advantage (Figure 6a) — that Griffin learns as fast as a Transformer — has no pre-training-scale analog because the pre-trained models are evaluated zero-shot, not fine-tuned on the task.

Mitigation status. Not addressed. The paper treats the tiny-scale findings as directly relevant to the pre-training-scale behavior without acknowledging the scale gap. This is a common practice in mechanistic interpretability and architecture analysis — studying small models to understand large ones — but it requires the assumption that the mechanisms are scale-invariant, which is not validated. The paper does not discuss this assumption or suggest larger-scale synthetic experiments as future work.

The Pallas Kernel Optimization Is Hardware-Specific — Efficiency Gains Do Not Transfer to Other Accelerators Without Equivalent Engineering Investment

The assumption or constraint. Section 4.2 describes a custom Pallas kernel that achieves ~3× speedup over the native Jax implementation of the RG-LRU scan on TPU-v3, and the paper attributes the ~10–20% end-to-end training speed improvement of the full Hawk model to this kernel. The decision to use a linear scan rather than an associative scan is justified by empirical measurements on TPU-v3 showing the associative scan is slower. The paper explicitly scopes this finding to its hardware in a footnote (Section 4.2):

"The conclusions drawn here do not necessarily apply to other accelerators."

The consequence. The paper's training efficiency claims — that Hawk and Griffin match Transformer training speed at short sequences (Section 4.3, Figure 3) — depend on the existence of an efficient recurrence implementation for the target hardware. On TPU-v3, the custom Pallas kernel achieves this; on other accelerators (Nvidia GPUs, AMD GPUs, Apple Silicon, custom ASICs), the implementation story is different:

On Nvidia GPUs, the associative scan may be competitive or superior due to different memory hierarchy characteristics and warp-level shuffle instructions that can accelerate tree reductions. FlashAttention-style kernel fusion (Dao et al., 2022a) could potentially make the associative scan efficient enough to be preferable.
On hardware without a Pallas equivalent, the engineering effort to write a custom low-level kernel for the RG-LRU scan may be substantial, and the naive Jax/PyTorch implementation may be too slow for practical training.
The paper's finding that the associative scan is "up to 50% slower" on TPU-v3 (Appendix D.2, Figure 8b) is specific to that architecture's memory access patterns and cannot be assumed to generalize to GPUs, where the random access patterns of the tree reduction may be handled more efficiently by the L1/L2 cache hierarchy.

A practitioner wanting to adopt Hawk or Griffin on non-TPU hardware would need to invest in hardware-specific kernel development to achieve competitive training speeds, or accept the performance penalty of a naive scan implementation. The paper provides no guidance or reference implementation for other hardware platforms.

What evidence exists in the paper. The Pallas kernel speedup is measured on TPU-v3 (Appendix D.2, Figure 8). The slower associative scan is also measured on TPU-v3. No measurements on any other hardware platform are reported. The paper's training speed comparison against Transformers (Figure 3) uses the optimized Pallas kernel for Hawk/Griffin and the standard Jax attention implementation for the Transformer baseline, both on TPU-v3 — so the comparison is fair for that specific hardware but does not indicate what the relative speeds would be on other platforms.

Mitigation status. The paper acknowledges the hardware-specificity of the kernel optimization (Section 4.2: "we focus on developing an efficient implementation tailored to this device") but does not provide implementations, benchmarks, or guidance for other hardware. The architectural description of the RG-LRU is hardware-agnostic — the layer is defined mathematically and can be implemented on any platform — but the paper's efficiency claims depend on the existence of an efficient implementation. A practitioner reading the paper to decide whether to adopt Hawk/Griffin on their own hardware would need to budget for implementation work not accounted for in the paper.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the architecture search space for efficient language models from a binary choice — "Transformers with global attention" versus "pure recurrent models that struggle with retrieval" — to a spectrum of hybrid designs where a small, fixed fraction of local attention layers recovers the precise token-manipulation capabilities that pure RNNs lack. Before Griffin, the dominant narrative was that recurrent architectures faced a fundamental trade-off: you could have efficient inference with a fixed-size state, or you could have accurate long-range retrieval, but not both. Mamba (Gu and Dao, 2023) had shown that input-dependent selectivity could close some of the quality gap, and H3 (Dao et al., 2022b) had demonstrated that SSMs combined with small convolutions could handle long-range tasks, but neither had provided a systematic demonstration at scale that a recurrent-attention hybrid could exceed Transformer quality while maintaining the inference efficiency advantages of recurrence. Griffin's scaling curves (Figure 1a) — with the hybrid model below the Transformer curve at all FLOPs budgets from ~100M to 14B parameters — establish that the hybrid design is not a compromise that sacrifices quality for efficiency, but a genuinely superior architecture in the quality-vs-compute Pareto sense.

This is not a paradigm shift in the Kuhnian sense — the core components (diagonal linear recurrences, sliding-window attention, gated MLPs) all existed in the literature. Rather, it is a synthesis and scaling validation that reorients the field's default assumptions. Three specific shifts are worth articulating:

First, the burden of proof in architecture design has shifted toward recurrent and hybrid models. Before this work, the default assumption for anyone building a large language model was that you use Transformers with global attention, perhaps with MQA or GQA to reduce the KV cache (as in Llama-2, Mistral, and Gemini). Griffin demonstrates that you can achieve competitive or better quality with no global attention at all — only a small number of local attention layers (one per three residual blocks) with a fixed window size. This means the default should no longer be "Transformers unless proven otherwise." Future architecture explorations should include recurrent and hybrid baselines as first-class candidates, not as niche alternatives for memory-constrained settings. The paper's finding that Griffin with a fixed 1024-token local attention window outperforms global attention Transformers at all training sequence lengths tested (Appendix E, Figure 9) is particularly striking — it suggests that global attention may be overkill for the types of dependencies present in natural language, and that a well-designed combination of recurrence (for long-range transport) and local attention (for precise short-range retrieval) covers the dependency spectrum more efficiently.

Second, the paper reconciles the tension between the SSM community's focus on theoretical properties and the practical demands of large-scale language modeling. The SSM literature from S4 (Gu et al., 2021a) through the LRU (Orvieto et al., 2023b) had developed an elegant theoretical framework — continuous-time state spaces, HiPPO initialization, orthogonal polynomial projections, complex eigenval ues — that guaranteed certain long-range memory properties. But as the paper demonstrates, these theoretical guarantees may be unnecessary for language modeling at scale. The RG-LRU dispenses with all of them: it is real-valued, discrete-time from first principles, initialized with a simple uniform distribution of decay rates between 0.9 and 0.999, and defined without reference to any continuous-time system. Yet it scales competitively with — and in Griffin's case, exceeds — Transformers. This is an instance of a broader pattern in deep learning: as models and datasets grow, inductive biases that matter at small scale can be subsumed by learned behavior from data. The RG-LRU's success suggests that for language modeling, the field may have been over-investing in theoretical properties from the long-range arena (Tay et al., 2020) that do not transfer to the statistical structure of natural language. The implication is not that SSM theory is useless — it may be essential for other modalities (audio, time series, genomics) where dependencies span truly long horizons — but that for text, the effective memory length required is shorter than the theory assumed, and simple gated recurrences suffice.

Third, the paper establishes verifier-free, hardware-aware implementation as a first-class architectural constraint, not an afterthought. Section 4's detailed analysis of the memory-bandwidth bottleneck for diagonal RNNs, the custom Pallas linear scan kernel, and the block-diagonal gate parameterization for sharding are not mere engineering footnotes — they are architectural decisions that shaped the RG-LRU design. The finding that the associative scan is slower than a sequential linear scan on TPU-v3 (Figure 8a) is counterintuitive and has architectural consequences: it means recurrence designers should not assume that theoretically parallel algorithms are faster in practice, and should instead benchmark both on their target hardware. The paper's methodological contribution here is a template for how to evaluate recurrent architectures: measure FLOPs-to-byte ratios, identify memory-boundedness, write custom kernels that keep the recurrent state in fast memory, and design the recurrence to shard cleanly across devices without additional communication. Future work on efficient architectures should treat this hardware-aware methodology as standard practice, not optional optimization.

Follow-Up Research This Work Enables

Ablating the recurrence gate against a fixed-decay LRU baseline at scale. The paper presents the recurrence gate $r_t$ as a key innovation but never trains a version of Hawk or Griffin without it — that is, a model where $a_t = a^c$ (a fixed learned decay rate per dimension) with only the input gate $i_t$ , using the identical residual backbone, gated MLP, and Conv1D. A controlled experiment training a gated vs. ungated RG-LRU across multiple model scales (100M, 400M, 1B) on the same data and tuning budget would isolate whether the recurrence gate provides a meaningful improvement or whether the input gate and the norm-preserving update alone are sufficient. The null hypothesis — that the recurrence gate is unnecessary complexity and a fixed-decay real-valued LRU performs equivalently — is plausible given that the LRU already demonstrated strong long-range performance without content-dependent decay modulation. If confirmed, this would simplify the architecture and reduce the parameter count (removing $W_a$ and $b_a$ ). If the gate does matter, the result would quantify its contribution and provide evidence for the "super-exponential memory" hypothesis (Section 2.4) — that content-dependent decay modulation is what enables recurrent models to preserve information across long sequences of uninformative tokens. A strong follow-up would train both variants at 1B scale on 100B+ tokens and measure not just held-out loss but also performance on long-context retrieval tasks (like the phonebook lookup) to test whether the gate specifically improves memory retention.

Measuring the Pareto frontier of recurrent-to-attention ratios in hybrid architectures. Griffin uses a 2:1 ratio (two recurrent blocks per one local attention block) with a fixed 1024-token attention window. The synthetic experiments (Figure 6a-b) used a 4:1 ratio (one attention layer in five) at 250K parameters and found it sufficient for copying and induction heads. This raises the question: what is the minimum amount of attention needed at scale, and how does the optimal ratio depend on model size and sequence length? A systematic sweep — training 400M-parameter Griffin variants with recurrent-to-attention ratios of 1:0 (pure Hawk), 5:1, 3:1, 2:1, 1:1, and 0:1 (pure local attention Transformer) — would map out the Pareto frontier of quality vs. KV cache size. The quality metric should include both held-out loss and retrieval capability (phonebook lookup accuracy at different context lengths), since the paper shows these two capabilities dissociate: Hawk has competitive loss but fails at retrieval, while local-attention-only Transformers have good retrieval within their window but poor long-range loss (Appendix E). A strong follow-up would plot accuracy on the phonebook task at 4K context length against inference throughput (tokens/second at 4K sequence length) for each ratio, giving practitioners a direct trade-off curve to inform architecture choices for their deployment constraints. The paper's finding that even one local attention layer was sufficient at tiny scale suggests the knee of this curve might be at a high recurrent-to-attention ratio (e.g., 4:1 or 5:1), meaning future models could use even less attention than Griffin while retaining retrieval capabilities.

Training Griffin on 2T tokens to test whether the advantage over Transformers persists or saturates. The paper's most headline-grabbing claim — that Griffin matches Llama-2 despite being trained on ~7× fewer tokens — is based on a 300B-token training run, well below the Chinchilla-optimal token count for a 7B model (~140B) and far below Llama-2's 2T tokens. The scaling curves in Figure 1a show Griffin maintaining a consistent gap below the MQA Transformer at all measured FLOPs budgets, but these curves extend only to ~14B parameters, corresponding to roughly 300B–400B tokens. A critical open question is whether Griffin's architectural advantage is truly a token efficiency gain (better loss at every token count) or a shift in the scaling law exponent (the curves eventually converge or cross). Training Griffin at 7B on 1T or 2T tokens of MassiveText, and comparing against an identically-trained MQA Transformer at the same token counts, would distinguish these hypotheses. If the gap persists or widens at Chinchilla-optimal and beyond-Chinchilla token counts, Griffin would be unambiguously superior on both quality and efficiency axes. If the gap closes, then Griffin's advantage is primarily in the low-data regime (matching larger models with fewer tokens) but Transformers catch up with sufficient data — still practically useful for data-constrained settings, but not a universal replacement. A strong follow-up would also train an MHA Transformer baseline (not just MQA) at the same scale to quantify how much of Griffin's parameter-count advantage comes from matching the recurrent block to MHA rather than MQA.

Developing a difficulty-adaptive, content-aware local attention window for Griffin. The paper uses a fixed local attention window of 1024 tokens, but Appendix E (Figure 9) shows the performance gap between Griffin (fixed window) and global attention Transformers shrinks as the training sequence length grows, and the paper suggests "it is likely important to slowly also grow the local attention window size" (Appendix E). A fixed window is suboptimal because it treats all positions identically: a token at position 2000 in a 4K sequence has the same 1024-token attention span as a token at position 200. A content-adaptive variant — where the attention window expands when the model's internal uncertainty or the recurrence gate values indicate that long-range retrieval is needed — could improve performance on very long sequences without substantially increasing the average cache size. Concretely, the recurrence gate $r_t$ (Equation 1) already provides a per-token signal about whether the model is in "retention mode" ( $r_t \approx 0$ , preserving state) or "update mode" ( $r_t \approx 1$ , incorporating new information). A low $r_t$ value could trigger expanding the local attention window, since it indicates the current token is not informative and the model may need to retrieve information from further back. A strong follow-up would train a Griffin variant where each local attention layer's window size is dynamically scaled based on the average $r_t$ value in the preceding recurrent blocks, and measure whether this improves phonebook lookup accuracy beyond the fixed 1024-token window while keeping average cache size manageable.

Benchmarking the RG-LRU on non-language sequential modalities to test the generality of the real-valued, discrete-time design choice. The paper explicitly scopes its design choices to language modeling: real-valued recurrences "were not beneficial for language modelling in practice" (Section 2.4), and the RG-LRU "does not use initialization inspired by the theory of orthogonal polynomials" (Section 2.4). But language has a specific statistical structure — dependencies are relatively local, with occasional long-range references (coreference, topic continuity). Other modalities have fundamentally different temporal structures: audio waveforms have fine-grained periodic structure where complex-valued oscillatory dynamics might be essential; time-series forecasting often involves seasonal patterns spanning thousands of timesteps; genomics involves long-range motif matching where precise retrieval matters. A systematic evaluation of the RG-LRU (and its complex-valued CG-LRU variant, defined in Appendix B) on benchmarks from these modalities — the Long Range Arena (Tay et al., 2020) for non-language sequential tasks, LibriSpeech or similar for audio modeling, and a standard time-series forecasting benchmark — would test whether the paper's architectural simplifications (real-valued, no HiPPO, no continuous-time discretization) are language-specific or general. Finding that the CG-LRU significantly outperforms the RG-LRU on audio but not on text would validate the paper's modality-specific design philosophy and provide guidance for future multi-modal architectures that might need different recurrence parameterizations for different input types.

Stress-testing Griffin's extrapolation on tasks requiring reasoning over retrieved information, not just next-token prediction. The paper's long-context evaluation (Figure 5, Figure 10) measures next-token prediction loss on books and arXiv articles as context length increases. Lower loss at long contexts is encouraging, but it does not demonstrate that the model can actually use information from 10K+ tokens ago to perform a task. The phonebook lookup (Figure 6c) partially tests this — it requires retrieving a specific token — but it is a single synthetic task with a simple structure (memorize name-number pairs and recall). A more diagnostic evaluation would test Griffin on tasks that require multi-hop reasoning over long contexts: given a long document, answer a question whose answer requires synthesizing information from two or more widely-separated passages. This is the kind of task that tests whether the recurrent state encodes information in a way that supports compositional retrieval, or merely summarizes the gist in a way that improves next-token prediction but not precise reasoning. The SCROLLS benchmark (Shaham et al., 2022) or the LongBench dataset provide standardized long-context reasoning tasks. Evaluating pre-trained Griffin-7B on these benchmarks — and comparing against both the MQA Transformer baseline and Hawk — would reveal whether the local attention layers in Griffin enable compositional reasoning over retrieved information, or whether the model's strong extrapolation on next-token prediction is a surface-level phenomenon that does not translate to deeper understanding.

Practical Applications and Downstream Use Cases

Batch inference for RLHF data generation and scoring pipelines. In reinforcement learning from human feedback (RLHF) and related techniques (e.g., rejection sampling fine-tuning, constitutional AI), the training loop requires generating thousands to millions of responses from the language model, which are then scored by a reward model. These pipelines are throughput-bound: the faster you can generate tokens, the faster your RLHF iteration cycle. Griffin's 4× higher throughput at 1B scale with long sequences (Figure 1b) — which comes from both lower per-token latency and the ability to fit larger batch sizes on a single device due to the small fixed-size recurrent state — translates directly to faster and cheaper RLHF cycles. For a 7B-scale Griffin deployed on TPU-v3 or equivalent hardware, the throughput advantage over an MQA Transformer of comparable quality means that a given RLHF training run could be completed in roughly one-quarter the wall-clock time, or with one-quarter the number of accelerators. Since RLHF is increasingly used not just for chat models but also for code generation (AlphaCode 2), mathematical reasoning, and instruction-following, this efficiency gain has broad applicability across the LLM training pipeline. The caveat is that the throughput advantage at 7B scale has not been directly measured — the paper only evaluates 1B models — but the analytical model in Section 5.1 suggests the advantage should persist because the recurrent state remains tiny relative to parameters at all scales.

On-device or edge deployment of long-context models. As language models move from the cloud to edge devices (laptops, phones, wearables), two constraints become paramount: memory footprint and inference latency. The KV cache is the dominant memory consumer during decoding for Transformers — at 100K tokens, even an MQA model's KV cache can exceed the model parameters themselves, making long-context inference infeasible on memory-constrained devices. Hawk's fixed-size recurrent state (proportional to $N \cdot D_{RNN}$ , independent of sequence length) eliminates this problem entirely: a 1B Hawk model on a phone could theoretically process a 100K-token context with the same per-token memory cost as a 1K-token context, because the recurrent state size never grows. The Conv1D state adds a negligible 3 × DRNN elements per recurrent block. Griffin's local attention cache is bounded to the window size (1024 tokens × $d_{head}$ × 2 × N_local), which is also fixed and independent of the total context length. This means a Griffin model could be deployed on-device for tasks like long-form document summarization, personal knowledge base querying, or multi-turn conversation where the entire conversation history must be retained — all with predictable, bounded memory usage. The paper's extrapolation results (Figure 5) suggest the models would continue to benefit from contexts well beyond their training length, making this a practical deployment scenario today for models trained at 8K sequence length and deployed at 32K+.

Cost-efficient training of long-context models from scratch. Organizations training large language models from scratch face a growing tension: downstream applications demand longer context windows (32K, 128K, 1M tokens), but training on long sequences is expensive because attention's quadratic cost scales as $O(T^2 D)$ for global attention or $O(T W D)$ for windowed attention once $T > W$ . Griffin's training time per step remains approximately constant as sequence length increases (Figure 3), because the RG-LRU scan cost is $O(T D_{RNN})$ and the linear layers dominate at $O(T D^2)$ regardless of architecture. For a team training a 7B model on 32K-token sequences, the difference between Griffin and a global-attention Transformer could be the difference between a 4-week training run and a 6-week training run — or between fitting the model on a given cluster and needing to requisition more accelerators. The paper shows this advantage is largest at small model sizes (Figure 3a: 400M), but it persists at 7B (Figure 3c), though the relative speedup is smaller because the $O(T D^2)$ linear layers dominate both architectures. For very large models (70B+) trained on very long sequences (128K+), the attention cost fraction grows relative to the linear layers (since $T$ increases while $D$ stays fixed), so Griffin's training speed advantage should become more pronounced again. This makes Griffin particularly attractive for the next generation of long-context models, where both training and inference efficiency are critical. A practical consideration: the custom Pallas kernel for the RG-LRU scan is TPU-specific, so teams training on GPU clusters would need to invest in an equivalent CUDA kernel (or rely on a naive implementation that negates the training speed advantage) until optimized GPU kernels become available in open-source frameworks.

When to Prefer This Method

The paper positions Griffin and Hawk as alternatives to Transformers with global attention across multiple axes — quality, training efficiency, inference latency, inference throughput, and memory usage — and the choice depends on which constraints are binding for a specific deployment. The paper's results support the following decision logic:

Prefer Griffin (hybrid) when you need both competitive quality and efficient long-context inference, and you can accept a small, fixed-size local attention cache. Griffin matches or exceeds Transformer quality at all tested scales (Figure 1a, Table 1), maintains high throughput at long sequences (Figure 1b), and can perform precise retrieval within its attention window (Figure 6c). This makes it suitable for most production language model deployments — chatbots, code assistants, document analysis — where the 1024-token local attention window covers the majority of retrieval needs and the recurrent layers handle longer-range context. The primary trade-off is the slightly larger parameter count (the recurrent block was matched to MHA, not MQA).
Prefer Hawk (pure RNN) when inference throughput and memory footprint are the absolute binding constraints, and you can accept reduced zero-shot retrieval capability (which can potentially be recovered through fine-tuning). Hawk eliminates the KV cache entirely, achieving the highest throughput of any model in the paper (Figure 1b) and the smallest possible memory footprint during decoding (only the fixed hidden state, no attention cache). Hawk is the right choice for on-device deployment, high-volume batch inference where every millisecond of latency matters, or applications where the context length is expected to be enormous (100K+ tokens) and any attention cache, even local, would become burdensome. The paper shows Hawk's downstream quality is competitive with Transformers (Table 1), and the gap in held-out loss closes as scale increases (Figure 1a).
Prefer an MQA Transformer (global attention) when your application requires precise retrieval of arbitrary tokens from contexts that may exceed the local attention window size (1024 tokens in Griffin's configuration), and you cannot rely on the recurrent state to faithfully preserve that information. The phonebook lookup results (Figure 6c) show that the MQA Transformer perfectly retrieves information up to its training sequence length, whereas Griffin degrades beyond the attention window. If your use case involves, for example, answering questions about specific facts buried in 10K-token legal documents where the relevant passage may be far from the query, a global-attention Transformer (or a Griffin variant with a much larger local attention window, which would increase cache size) may be necessary. However, the paper's long-context extrapolation results (Figure 5) show that Transformers' advantage on retrieval is coupled with a disadvantage on general next-token prediction at very long contexts — Transformers cannot extrapolate beyond their training length, while Griffin can — so this trade-off is context-length-dependent.
Do not prefer any of these when your deployment hardware has no efficient implementation of the RG-LRU scan. The paper's training and inference efficiency claims for Hawk and Griffin depend on the custom Pallas linear scan kernel (Section 4.2) that achieves ~3× speedup over the naive Jax implementation on TPU-v3. On hardware without an equivalent optimized kernel — most notably Nvidia GPUs at the time of writing, unless an optimized CUDA implementation has been developed subsequently — training and inference may be substantially slower than the paper reports, potentially erasing the efficiency advantage over Transformers. This is a temporary limitation that will resolve as optimized kernels are developed for other hardware platforms, but it is a practical barrier to immediate adoption outside TPU environments. Teams considering Griffin for GPU training should budget time for kernel development or wait for community implementations.