Retentive Network: A Successor to Transformer for Large Language Models

ArXiv: 2307.08621

🎯 Pitch

RetNet collapses the tradeoff between fast parallel training and cheap O(1) inference into a single architecture by revealing that attention and recurrence are mathematically dual forms of the same sequence model. For a 7B model, this unifies training parallelism with an 8.4× decoding speedup and a 70% GPU memory reduction compared to Transformers.

---

1. Executive Summary

This paper introduces Retentive Network (RetNet) as a foundation architecture for large language models that simultaneously achieves training parallelism, low-cost inference, and competitive performance — breaking through what the authors call the "impossible triangle." Evaluated on language modeling benchmarks using models up to 6.7B parameters trained on a curated corpus of The Pile, C4, and The Stack, RetNet introduces the retention mechanism with three computation paradigms — parallel representation for training (enabling full GPU utilization), recurrent representation for O(1) inference (eliminating the key-value cache), and chunkwise recurrent representation for efficient long-sequence modeling (encoding each local block in parallel while recurrently summarizing across chunks). For a 7B model with 8k input length, RetNet decodes 8.4× faster and saves 70% of GPU memory compared to Transformer with key-value caches, while achieving comparable scaling curves and downstream task performance, establishing that the "impossible triangle" can be resolved only when the sequence modeling mechanism supports dual parallel-recurrent forms derived from a shared theoretical foundation.

2. Context and Motivation

The Core Problem: The Transformer's Inference Bottleneck

This paper addresses a fundamental tension in the design of large language model architectures. The Transformer, since its introduction in "Attention is All You Need" (Vaswani et al., 2017), has become the de facto architecture for LLMs precisely because it solved a critical problem: training parallelism. Unlike recurrent neural networks (RNNs) that process tokens one at a time in sequence, Transformers process all tokens simultaneously through the self-attention mechanism, enabling efficient utilization of GPU hardware during training. This is the reason Transformers enabled the scaling revolution — without training parallelism, training models on hundreds of billions of tokens would be prohibitively slow.

However, the paper identifies a crucial asymmetry: the very mechanism that enables training parallelism creates an inference bottleneck. During autoregressive decoding, the Transformer must attend to all previously generated tokens to produce each new token. This requires storing a growing key-value (KV) cache that scales linearly with sequence length, leading to $O(N)$ complexity per decoding step, where $N$ is the sequence length. The paper cites Shazeer (2019) to frame this as a "memory-bound key-value cache" problem. As the paper states in Section 1:

"training parallelism of Transformers is at the cost of inefficient inference, because of the $O(N)$ complexity per step and memory-bound key-value cache, which renders Transformers unfriendly to deployment. The growing sequence length increases GPU memory consumption as well as latency and reduces inference speed."

This is not a minor implementation detail — it directly impacts real-world deployment. When a user interacts with an LLM, every generated token requires scanning the entire conversation history stored in the KV cache. For long conversations, document analysis tasks, or high-throughput serving scenarios, this linear growth in memory and computation becomes the dominant factor in deployment cost and user-perceived latency. The paper's Figure 6 demonstrates this concretely: for a 6.7B model, GPU memory grows approximately linearly from ~15GB to ~42GB as sequence length increases from 2048 to 8192, while throughput drops from roughly 280 words-per-second to roughly 80 words-per-second.

The "Impossible Triangle" — Three Desiderata That Previous Approaches Failed to Achieve

The paper crystallizes the architectural challenge as the "impossible triangle" (Figure 2), where three properties are needed but no existing architecture achieves all three simultaneously:

1. Training parallelism: The ability to process tokens in parallel during training, enabling efficient use of GPU hardware and making large-scale training feasible.
2. Low-cost inference: $O(1)$ complexity per decoding step in terms of memory and computation, enabling fast, memory-efficient generation regardless of context length.
3. Good performance: Competitive language modeling quality and downstream task performance — architectures cannot sacrifice capability for efficiency.

The deeper significance is that these three properties have remained mutually exclusive, forcing practitioners into a choose-two-out-of-three compromise. Each compromise has real consequences:

• Choosing training parallelism + good performance gives you the Transformer, but you pay the inference cost. This is the status quo — the entire LLM deployment ecosystem has been built around accommodating the Transformer's KV cache, through techniques like KV-cache quantization, paged attention, and speculative decoding. These are workarounds for an architectural limitation.
• Choosing low-cost inference + good performance gives you recurrent models, which decode efficiently but cannot be parallelized during training. As the paper notes, training RNNs on modern-scale datasets would be prohibitively slow, fundamentally limiting how large these models can become.
• Choosing training parallelism + low-cost inference gives you linearized attention or state-space models, which have been the focus of substantial research but consistently underperform Transformers in modeling quality.

The "impossible" framing is deliberate and provocative. It positions the architectural problem as a fundamental tradeoff imposed by the math of sequence modeling, not a temporary engineering challenge. The paper's core claim is that this triangle is not, in fact, impossible — but it requires a mechanism that unifies parallel and recurrent computation within a single mathematical framework.

Three Strands of Prior Work and Their Specific Shortcomings

The paper organizes previous attempts to resolve the impossible triangle into three research directions (Section 1), each with distinct theoretical approaches and identifiable failure modes:

Strand 1: Linearized Attention

The first approach approximates standard attention scores $\text{softmax}(q_i \cdot k_j)$ using kernel methods, rewriting attention as $\phi(q_i) \cdot \phi(k_j) / \sum_{n=1}^{|x|} \phi(q_i) \cdot \phi(k_n)$, where $\phi$ is a feature map. The key reference is Katharopoulos et al. (2020). The attraction is elegant: by removing the softmax, the computations factor into a form that permits recurrent inference — the attention matrix never needs to be materialized in full, and previous context can be summarized in a fixed-size state.

Why it fails, per the paper: Linearized attention achieves the computational goal (training parallelism + $O(1)$ inference) but falls short on performance. The paper is explicit:

"the modeling capability and performance are worse than Transformers, which hinders the method's popularity"

The technical reason, which the paper implies rather than exhaustively explains, is that kernel-based approximations lose the sharp selectivity of softmax attention. Softmax attention naturally focuses on a small number of highly relevant tokens — the exponential in the softmax creates a winner-take-all dynamic. Linearized attention distributes attention more uniformly, which may be insufficient for the kind of precise token-to-token routing that complex reasoning tasks require. Additionally, the paper notes that linear attention "struggles to effectively encode position information, rendering the models less performant" (Section 2.4).

Strand 2: Recurrent Models with Element-Wise Acceleration

The second approach returns to recurrent architectures — RNNs that naturally have $O(1)$ inference — but attempts to accelerate the inherently sequential training process. The paper specifically references RWKV (Peng et al., 2023) and Attention Free Transformer (AFT), which use element-wise operators to trade modeling capacity for training speed. RWKV replaces dot-product attention with simpler per-element computations and employs exponential decay for relative position information.

Why it fails, per the paper: The training acceleration comes at a cost to representation capacity. The paper argues:

"element-wise operators are used for acceleration, however, representation capacity and performance are harmed"

The technical issue is that element-wise operations (where each token position is transformed independently, perhaps with a learned decay but without dense token-to-token interactions) cannot capture the same richness of dependencies as the full $QK^T$ matrix multiplication in attention. The representational bottleneck is a rank constraint — element-wise operations, by construction, produce a diagonal or highly structured interaction matrix, whereas full attention can represent arbitrary (softmax-normalized) token-to-token interactions.

A critical practical problem the paper highlights: RWKV runs models recurrently for both training and inference, as noted in Table 1. This means RWKV does not achieve training parallelism — it processes tokens sequentially even during training. While element-wise operations make each sequential step cheaper, the fundamental sequential dependency remains, limiting how fast large-scale training can proceed compared to the fully parallel Transformer.

Strand 3: Structured State-Space Models (S4 and Variants)

The third approach replaces attention entirely with structured state-space models (SSMs). The key reference is S4 (Gu et al., 2021), which models sequences through a continuous-time state-space formulation discretized for deep learning. H3 (Dao et al., 2022) and Hyena (Poli et al., 2023) are variants that extend these ideas. These models achieve near-linear or $O(N \log N)$ training complexity and $O(1)$ inference.

Why it fails, per the paper: The issue is primarily performance, though the paper is less explicit about the mechanism. Table 1 shows that S4/H3/Hyena achieve training parallelism, $O(1)$ inference cost, and linear or $O(N \log N)$ memory — but they have not matched Transformer performance on language modeling benchmarks at scale. The structural constraints of SSMs (a fixed state transition matrix that is not content-aware) may limit their ability to selectively attend to information based on the specific input content — a capability that makes Transformers so effective at tasks requiring flexible reasoning.

The paper positions RetNet's retention mechanism as superior because it is content-aware — $Q$ and $K$ are computed as projections of the input $X$, making the interaction between tokens depend on their actual values, not just their positions. The authors note in Section 2.4:

"unlike Equation (2), if $Q_n$ and $K_n$ are content-unaware, the formulation can be degenerated to S4"

This reveals a key architectural insight: S4 is essentially a special case of retention where the query and key projections are fixed rather than learned from the input. RetNet recovers S4-like behavior when $Q$ and $K$ are content-unaware, but gains additional modeling capacity by making these projections input-dependent.

Summary of Prior Work Failure Modes

The paper's assessment is blunt: "None of the previous work can break through the impossible triangle, resulting in no clear winner compared with Transformers" (Section 1). All prior approaches sacrifice at least one of the three desiderata.

How RetNet Positions Itself: Unified Dual-Form Sequence Modeling

The paper's position is that the impossible triangle is only impossible if the sequence modeling mechanism is confined to a single computational form. The key insight is that parallel (matrix-multiplication-based) and recurrent (state-based) computation need not be different mechanisms — they can be different representations of the same underlying mathematical operation.

The paper derives this from scratch. Rather than trying to approximate attention (like linear attention), or accelerate RNNs (like RWKV), or replace attention with a different operation (like S4), the paper re-derives sequence modeling from first principles, starting with a recurrent state-space formulation:

$s_n = A s_{n-1} + K_n^\top v_n$ $o_n = Q_n s_n$

and then shows mathematically that the same computation can be rewritten in a parallel, attention-like form through diagonalization of the transition matrix $A$. The result is the retention mechanism (Equation 5), which is exactly equivalent to the recurrent formulation — not an approximation, not a simplification, but an algebraic rewrite.

This is fundamentally different from prior work. Linear attention approximates softmax attention with kernels — it is a different computation that roughly behaves like attention. Retention is the same computation expressed in two forms: one that is efficiently parallelizable (for training) and one that is $O(1)$ recurrent (for inference). The equivalence is exact, meaning there is no performance tradeoff from switching between forms:

"we can train the models in a parallel way while recurrently conducting inference" (Section 2.1)

The chunkwise recurrent form (Equation 7) fills in the final piece: it hybridizes the parallel and recurrent forms to enable efficient long-sequence training by processing local blocks in parallel while maintaining a recurrent summary of global context. This completes the computational picture:

• Training: Use parallel representation within chunks, recurrent across chunks → linear memory, GPU-efficient.
• Inference: Use pure recurrent representation → $O(1)$ per step, no KV cache needed.
• Long sequences: Use chunkwise recurrence → linear complexity in sequence length.

The paper positions this unification as validating a specific theoretical claim: that the "impossible triangle" resolves only when both the parallel and recurrent forms are aspects of a single, well-defined computation — not when different approximations or compromises are stitched together.

Deeper Motivation: The Inference-Dominated Future of LLMs

While the paper does not explicitly frame it this way, there is a clear implicit motivation: as LLMs move from research to deployment, inference costs dominate. Training is a one-time capital expense; inference is an ongoing operational expense that scales with usage. A model that is 8.4× faster at decoding and uses 70% less GPU memory (as shown in Figure 1 for the 7B model at 8k context) translates directly to serving cost reduction and improved user experience through lower latency.

The paper also hints at deployment scenarios beyond cloud serving. The acknowledgement mentions interest in "deploying RetNet models on various edge devices, such as mobile phones." An architecture with $O(1)$ inference and minimal memory overhead is far more amenable to on-device deployment than a Transformer that must store a growing KV cache for every conversation. This motivation — making LLMs practical beyond the data center — underlies the paper's emphasis on memory, throughput, and latency measurements.

The training efficiency claim is also practically significant: even compared to FlashAttention (Dao et al., 2022), which is a highly optimized Transformer implementation, RetNet achieves competitive or better throughput with lower memory consumption using vanilla PyTorch code (Table 4). The implication is that RetNet achieves efficiency through algorithmic structure, not just engineering optimization, and has further room for improvement through kernel fusion.

In summary, the paper addresses a gap that is both theoretically fundamental (can we unify parallel and recurrent computation?) and practically urgent (can we deploy LLMs without the KV cache bottleneck?), and positions itself as the first architecture to resolve all three constraints through mechanism unification rather than approximation or tradeoff.

3. Technical Approach

3.1 Reader Orientation

Retentive Network (RetNet) is a stack of neural network layers — like Transformer — but where the core sequence-mixing operation is replaced with a new mechanism called retention, which supports three mathematically equivalent but computationally different modes: parallel (for training), recurrent (for memory-efficient $O(1)$ inference), and chunkwise recurrent (for efficient long-sequence training). The architecture solves the problem that Transformers require a growing key-value cache during generation, making inference expensive: RetNet achieves the same training parallelism and modeling quality as Transformers while enabling constant-cost per-token inference by deriving parallel and recurrent computation as two algebraic forms of the same underlying mathematical operation, rather than as approximations of each other.

3.2 Big-Picture Architecture (Diagram in Words)

A RetNet model consists of L identical blocks stacked sequentially, each containing two sub-modules in the standard residual pre-LayerNorm layout familiar from Transformers:

1. Multi-Scale Retention (MSR) module: The novel sequence-mixing component that replaces self-attention. Given a sequence of token representations, it produces context-aware outputs using the retention mechanism with multiple decay rates (one per attention head). This module can run in three modes — parallel, recurrent, or chunkwise recurrent — depending on whether the system is training, inferring, or handling long sequences.

2. Feed-Forward Network (FFN) module: A standard two-layer MLP with GELU activation, identical in role to the Transformer FFN, but with a narrower intermediate dimension (see parameter allocation details in Section 3.4) to match total parameter count.

Information flows: input tokens → word embedding → $X^0$ (packed embeddings) → for $l = 1$ to $L$: $Y^l = \text{MSR}(\text{LN}(X^{l-1})) + X^{l-1}$, then $X^l = \text{FFN}(\text{LN}(Y^l)) + Y^l$ → final output $X^L$.

The retention mechanism itself has internal components that vary by computation mode: in parallel mode, it materializes a $|x| \times |x|$ interaction matrix like attention (but without softmax); in recurrent mode, it maintains a fixed-size state matrix $S_n \in \mathbb{R}^{d \times d}$ across timesteps; in chunkwise mode, it hybridizes both — processing local chunks with the parallel form and passing cross-chunk information through the recurrent state.

3.3 Roadmap for the Deep Dive

• First, the core retention mechanism and its derivation from a recurrent state-space model, which establishes the mathematical foundation that enables the dual parallel-recurrent form. This is the "why it works" section — the derivation shows that retention is not an approximation of attention but rather an exact algebraic rewrite of a recurrence.
• Second, the three computation paradigms (parallel, recurrent, chunkwise recurrent) in full detail, with their exact equations, when each is used, and the computational properties that make each suitable for its context.
• Third, the multi-scale retention (MSR) module, which introduces multiple decay rates across heads, the gating mechanism, and the GroupNorm-based normalization scheme — the engineering components that make the mathematical mechanism work well in practice.
• Fourth, the overall RetNet architecture — how MSR and FFN are combined, parameter allocation to match Transformer's parameter count, training protocol, and inference protocol.
• Fifth, practical design choices and numerical stabilization tricks (score normalization, decay initialization, head dimension selection) that are essential for stable training and are validated through ablation studies.
• Sixth, the relationship to prior methods, explaining precisely how retention differs from linear attention, S4, and RWKV — this clarifies what is genuinely novel versus what builds on existing ideas.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architecture design paper whose core idea is that sequence modeling can be formulated with a mechanism that is exactly equivalent in its parallel and recurrent forms, rather than requiring one form to approximate the other. The key insight is that starting from a linear recurrent state-space model with a carefully chosen diagonalizable transition matrix, and making the query and key projections content-aware, yields a mechanism that trains like attention but infers like an RNN.

---

Derivation of the Retention Mechanism: From Recurrence to Attention

The paper does not start with attention and try to make it recurrent (the linear attention approach). Instead, it starts with a recurrent state-space model and derives its parallel equivalent. This derivation order is conceptually important: the recurrence is the definition, and the parallel form is the consequence.

Consider a sequence modeling problem where we want to map an input function $v(n)$ to an output $o(n)$ through a hidden state $s_n$. At each timestep $n$, we receive a new input $v_n = v(n)$ (a vector derived from the input token) and update a state $s_n$:

$s_n = A s_{n-1} + K_n^\top v_n$

where:

• $s_n \in \mathbb{R}^{d \times d}$ is the state matrix at time $n$,
• $A \in \mathbb{R}^{d \times d}$ is a transition matrix that controls how past information decays,
• $K_n \in \mathbb{R}^{1 \times d}$ is a row vector that projects the current input $v_n$ to the state space,
• $v_n \in \mathbb{R}^d$ is a value vector derived from the input token.

The output $o_n$ is then produced by querying the state:

$o_n = Q_n s_n$

where $Q_n \in \mathbb{R}^{1 \times d}$ is a row vector that reads from the state.

What this computes operationally: At each time step, the model takes the current input value $v_n$, encodes it into a "key" form $K_n^\top v_n$ (an outer product producing a $d \times d$ matrix), adds it to the exponentially decayed previous state $A s_{n-1}$, and stores the result in $s_n$. The output $o_n$ is obtained by multiplying the state by a query vector $Q_n$. This is a linear recurrence: the state summarizes all past information through additive updates, with $A$ controlling the decay rate of older information.

Why this form: The linearity is essential — it is what allows the recurrence to be unrolled into a parallel computation later. A nonlinear recurrence (like an LSTM) could not be algebraically rewritten into a matrix multiplication form. The additive update means each token's contribution can be isolated and computed independently if we know the decay factors.

By unrolling the recurrence, we can express the output directly in terms of all past inputs without the intermediate state:

$o_n = \sum_{m=1}^n Q_n A^{n-m} K_m^\top v_m$

What this says: The output at position $n$ is a sum over all past positions $m$, where each past value $v_m$ is weighted by (1) a query-key interaction $Q_n A^{n-m} K_m^\top$ that depends on both the current and past positions, and (2) the transition matrix $A$ raised to the power $n-m$, implementing exponential decay with distance. This looks structurally like attention — a weighted sum over past values — but with a specific form for the weights.

Now comes the crucial step: the paper makes $Q_n$ and $K_n$ content-aware by computing them as learned linear projections of the input:

$Q = XW_Q, \quad K = XW_K$

where $X \in \mathbb{R}^{|x| \times d_{\text{model}}}$ is the input sequence, and $W_Q, W_K \in \mathbb{R}^{d \times d}$ are learnable weight matrices (per head).

Why content-awareness matters: If $Q$ and $K$ were fixed (not dependent on the input), the model would be content-unaware, like S4. The whole point of attention in Transformers is that which tokens attend to which other tokens depends on their content. Making $Q$ and $K$ learned projections of $X$ gives retention the same input-dependent routing capability as attention.

The transition matrix $A$ is then diagonalized:

$A = \Lambda (\gamma e^{i\theta}) \Lambda^{-1}$

where:

• $\gamma \in \mathbb{R}^d$ is a vector of real decay rates (one per dimension),
• $\theta \in \mathbb{R}^d$ is a vector of rotation angles (one per dimension),
• $e^{i\theta}$ applies a complex rotation in each dimension,
• $\Lambda$ is the eigenvector matrix.

What diagonalization achieves: Raising a diagonal matrix to a power is cheap (just raise each diagonal entry) — $(\gamma e^{i\theta})^{n-m}$ is computed element-wise, giving $\gamma^{n-m} e^{i\theta(n-m)}$, which is exponential decay ($\gamma^{n-m}$) combined with rotation proportional to distance ($e^{i\theta(n-m)}$). The rotation component serves as a relative position encoding, since the rotation angle depends only on the distance $n-m$ between positions, not on the absolute positions.

By absorbing $\Lambda$ into the weight matrices (since $\Lambda$ is learned and fixed, the projection $W_Q \Lambda$ is equivalent to a different learned weight matrix), the unrolled form simplifies to:

$o_n = \sum_{m=1}^n Q_n (\gamma e^{i\theta})^{n-m} K_m^\top v_m$

Further simplifying $\gamma$ to a scalar per head (shared across dimensions within a head):

$o_n = \sum_{m=1}^n \gamma^{n-m} (Q_n e^{i n \theta}) (K_m e^{i m \theta})^\dagger v_m$

where $^\dagger$ denotes the conjugate transpose, and $\Theta_n = e^{i n \theta}$ rotates query and key vectors by an amount proportional to their position — this is the xPos relative position encoding (Sun et al., 2022), which the paper acknowledges was proposed for Transformers and now emerges naturally from the derivation.

Key insight of the derivation: Starting from a linear recurrence with a diagonalizable complex transition matrix $A$, and making queries and keys content-aware, we arrive at a formulation that is structurally identical to attention with exponential decay and rotary position embeddings, but without the softmax. The softmax in standard attention prevents the recurrent formulation — softmax requires seeing all scores to normalize, which is fundamentally non-recurrent. Removing softmax and using exponential decay instead (through $\gamma^{n-m}$) is what makes the dual form possible.

What this derivation shows about the "impossible triangle": The parallel and recurrent forms are not two different mechanisms approximating each other — they are the same computation expressed in two algebraically equivalent ways. The recurrence (Equation 1) is the definition; the parallel form (Equation 5) is the unrolled version. Switching between them costs nothing in modeling fidelity because they are mathematically identical. This is the foundational reason RetNet can achieve training parallelism and $O(1)$ inference simultaneously.

---

The Parallel Representation of Retention

The parallel form is derived by applying the unrolled equation to all positions simultaneously in matrix form:

$Q = (XW_Q) \odot \Theta, \quad K = (XW_K) \odot \bar{\Theta}, \quad V = XW_V$

$\Theta_n = e^{i n \theta}, \quad D_{nm} = \begin{cases} \gamma^{n-m}, & n \geq m \\ 0, & n < m \end{cases}$

$\text{Retention}(X) = (Q K^\top \odot D) V$

where:

• $X \in \mathbb{R}^{|x| \times d_{\text{model}}}$ is the input sequence,
• $W_Q, W_K, W_V$ are learnable weight matrices,
• $\Theta \in \mathbb{C}^{|x| \times d}$ applies complex rotation by $n\theta$ to each position $n$,
• $\bar{\Theta}$ is the complex conjugate of $\Theta$ (rotation by $-m\theta$),
• $\odot$ denotes element-wise multiplication,
• $D \in \mathbb{R}^{|x| \times |x|}$ is a causal mask combined with exponential decay — it is zero for $n < m$ (causal: position $n$ cannot attend to positions after it) and $\gamma^{n-m}$ for $n \geq m$ (exponential decay with distance).

What it computes: First, queries and keys are rotated by position-dependent complex rotations ($\Theta_n = e^{i n \theta}$ and its conjugate), encoding absolute position information that through the dot product becomes relative position (the $n\theta - m\theta = (n-m)\theta$ in the exponent). Then the query-key similarity matrix $QK^\top \in \mathbb{C}^{|x| \times |x|}$ is computed, element-wise multiplied by the decay-and-causal mask $D$, and used to weight the values $V$. The result is a matrix of the same shape as $X$, where each position's output is a sum of all previous positions' values, weighted by query-key similarity and exponentially decayed by distance.

Why this form enables training parallelism: The computation is a sequence of matrix multiplications ($QK^\top$, then $\odot D$, then $\times V$) that can be executed efficiently on GPUs. All positions are processed simultaneously — there is no sequential dependency within the layer during training. This is exactly why Transformers train efficiently, and retention preserves this property.

Why the complex rotations: The $\Theta$ and $\bar{\Theta}$ factors implement relative position encoding through absolute position encoding. When computing $(Q_n e^{i n \theta})(K_m e^{i m \theta})^\dagger = Q_n K_m^\dagger e^{i (n-m) \theta}$, the absolute positions $n$ and $m$ cancel to leave a factor depending only on relative distance $n-m$. This means the model can learn position-dependent attention patterns (e.g., attend more to recent tokens) without being limited to a fixed maximum context length — the relative encoding generalizes to any sequence length.

Critical difference from attention: There is no softmax. In standard attention, the scores $QK^\top / \sqrt{d}$ are passed through softmax to produce a probability distribution, which normalizes the scores and introduces competition between positions. In retention, the exponential decay $\gamma^{n-m}$ serves as a non-competitive, distance-based weighting — each past token's contribution is decayed independently, regardless of what other tokens are present. This non-competitive nature is what enables the recurrent form: if scores depended on all other tokens' scores (as softmax does), you could not maintain a fixed-size state.

---

The Recurrent Representation of Retention

The recurrent form is obtained by recognizing that the unrolled sum can be computed incrementally with a fixed-size state matrix:

$S_n = \gamma S_{n-1} + K_n^\top V_n$

$\text{Retention}(X_n) = Q_n S_n, \quad n = 1, \ldots, |x|$

where:

• $S_n \in \mathbb{R}^{d \times d}$ is the state matrix at step $n$,
• $K_n^\top \in \mathbb{R}^{d \times 1}$ is the transpose of the key vector, so $K_n^\top V_n \in \mathbb{R}^{d \times d}$ is an outer product,
• $\gamma$ is the scalar decay rate,
• $Q_n \in \mathbb{R}^{1 \times d}$ is the query vector,
• $Q_n S_n \in \mathbb{R}^{1 \times d}$ is the output (the query reads from the state).

What it computes: At inference time, for each new token $n$, the model computes its key $K_n$ and value $V_n$, forms the outer product $K_n^\top V_n$ (which encodes the token's contribution to future context), and adds it to the exponentially decayed previous state $\gamma S_{n-1}$. The state $S_n$ now summarizes all past tokens, with older tokens receiving exponentially smaller weight due to the repeated multiplication by $\gamma$. The output is then $Q_n S_n$ — the current query reads from the accumulated state to produce context-aware output.

Why this is $O(1)$: The state $S_n$ is a fixed-size $d \times d$ matrix regardless of sequence length. Each new token requires: (1) computing $K_n$ and $V_n$ from the input, (2) the outer product $K_n^\top V_n$ and addition to $\gamma S_{n-1}$ — cost $O(d^2)$, and (3) the query-state product $Q_n S_n$ — also $O(d^2)$. The cost per token does not grow with sequence length. In contrast, Transformer's KV cache grows to $O(nd)$ memory and self-attention over the cache costs $O(nd)$ computation per token.

Memory implications: The Transformer's KV cache stores $K_{1:n} \in \mathbb{R}^{n \times d}$ and $V_{1:n} \in \mathbb{R}^{n \times d}$ for each layer and each head. For $n = 8192$, $d = 128$ (head dimension), 32 layers, this is $2 \times 8192 \times 128 \times 32 \approx 67\text{M}$ floats. RetNet's state is a single $d \times d = 128 \times 128 = 16\text{k}$ matrix per head per layer — the same for all sequence lengths. This is why Figure 6a shows RetNet's memory staying flat while Transformer's grows linearly.

Crucially — exact equivalence: The recurrent form computes exactly the same outputs as the parallel form would produce, not an approximation. This is because the recurrence is the definition, and the parallel form is just the unrolled computation. There is no information lost in the recurrent state because the linear recurrence with exponential decay is Markovian — the state $S_n$ is a sufficient statistic for all past tokens when computing future outputs under the exponential decay weighting.

---

The Chunkwise Recurrent Representation of Retention

For long sequences during training, neither the pure parallel form (which requires $O(|x|^2)$ memory for the $QK^\top \odot D$ matrix) nor the pure recurrent form (which is sequential and cannot use GPU parallelism) is ideal. The chunkwise form hybridizes both:

The input sequence is divided into chunks of length $B$. Within each chunk $i$, the parallel form (Equation 5) is used — this exploits GPU parallelism for the local computation. Across chunks, a recurrent state $R_i$ is maintained that summarizes all chunks up to $i-1$:

$Q_{[i]} = Q_{Bi:B(i+1)}, \quad K_{[i]} = K_{Bi:B(i+1)}, \quad V_{[i]} = V_{Bi:B(i+1)}$

$R_i = K_{[i]}^\top (V_{[i]} \odot \zeta) + \gamma^B R_{i-1}, \quad \zeta_{ij} = \gamma^{B-i-1}$

$\text{Retention}(X_{[i]}) = \underbrace{(Q_{[i]} K_{[i]}^\top \odot D) V_{[i]}}_{\text{Inner-Chunk}} + \underbrace{(Q_{[i]} R_{i-1}) \odot \xi}_{\text{Cross-Chunk}}, \quad \xi_{ij} = \gamma^{i+1}$

where:

• $B$ is the chunk size (set to 512 in the paper's experiments),
• $Q_{[i]}, K_{[i]}, V_{[i]} \in \mathbb{R}^{B \times d}$ are the slices of the query, key, and value matrices for chunk $i$,
• $R_i \in \mathbb{R}^{d \times d}$ is the recurrent state summarizing chunks $0$ through $i$,
• $\zeta \in \mathbb{R}^B$ accounts for the decay within the chunk (each position in the chunk is at a different distance from the chunk boundary),
• $\gamma^B$ decays the previous state by $B$ steps (since the recurrence jumps across the entire chunk at once),
• $D \in \mathbb{R}^{B \times B}$ is the causal decay mask (same as in Equation 5 but for the chunk-local positions),
• $\xi \in \mathbb{R}^B$ adjusts the cross-chunk contribution based on position within the current chunk.

What it computes, step by step:

1. Inner-chunk computation: For positions within chunk $i$, compute attention-like interactions using the parallel form — $Q_{[i]} K_{[i]}^\top \odot D$ gives the within-chunk attention scores, which weight $V_{[i]}$. This is fast because $B$ is small (512), and all positions in the chunk are processed in parallel.
2. Cross-chunk computation: $Q_{[i]} R_{i-1}$ queries the recurrent state representing all previous chunks — this is an $O(B \times d \times d)$ operation, not $O(B \times n)$. The result is element-wise multiplied by $\xi$ to account for positional decay.
3. State update: $R_i$ is updated by adding the chunk's contribution $K_{[i]}^\top (V_{[i]} \odot \zeta)$ (with correct internal decay) to the decayed previous state $\gamma^B R_{i-1}$. This state is then used for subsequent chunks.

Why this is efficient: The inner-chunk computation costs $O(B^2 d)$ (the attention matrix for the chunk); the cross-chunk computation costs $O(B d^2)$ (querying the recurrent state); and the state update costs $O(B d^2)$. With $B$ set moderately (512) and $d$ typically smaller than $B$ (head dimension 256), the total is approximately $O(B^2 d)$ per chunk — linear in the total sequence length when amortized across chunks. The pure parallel form would cost $O(n^2 d)$, which is quadratic.

When each representation is used:

• Training (standard): Chunkwise recurrence (Equation 7) with chunk size 512, enabling efficient training on 8k+ sequences.
• Training (very short sequences): Pure parallel representation (Equation 5) for maximum GPU utilization when $|x|$ is small enough that $|x|^2$ memory is affordable.
• Inference: Pure recurrent representation (Equation 6) for $O(1)$ per-token cost and no KV cache.

The pseudocode in Figure 4 of the paper provides concrete implementations for all three modes, showing their interfaces: ParallelRetention(q, k, v, decay_mask), RecurrentRetention(q, k, v, past_kv, decay), and ChunkwiseRetention(q, k, v, past_kv, decay_mask, chunk_decay, inner_decay).

---

Gated Multi-Scale Retention (MSR) Module

The retention mechanism described above operates on a single head with a single decay rate $\gamma$. The MSR module extends this to multiple heads with different decay rates and adds gating:

Multi-head setup with varying decay rates:

The module uses $h = d_{\text{model}} / d$ retention heads, where $d$ is the head dimension (256 for queries and keys, 512 for values in the paper's experiments). Each head has its own learned weight matrices $W_Q, W_K, W_V \in \mathbb{R}^{d \times d}$.

Crucially, each head is assigned a distinct decay rate $\gamma$:

$\gamma = 1 - 2^{-5 - \text{arange}(0, h)} \in \mathbb{R}^h$

What this formula produces: For $h = 8$ heads with default initialization, this yields $\gamma$ values ranging from approximately $1 - 2^{-5} \approx 0.96875$ (head 0, slow decay, retains information longer) to $1 - 2^{-12} \approx 0.99976$ (head 7, very slow decay, retains information much longer). The exponential spacing means heads cover a wide range of temporal scales.

Why multiple decay rates: Different heads can specialize in different temporal ranges — fast-decaying heads focus on very recent context (syntactic patterns, local coherence), while slow-decaying heads retain long-range dependencies (topic tracking, document-level discourse). This is analogous to how multi-head attention in Transformers allows different heads to learn different attention patterns, but here the differentiation is explicitly structured through the decay rate rather than emerging purely from learned weights.

In the paper's scaling experiments with larger models, $\gamma$ is set to:

$\gamma = 1 - e^{\text{linspace}(\log 1/32, \log 1/512, h)}$

which provides a smooth interpolation in log-space between decay factors of 1/32 and 1/512. The paper keeps $\gamma$ fixed (not learned) and identical across layers for simplicity.

Gating mechanism:

After computing the retention output for each head, the outputs are concatenated, group-normalized, and then gated:

$\text{head}_i = \text{Retention}(X, \gamma_i)$

$Y = \text{GroupNorm}_h(\text{Concat}(\text{head}_1, \ldots, \text{head}_h))$

$\text{MSR}(X) = (\text{swish}(X W_G) \odot Y) W_O$

where:

• $W_G \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$ projects the input to produce gate values,
• $\text{swish}(x) = x \cdot \sigma(x)$ is the Swish activation function (Ramachandran et al., 2017), a smooth alternative to ReLU,
• $\odot$ is element-wise multiplication (gating),
• $W_O \in \mathbb{R}^{2d \times d}$ projects from the widened value dimension back to $d$,
• $\text{GroupNorm}_h$ normalizes each head's output separately (see below).

What the gate does: The input $X$ is projected through $W_G$ and passed through the Swish nonlinearity to produce a gating signal. This gate is multiplied element-wise with the concatenated retention output $Y$. This is analogous to the gating in Gated Linear Units (GLU) and provides additional nonlinearity to the retention layer, which is otherwise a linear operation on the values (the query-key interaction is linear in $V$, and without the gate, the entire MSR module would be a linear function of the input).

The ablation study (Table 6) confirms the gate's importance: removing it ("$-$swish gate") increases in-domain perplexity from 26.05 to 27.84, a degradation of approximately 7%.

Why GroupNorm instead of LayerNorm:

The paper uses GroupNorm (Wu & He, 2018) to normalize each head's output independently, following the Sub-LayerNorm approach (Wang et al., 2022). The reason given in the paper:

"Notice that the heads use multiple $\gamma$ scales, which results in different variance statistics. So we normalize the head outputs separately."

Heads with different decay rates naturally produce outputs with different variance — a fast-decaying head accumulates less information (and thus produces smaller-magnitude outputs) than a slow-decaying head. Applying a single LayerNorm across all heads would force these natural differences into a common scale, potentially washing out the multi-scale benefit. GroupNorm with groups equal to the number of heads normalizes each head independently, preserving the natural scale differences induced by the different decay rates.

The ablation in Table 6 confirms: removing GroupNorm ("$-$GroupNorm") increases perplexity from 26.05 to 27.54.

Parameter allocation (matching Transformer):

To keep total parameter count comparable to Transformer, the paper makes specific choices about dimensions. In a standard Transformer, self-attention uses $4d^2$ parameters ($W_Q, W_K, W_V, W_O$, each $d \times d$) and FFN uses $8d^2$ parameters (with intermediate dimension $4d$). In RetNet:

• $W_Q, W_K \in \mathbb{R}^{d \times d}$ (standard),
• $W_G \in \mathbb{R}^{d \times 2d}$ (gate projection, outputs $2d$),
• $W_V \in \mathbb{R}^{d \times 2d}$ (value projection is doubled),
• $W_O \in \mathbb{R}^{2d \times d}$ (output projection from doubled dimension back to $d$),
• This gives $d \times d + d \times d + d \times 2d + d \times 2d + 2d \times d = 8d^2$ parameters in MSR, compared to $4d^2$ in Transformer self-attention.

To compensate, the FFN intermediate dimension is reduced from $4d$ to $2d$, reducing FFN parameters from $8d^2$ to $4d^2$. Total per-layer parameters: MSR ($8d^2$) + FFN ($4d^2$) = $12d^2$ in RetNet, matching Transformer's self-attention ($4d^2$) + FFN ($8d^2$) = $12d^2$.

The widened value dimension ($2d$ instead of $d$) and gate projection provide additional representation capacity in the retention module itself, which is necessary because retention lacks the nonlinear softmax that attention uses — the additional parameters in MSR compensate for this loss of nonlinearity.

---

Retention Score Normalization (Numerical Stability)

The paper introduces three normalization factors to stabilize the numerical flow of retention layers, exploiting GroupNorm's scale invariance property (multiplying a head's output by a scalar does not change the result after GroupNorm because GroupNorm divides by the standard deviation):

1. Query-key normalization: $QK^\top$ is divided by $\sqrt{d}$ before further computation:

   $\text{scores} = \frac{QK^\top}{\sqrt{d}}$

   This is identical to the scaling used in Transformer attention and prevents the dot-product values from growing with dimension, which would cause vanishing gradients through softmax (or, in retention's case, would cause extreme values before the decay mask multiplication).

2. Decay mask normalization: The decay mask $D$ is replaced with $\tilde{D}$ where each row is normalized by the sum of that row:

   $\tilde{D}_{nm} = \frac{D_{nm}}{\sqrt{\sum_{i=1}^n D_{ni}}}$

   This ensures each position's total "attention mass" over the past is approximately constant, preventing positions early in the sequence (which accumulate mass from many tokens) from having much larger scores than positions late in the sequence. The square root normalization (rather than dividing by the sum) moderates this effect.

3. Score normalization: The retention scores $R = QK^\top \odot D$ are normalized by the maximum absolute row sum:

   $\tilde{R}_{nm} = \frac{R_{nm}}{\max\left(\left|\sum_{i=1}^n R_{ni}\right|, 1\right)}$

   This prevents any single row of the score matrix from dominating, which could cause training instability. The $\max(\cdot, 1)$ prevents division by very small numbers.

After these normalizations, the retention output becomes $\text{Retention}(X) = \tilde{R} V$.

Why these don't change the model's behavior: Because GroupNorm normalizes each head's output to zero mean and unit variance (per head), any scalar multiplication applied before GroupNorm is effectively canceled. If the retention output is multiplied by a constant $\alpha$, then $\text{GroupNorm}(\alpha \cdot \text{head}) = \text{GroupNorm}(\text{head})$ — the $\alpha$ divides out in the standardization. This scale invariance means these normalizations control numerical precision during the forward and backward passes without affecting the model's representational capacity.

---

Overall RetNet Architecture and Training/Inference Protocol

Layer structure:

An $L$-layer RetNet processes input embeddings $X^0$ through $L$ identical blocks:

$Y^l = \text{MSR}(\text{LN}(X^l)) + X^l$

$X^{l+1} = \text{FFN}(\text{LN}(Y^l)) + Y^l$

where $\text{LN}(\cdot)$ is LayerNorm (Ba et al., 2016) and $\text{FFN}(X) = \text{gelu}(X W_1) W_2$ with $W_1 \in \mathbb{R}^{d_{\text{model}} \times 2d_{\text{model}}}$, $W_2 \in \mathbb{R}^{2d_{\text{model}} \times d_{\text{model}}}$. This is the standard pre-LayerNorm residual architecture identical to modern Transformer implementations.

Training protocol:

During training, the chunkwise recurrent representation (Equation 7) is used with chunk size $B = 512$. The paper notes two possibilities:

"We use the parallel (Equation (5)) and chunkwise recurrent (Equation (7)) representations during the training process. The parallelization within sequences or chunks efficiently utilizes GPUs to accelerate computation."

For sequences shorter than or equal to the chunk size, the pure parallel form is used. For longer sequences, the chunkwise form processes each chunk in parallel while maintaining cross-chunk recurrence. The chunk size of 512 balances GPU utilization (larger chunks = better parallelism) with memory efficiency (smaller chunks = less memory for the chunk-local attention matrix).

Inference protocol:

During autoregressive decoding, the pure recurrent representation (Equation 6) is used exclusively. For each new token:

1. The token is embedded and passed through each layer sequentially.
2. In each MSR module, $Q_n$, $K_n$, $V_n$ are computed from the current input.
3. The state $S_n = \gamma S_{n-1} + K_n^\top V_n$ is updated.
4. The output $Q_n S_n$ is computed, passed through GroupNorm and gating.
5. The FFN processes the result.
6. The output is projected to vocabulary logits for the next token prediction.

Model configurations (from Appendix A, Table 7):

All models use: polynomial decay learning rate schedule, 375 warmup steps, 4M tokens per batch, AdamW with $\beta = (0.9, 0.98)$ and weight decay 0.01, gradient clipping 2.0, dropout 0.1, 25,000 training steps (100B tokens total), and DeepNet initialization (Wang et al., 2022) for training stability.

---

Design Choices and Their Justifications

Fixed gamma (not learned): The decay rates $\gamma$ are set to fixed values rather than being learned parameters. This ensures the exponential decay structure is preserved — if $\gamma$ were learned, it could become 1 (no decay, equivalent to uniform attention) or 0 (no memory), destabilizing the recurrent state. Fixed $\gamma$ also means the recurrent computation is deterministic given the inputs, simplifying analysis. The ablation ("$-$multi-scale decay" in Table 6: setting all heads to $\gamma = 127/128$) shows a perplexity increase from 26.05 to 27.02, confirming multi-scale decay improves performance over a single shared rate. Removing decay entirely ("$-$gamma decay," $\gamma = 1$) increases perplexity to 27.86 — even worse, confirming that exponential decay is not just a computational convenience but a useful inductive bias.

Head dimension of 256: The paper uses a larger head dimension (256 for $Q$ and $K$, 512 for $V$) than typical Transformers (which often use 64 or 128). The reasoning from Section 3.6:

"From the recurrent perspective of Equation (1), the head dimension implies the memory capacity of hidden states."

The state matrix $S_n \in \mathbb{R}^{d \times d}$ has capacity that scales with $d^2$ — a 256-dimensional state can store far more information than a 64-dimensional one. The ablation (Table 6, "Reduce head dimension" from 256 to 64) shows perplexity increasing from 26.05 to 27.68, confirming that larger head dimension (fewer heads with higher state capacity) is beneficial for retention.

Content-aware $Q$ and $K$ through learnable projections: This distinguishes RetNet from S4. In S4, the state transition and output matrices are fixed (or learned once, independent of input), making the model content-unaware — it processes all inputs the same way regardless of their content. In retention, $Q = XW_Q$ and $K = XW_K$ means the model dynamically decides which past information to retrieve based on the current input content, which is the key capability that makes attention powerful.

No softmax and its consequences: The absence of softmax in retention is deliberate and necessary — softmax would make the recurrent form impossible because the normalization depends on all scores globally. However, removing softmax means the model loses the natural normalization that prevents score explosion and the competition between positions. The paper compensates with: (a) the three numerical normalization tricks (score normalization section above), (b) the gating mechanism (provides nonlinearity that softmax would otherwise provide), and (c) the fixed exponential decay (provides a structured alternative to softmax's learned sharpness).

Two-fold value dimension ($2d$ for $V$, $d$ for $Q$ and $K$): The value dimension is doubled to $2d$, then projected back to $d$ by $W_O$. This gives the value projections more capacity to encode information that will be stored in the state matrix $S_n$. Since the state matrix is the only way information persists across timesteps in the recurrent form, having richer value representations is particularly important. The extra parameters in MSR ($8d^2$ vs. Transformer's $4d^2$) are balanced by reducing FFN parameters.

DeepNet initialization: The models use DeepNet initialization (Wang et al., 2022), which scales residual connections to maintain constant variance through deep networks. This is particularly important for RetNet because the retention mechanism involves repeated matrix multiplications that could cause exploding or vanishing signals in deep stacks.

4. Key Insights and Innovations

Innovation 1: The "Impossible Triangle" as a Unifying Diagnostic Framework

The paper's most distinctive conceptual contribution is not any single piece of math but rather the diagnostic reframing of the architecture design space through the "impossible triangle" (Figure 2). Before RetNet, the tension between training parallelism, inference cost, and model quality was understood implicitly — practitioners knew Transformers were expensive at inference, and knew recurrent models were slow to train — but the field lacked a crisp framework that made the tradeoffs explicit and identified why prior solutions were partial.

The "impossible triangle" crystallizes three desiderata — training parallelism, $O(1)$ inference, and competitive performance — and asserts that no prior architecture achieves all three simultaneously. This is not a mathematical theorem but a pattern diagnosis backed by the systematic comparison in Table 1, which maps six architecture families onto the three axes. The diagnosis is sharp: Transformer achieves parallelism and performance but fails on inference cost; linear attention achieves parallelism and low-cost inference but fails on performance; RWKV achieves low-cost inference and decent performance but fails on training parallelism; S4/H3 achieve parallelism and low-cost inference but fail on performance. Every prior approach occupies an edge of the triangle, never the center.

What makes this framing significant beyond a convenient taxonomy is that it redefines the architectural search problem. The goal is no longer "make attention more efficient" (which treats the Transformer's inference cost as an engineering problem to be optimized around) or "make RNNs more parallel" (which treats recurrence as a training inefficiency to be mitigated). The goal becomes: find a single computational mechanism whose algebraic structure permits both parallel and recurrent evaluation without approximation. The triangle diagnosis implies that piecemeal fixes — kernel fusion, KV-cache compression, element-wise approximations — are addressing symptoms, not the root constraint. The root constraint is that softmax attention is algebraically non-factorable; any architecture that uses softmax will have $O(N)$ inference. Any architecture that avoids softmax but loses content-awareness (S4) or selectivity (linear attention) will lose performance.

The diagnostic power extends to evaluating future architectures. The paper's own Table 1 serves as a template: any proposed successor architecture can be classified by whether it achieves each of the three properties, immediately revealing which tradeoff it makes. The "impossible" framing also makes RetNet's claimed breakthrough more falsifiable — if any of the three properties is compromised in practice (e.g., if the recurrent form degrades at very long contexts, or if training parallelism is only partial), the triangle remains unbroken. This is a higher bar than simply showing "better than Transformer on benchmark X" and forces the evaluation to be multi-dimensional (Figures 5 and 6, Tables 3 and 4).

The triangle is not just a rhetorical device — it organizes the paper's entire evaluation. The three axes map directly to the three result sections: training cost (Table 4) tests parallelism, inference cost (Figure 6) tests $O(1)$ behavior, and scaling curves plus downstream tasks (Figure 5, Table 3) test performance. A paper that only reported perplexity would not have demonstrated triangle-breaking. The fact that all three axes are evaluated — and that RetNet shows improvement on inference cost without sacrificing the other two — is what makes the claim credible.

This is a fundamental reframing, not an incremental one, because it changes what "architectural progress" means. Rather than optimizing a single metric (perplexity) with an acceptable efficiency budget, the framework demands simultaneous satisfaction of constraints that were previously considered mutually exclusive. The paper does not prove they are logically impossible (it shows they are not), but the diagnosis that they appeared impossible — and that this appearance structured the entire field's research directions — is what gives the framework its explanatory power.

Caveat: The triangle is intuitive but coarse. "Good performance" is a continuous spectrum, not a binary property. The paper's evidence (Table 5) shows RetNet outperforming H3, Hyena, RWKV, and Linear Transformer on language modeling perplexity, but all models except Linear Transformer achieve perplexity within roughly 15% of each other. Whether these differences constitute "good" vs. "not good" performance depends on the application. A stricter version of the triangle might require matching Transformer exactly at all scales, which the paper's scaling curves (Figure 5) suggest RetNet does — but only up to 6.7B parameters. The claim that the triangle is "broken" is well-supported at the scales tested but should be understood as meaning "the tradeoff can be substantially reduced," not "all three properties are perfectly achieved without any residual tension."

---

Innovation 2: Deriving Attention from Recurrence (Not Approximating Attention with Recurrence)

The paper's second major conceptual innovation is the inversion of the standard derivation direction in efficient attention research. The dominant approach — linearized attention, kernel attention, and most Transformer variants — starts with softmax attention and asks: how can we approximate or simplify this to make it cheaper? The approximation is motivated by efficiency, and the performance gap relative to full attention is the cost paid for that efficiency. RetNet inverts this logic: it starts with a recurrent state-space model (Equation 1) as the definition of the computation, and derives its parallel equivalent as an algebraic consequence of diagonalizing the transition matrix and making queries and keys content-aware.

This inversion matters for two reasons. First, it means the recurrent form is exact, not approximate. In linear attention, the kernel $\phi(q)\phi(k)$ approximates $\exp(q \cdot k)$, and the recurrent inference form is an approximation of that approximation — errors compound. In retention, the recurrent form is the definition; the parallel form is an unrolling. Switching between them introduces zero modeling error, because they compute the same function. This is not a claim about empirical similarity — it is a mathematical property of the derivation: Equations 5 and 6 are algebraically equivalent.

Second, the inversion changes what properties the mechanism inherits. When you start from attention and approximate it, you inherit attention's structure — the dot-product interaction, the softmax normalization — and then modify it. The modifications (removing softmax, adding kernel features) are ad hoc from the perspective of the final recurrence. When you start from a recurrence and derive upward, the properties of the parallel form emerge from the recurrence's structure — the exponential decay, the diagonal transition matrix, the complex rotation for position encoding. These are not design choices bolted onto an attention approximation; they are mathematical consequences of choices made in the recurrence (diagonalizability of $A$, content-awareness of $Q$ and $K$, linearity of the state update).

This distinction is most visible in the comparison with S4 (Section 2.4). The paper notes that if $Q_n$ and $K_n$ are content-unaware, retention degenerates to S4. This is not a coincidence — it reveals that S4 is a special case of the retention framework, one where the query and key projections are fixed rather than learned from input. RetNet's contribution is recognizing that making these projections content-aware (through $XW_Q$ and $XW_K$) recovers the key advantage of attention — dynamic, input-dependent routing — while preserving the recurrent efficiency that S4 achieves through fixed state-space dynamics. The derivation path (recurrence → parallel with content-aware $Q$, $K$) makes this generalization natural; starting from attention and trying to make it recurrent would not naturally lead to S4 as a special case.

The inversion also explains why RetNet can use tricks from both the Transformer world (xPos/RoPE position encoding, multi-head design, gating) and the RNN/SSM world (exponential decay, state matrices) without these feeling bolted together. The xPos encoding emerges from the diagonalization of $A$ with complex eigenvalues — it is not a separate position encoding module added to the architecture, but a consequence of the complex rotation in the transition matrix. The multi-scale decay rates across heads emerge from the observation that different heads can use different $\gamma$ values in the recurrence, giving each head a different temporal receptive field. These features are derived, not designed.

The significance is conceptual: the paper argues that the historical accident of attention being invented first (and recurrence being seen as the "old" approach to be overcome) has misdirected architectural research. The "right" foundation for sequence modeling may be the recurrence, not attention, with attention emerging as a parallel implementation detail. This is a fundamental reframing, not an incremental improvement over linear attention, because it changes what is considered the primitive operation. If the recurrence is the primitive, then the research question shifts from "how do we approximate attention efficiently?" to "what properties should the recurrent state and transition have to enable powerful sequence modeling, and what parallel forms do those properties imply?"

The evidence supporting this reframing is the exact equivalence claim itself. If the parallel and recurrent forms produced different outputs, the inversion would be merely an interesting derivation but not a practical contribution. The paper's architecture works precisely because the equivalence is exact, and all experimental results (Figures 5 and 6, Tables 3 and 4) demonstrate that the same model weights produce the same outputs whether run in parallel or recurrent mode.

---

Innovation 3: The Chunkwise Recurrent Representation as a Third Computational Mode

At first glance, the chunkwise recurrent representation (Equation 7) appears to be an engineering optimization — a way to get linear memory complexity during training by hybridizing parallel and recurrent computation. But it represents a deeper conceptual contribution: the recognition that parallel and recurrent forms are endpoints of a continuous spectrum of computation-communication tradeoffs, and that the optimal point on this spectrum depends on hardware constraints and sequence length.

Prior work treated parallel and recurrent as binary: either you materialize the full attention matrix (Transformer) or you don't (RNN, linear attention at inference). The chunkwise form introduces a tunable parameter — chunk size $B$ — that controls how much computation is done in parallel (within chunks) versus sequentially (across chunks). At $B = |x|$, it reduces to pure parallel; at $B = 1$, it reduces to pure recurrent. At intermediate $B$, it achieves $O(|x|/B \times B^2 d) = O(|x| B d)$ computation and $O(B^2 + d^2)$ memory (the chunk attention matrix plus the recurrent state), which is linear in sequence length when $B$ is constant.

This is not just an implementation trick — it is a new architectural capability that neither Transformers nor RNNs possess. Transformers can be made more memory-efficient through gradient checkpointing or FlashAttention (which the paper compares against in Table 4), but the fundamental $O(N^2)$ attention matrix must be computed at some level. RNNs can be trained with truncated backpropagation through time, but the fundamental sequential dependency means GPU parallelism is limited. The chunkwise form occupies a regime neither can reach: long-sequence training that is both memory-efficient (linear in $N$) and GPU-parallel (within chunks).

The theoretical significance is that it completes the computational picture of the retention mechanism. The parallel form handles short sequences (where $N^2$ memory is affordable), the recurrent form handles inference (where per-step cost must be constant), and the chunkwise form handles long-sequence training (where both parallelism and memory matter). A mechanism with only the parallel and recurrent forms would have a gap: training on 100k-token sequences would require either $O(N^2)$ memory (parallel) or sequential processing (recurrent), both of which are prohibitive. The chunkwise form fills this gap, making retention applicable to the full range of sequence lengths that modern LLMs encounter.

The paper's choice of $B = 512$ for training on 8192-length sequences (Section 3.3) demonstrates the practical value. At $B = 512$, the chunk-local attention matrices are $512 \times 512$, which easily fit in GPU SRAM, while the cross-chunk recurrent state is a $256 \times 256$ matrix per head (the head dimension). This is vastly smaller than the $8192 \times 8192$ matrix a pure parallel form would require. Table 4 shows the consequence: RetNet trains with lower memory (34.5 GB for 1.3B model vs. 74.8 GB for vanilla Transformer) and higher throughput (73,345 wps vs. 10,832 wps) on 8192-length sequences.

The comparison with FlashAttention in Table 4 is particularly revealing. FlashAttention achieves memory efficiency by recomputing attention scores during the backward pass rather than storing them — it is an IO optimization, not an algorithmic one. It still computes the full $O(N^2)$ attention in the forward pass. RetNet achieves similar memory (34.5 GB vs. 38.8 GB for 1.3B) and throughput (73,345 wps vs. 63,965 wps) without specialized kernels, using vanilla PyTorch. The algorithmic structure — chunkwise decomposition of an inherently factorable computation — provides efficiency that FlashAttention achieves only through careful engineering. The paper notes this leaves room for further optimization: "RetNet has the potential to further reduce cost via advanced implementation, such as kernel fusion." If FlashAttention-like optimizations were applied to the chunkwise form, the efficiency advantage could grow further.

This is a fundamental algorithmic contribution that extends beyond the specific retention mechanism. Any sequence model whose computation can be factorized into local (within-chunk) and global (recurrent summary) components could adopt the chunkwise pattern. The paper's pseudocode (Figure 4) shows how cleanly the three modes share the same underlying operations — ParallelRetention, RecurrentRetention, and ChunkwiseRetention differ only in how they compose the same $q$, $k$, $v$, and decay tensors. This suggests the chunkwise pattern is a general design principle, not a RetNet-specific hack.

---

Innovation 4: Multi-Scale Decay as an Alternative to Learned Attention Patterns

The paper's use of fixed, multi-scale exponential decay rates ($\gamma = 1 - 2^{-5 - \text{arange}(0,h)}$) as the temporal weighting mechanism is a distinct conceptual departure from both softmax attention (where weights are purely content-dependent and unbounded in dynamic range) and fixed-position approaches (where weights depend only on relative distance but are uniform across all token pairs).

In standard attention, the weighting of past tokens is entirely learned and content-dependent: $w_{nm} = \text{softmax}(Q_n K_m^\top / \sqrt{d})$. This gives maximum flexibility — the model can learn to attend strongly to specific tokens regardless of their distance — but has two costs: it cannot be factorized for recurrence (because the softmax depends on all scores), and it can produce unstable attention patterns that generalize poorly to unseen sequence lengths (see the length-extrapolation literature that the paper's authors have contributed to, including xPos from Sun et al., 2022).

In fixed-decay approaches (like RWKV's exponential decay or the diagonal state transitions in S4), the weighting is purely positional and decays monotonically with distance. This enables recurrence but removes the content-dependence that makes attention powerful — the model cannot "look back" to a specific earlier token more strongly than its distance would suggest.

RetNet's multi-scale decay with content-aware queries and keys occupies a hybrid regime that neither pure attention nor pure decay models reach. The decay factor $\gamma^{n-m}$ provides a structured prior on temporal distance — recent tokens are weighted more heavily, with the decay rate controlling how quickly weight falls off. But the query-key dot product $Q_n K_m^\top$ modulates this prior based on content: a token at distance 100 with very high query-key similarity can still receive significant weight if the decay is slow enough, while an irrelevant token at distance 1 can be suppressed by a low query-key similarity. The decay sets the baseline temporal profile; the query-key interaction provides content-dependent deviations from that baseline.

The multi-scale aspect — different heads use different $\gamma$ values, ranging from fast decay (~0.97) to slow decay (~0.9998) — means different heads specialize in different temporal ranges. This is analogous to how multi-head attention can learn different attention patterns, but with an important difference: the patterns are structured by the decay rate rather than being purely emergent from learned weights. A head with $\gamma = 0.97$ effectively has a short temporal window (contributions from tokens more than ~50 steps back are negligible) regardless of what it learns; a head with $\gamma = 0.9998$ naturally retains information over thousands of steps. This structured diversity may be more robust than purely learned diversity, because the temporal specialization is guaranteed by the architecture rather than needing to be discovered through training.

The ablation evidence supports this interpretation. Table 6 shows:

• Removing multi-scale decay (all heads use $\gamma = 127/128 \approx 0.992$) increases perplexity from 26.05 to 27.02 — a meaningful but not catastrophic degradation, suggesting that uniform decay can partially compensate but loses the temporal specialization.
• Removing decay entirely ($\gamma = 1$, equivalent to uniform attention) increases perplexity to 27.86 — a larger degradation, confirming that the exponential decay prior is genuinely useful, not just a computational convenience.

The theoretical significance is that multi-scale fixed decay with content-aware modulation may represent a more principled way to handle the temporal structure of language than purely learned attention. Language has natural temporal locality — words in the same sentence are more relevant than words 100 sentences ago, on average — and embedding this locality as an architectural prior (through decay) rather than requiring the model to learn it from scratch may improve sample efficiency and generalization. The content-aware modulation then allows the model to override this prior when specific long-range dependencies exist (e.g., pronoun resolution across a long paragraph). This is a fundamental design principle, not an incremental trick: it argues that sequence models should have an explicit, structured notion of temporal distance that interacts with content-based routing, rather than treating distance as just another feature to be learned implicitly through position embeddings.

---

Innovation 5: Exact Dual-Form Equivalence as a Sufficient Condition for Breaking the Triangle

The paper's most theoretically significant claim — and the one that distinguishes it most sharply from all prior work — is that the "impossible triangle" can be broken only when the sequence modeling mechanism possesses an exact dual parallel-recurrent form, not when the two forms are approximations of each other. This is not stated as a theorem, but it emerges as the paper's central architectural thesis through the derivation structure and the contrast with prior approaches.

The thesis can be reconstructed as: the reason all prior architectures occupy an edge of the triangle is that they optimize one computational form (parallel for training, recurrent for inference) and accept approximation error when translating to the other form. Linear attention approximates softmax attention with kernels and accepts the performance loss as the cost of recurrent inference. RWKV uses a recurrent form for both training and inference, and accepts the loss of training parallelism. Transformer uses parallel attention for training and accepts the $O(N)$ inference cost because the softmax makes recurrence impossible. In every case, the "missing" property is sacrificed at the level of the mechanism's algebraic structure, not at the level of implementation.

RetNet's claim is that exact equivalence eliminates this sacrifice. Because Equations 5, 6, and 7 compute exactly the same function — not approximately, not up to kernel error, not up to truncation — switching between them at training time (parallel) and inference time (recurrent) introduces zero performance degradation. The model that trains in parallel is literally the same model that infers recurrently; there is no train-inference gap. This is what makes it possible to simultaneously achieve training parallelism and $O(1)$ inference without performance loss: there is no "tradeoff" because there is no approximation.

This claim elevates the retention derivation from a mathematical curiosity to a design principle for future architectures. The paper implicitly argues that architectural research should focus on finding sequence modeling operations that admit exact dual forms, rather than on making approximations tighter. The diagonalization step in the derivation — $A = \Lambda(\gamma e^{i\theta})\Lambda^{-1}$ — is not just a trick for this specific architecture; it demonstrates a general pattern: start with a linear recurrent operator, choose a diagonalizable transition matrix, and unroll to obtain the parallel form. Any linear recurrence with a diagonalizable transition matrix will have an exact parallel representation. The content-awareness and multi-scale aspects are then layered on top of this core algebraic structure.

The evidence for this thesis is the paper's demonstration that all three properties hold simultaneously. Table 4 shows training parallelism comparable to or better than FlashAttention. Figure 6 shows $O(1)$-like inference behavior (flat memory, length-invariant throughput, latency independent of batch size). Figure 5 and Table 3 show scaling curves and downstream performance comparable to Transformer. None of these results individually is revolutionary — but all three together, for the same architecture with the same weights, is unprecedented. Prior architectures that achieved two of the three (e.g., S4 achieving parallelism and $O(1)$ inference) always showed a performance gap relative to Transformer. RetNet closes this gap, and the exact dual-form equivalence is the proposed explanation for why.

This thesis, if validated by future work at larger scales, would fundamentally reshape architectural research away from the "start from attention and approximate" paradigm that has dominated since Katharopoulos et al. (2020) and toward a "start from recurrence and derive upward" paradigm. It suggests that the Transformer was a historical accident — we discovered the parallel form first (attention) and only later recognized it could be derived from a recurrence — rather than a fundamental primitive. The true primitive may be the recurrence, with attention as its parallel shadow.

Caveat: The paper does not prove that exact dual-form equivalence is necessary to break the triangle — only that it is sufficient and that it works for RetNet. It is possible that a different mechanism could achieve all three properties without exact equivalence (e.g., through a form of structured sparsity or learned truncation). The claim should be understood as a strong architectural hypothesis supported by one architecture's success, not a mathematical theorem. The scaling results (Figure 5) show RetNet matching Transformer up to 6.7B parameters, but the scaling trend at larger sizes (10B, 100B, 500B) is unknown. If RetNet's scaling falters relative to Transformer at these scales, the "exact dual-form equivalence is sufficient" claim would remain technically true (it did achieve all three at the tested scales) but its practical significance would be limited.

Nonetheless, as a diagnostic principle — "if you want to break the triangle, find an operation with an exact dual parallel-recurrent form" — this insight provides the clearest actionable guidance the paper offers to architecture designers. It explains why prior attempts failed (they approximated) and what RetNet does differently (it doesn't), and it sets a concrete mathematical criterion for evaluating future proposals (ask: is the parallel form exactly equivalent to the recurrent form, or is there approximation error?). This is a fundamental reframing of the architecture design problem, not an incremental contribution.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary training corpus is a curated compilation of The Pile (Gao et al., 2020), C4 (Dodge et al., 2021), and The Stack (Kocetkov et al., 2022). Language modeling perplexity is reported on an in-domain validation set drawn from this corpus. Out-of-domain evaluation uses Project Gutenberg 2019–2022 (PG22; Sun et al., 2022), QMSum (Zhong et al., 2021), GovReport (Huang et al., 2021), and SummScreen (Chen et al., 2021; Shaham et al., 2022). Downstream task evaluation uses HellaSwag (Zellers et al., 2019), BoolQ (Clark et al., 2019), COPA (Wang et al., 2019), PIQA (Bisk et al., 2020), Winograd, Winogrande (Levesque et al., 2012), and StoryCloze (Mostafazadeh et al., 2017).

• Base model(s). RetNet models are trained from scratch at three scales: 1.3B, 2.7B, and 6.7B parameters (detailed configurations in Appendix A, Table 7). The baseline Transformer uses identical layer counts, hidden dimensions, and total parameters per scale — with parameter allocation adjusted so that both architectures have matched total parameter counts (Transformer self-attention uses 4d² parameters while RetNet MSR uses 8d² parameters, compensated by reducing RetNet's FFN intermediate dimension from 4d to 2d). For the efficient architecture comparison (Section 3.5), all models use 200M parameters with 16 layers and hidden dimension 1024. The choice of PaLM-style models (via DeepNet initialization and TorchScale) reflects the paper's focus on large-scale training stability.

• Metrics. The primary metric is language modeling perplexity on validation sets — lower is better. For downstream tasks, accuracy (%) is reported under zero-shot and 4-shot settings. For training efficiency: GPU memory consumption (GB) and training throughput (words per second, wps) measured on 8× NVIDIA A100-80GB GPUs with sequence length 8192. For inference efficiency: GPU memory (GB) during decoding, decoding throughput (wps), and decoding latency (ms) measured on a single A100-80GB GPU with the 6.7B model at various sequence lengths (2048–8192) and batch sizes (1–8). The paper does not report confidence intervals or statistical significance tests on perplexity differences.

• Baselines. Three tiers of comparison are used. (1) Standard Transformer (Vaswani et al., 2017) with identical model size and training data — this is the primary performance baseline. (2) Transformer with FlashAttention (Dao et al., 2022) — included specifically in training cost comparisons (Table 4) to benchmark against a highly optimized Transformer implementation, since FlashAttention uses kernel fusion and recomputation to reduce memory and improve throughput. (3) Efficient architecture variants at 200M scale: Linear Transformer (Katharopoulos et al., 2020), RWKV (Peng et al., 2023), H3 (Dao et al., 2022), and Hyena (Poli et al., 2023) — compared in Section 3.5 for language modeling perplexity on in-domain and out-of-domain corpora. For H3, the head dimension is set to 8; for RWKV, the TimeMix module replaces self-attention layers while FFN layers are kept consistent with other models for fair comparison.

• Generation budget / compute accounting. Training cost is measured by total GPU memory consumption and training throughput at fixed sequence length (8192) and batch configuration, with the same hardware (8× A100-80GB). For the 6.7B and 13B models, tensor parallelism is enabled. For RetNet, the chunkwise recurrent representation (Equation 7) is used with chunk size B = 512 during training; the paper notes this is implemented in vanilla PyTorch without kernel fusion, making the comparison to FlashAttention (which uses heavily optimized CUDA kernels) conservative. Inference cost is measured holding model size fixed (6.7B) and varying sequence length and batch size, with Transformer using KV-cache reuse and RetNet using the recurrent representation (Equation 6). No total-FLOPs counting is used — comparisons rely on wall-clock throughput and memory measurements, which incorporate real hardware constraints.

• Cross-validation / statistical protocol. None reported. The 500-question MATH test set split is not used (this is a language modeling paper, not MATH). Perplexity results are reported as single numbers without variance estimates across training runs. The authors train each model configuration once from scratch at each scale (100B tokens, 25k steps). The 200M-scale comparison (Table 5) uses a single training run per architecture. There is no mention of multiple seeds, bootstrap confidence intervals, or statistical testing for perplexity differences. This is standard practice for language modeling architecture papers at this scale (training multiple 6.7B models from scratch would be prohibitively expensive), but it means small perplexity differences (~0.5 PPL) should be interpreted cautiously.

---

Main Quantitative Results

Language Modeling Scaling Curves (Transformer vs. RetNet)

The headline result for modeling quality is Figure 5: RetNet achieves comparable or better validation perplexity compared to Transformer across three model sizes (1.3B, 2.7B, 6.7B) trained on 100B tokens. At 1.3B, the curves are nearly overlapping. At 2.7B and 6.7B, RetNet pulls ahead slightly. The paper states:

"We empirically observe that RetNet tends to outperform Transformer when the model size is larger than 2B."

Specific perplexity values are not reported in the main text for Figure 5 (the y-axis uses a narrow range, approximately 13.0–15.0, making precise reading difficult), but the scaling trend is visually clear: RetNet's validation perplexity decreases with model size at a rate comparable to or slightly better than Transformer. Appendix B (Table 8) provides grouped results by context length for the 6.7B model: at context length 512, RetNet achieves 13.09 vs. Transformer's 13.55; at 1024, 12.14 vs. 12.56; at 2048, 11.98 vs. 12.35. RetNet holds a consistent 0.37–0.46 PPL advantage across all context lengths, and importantly, RetNet's perplexity improves monotonically with longer context (13.09 → 12.14 → 11.98), demonstrating that the model can effectively use additional context.

Why this matters: The scaling curves are the critical evidence against the "impossible triangle" claim that architectures must sacrifice performance for efficiency. If RetNet had shown worse scaling (e.g., converging to a higher perplexity floor, or improving more slowly with model size), it would confirm that the dual-form mechanism trades modeling capacity for efficiency — like linear attention. The fact that RetNet matches or exceeds Transformer scaling at these sizes supports the paper's core thesis that exact dual-form equivalence does not inherently constrain model quality.

Caveats: The models are trained on only 100B tokens, which is far below the compute-optimal token count for these model sizes. At 6.7B parameters and 100B tokens, the models are significantly undertrained by Chinchilla standards (~130B tokens would be compute-optimal for a 6.7B model, and production models at this scale are typically trained on 1T+ tokens). The observed scaling behavior may change at higher token budgets — architectures can exhibit different scaling exponents in the undertrained vs. compute-optimal regimes. Additionally, the largest model tested (6.7B) is modest by contemporary standards; GPT-3-scale experiments (175B) or beyond would provide stronger evidence that the scaling trend holds.

---

Downstream Task Performance (Zero-Shot and Few-Shot)

Table 3 reports accuracy on seven downstream benchmarks for the 6.7B models under zero-shot and 4-shot evaluation. RetNet outperforms Transformer on aggregate by 3.44 percentage points in zero-shot (69.51 vs. 66.07) and by 3.32 points in 4-shot (69.76 vs. 66.44).

Breaking down by task:

• HellaSwag (zero-shot): RetNet 60.7 vs. Transformer 55.9 (+4.8). HellaSwag (4-shot): RetNet 60.5 vs. Transformer 55.8 (+4.7).
• BoolQ (zero-shot): RetNet 62.2 vs. Transformer 62.0 (+0.2) — essentially tied. BoolQ (4-shot): RetNet 60.1 vs. Transformer 58.7 (+1.4).
• COPA (zero-shot): RetNet 77.0 vs. Transformer 69.0 (+8.0) — largest gap. COPA (4-shot): RetNet 78.0 vs. Transformer 71.0 (+7.0).
• PIQA (zero-shot): RetNet 75.4 vs. Transformer 74.6 (+0.8). PIQA (4-shot): RetNet 76.0 vs. Transformer 75.0 (+1.0).
• Winograd (zero-shot): RetNet 77.2 vs. Transformer 69.5 (+7.7). Winograd (4-shot): RetNet 77.9 vs. Transformer 71.9 (+6.0).
• Winogrande (zero-shot): RetNet 58.1 vs. Transformer 56.5 (+1.6). Winogrande (4-shot): RetNet 59.9 vs. Transformer 57.3 (+2.6).
• StoryCloze (zero-shot): RetNet 76.0 vs. Transformer 75.0 (+1.0). StoryCloze (4-shot): RetNet 75.9 vs. Transformer 75.4 (+0.5) — smallest gap.

The pattern is consistent: RetNet matches or exceeds Transformer on every task in every setting. The largest advantages appear on COPA (+7–8 points) and Winograd (+6–8 points), while BoolQ, PIQA, and StoryCloze show modest differences (0.2–1.4 points). Notably, the 4-shot setting does not uniformly improve performance for either model — Transformer's zero-shot average (66.07) is essentially the same as its 4-shot average (66.44), and RetNet's zero-shot (69.51) and 4-shot (69.76) are nearly identical, suggesting that at 6.7B parameters with 100B training tokens, these models have limited in-context learning capability regardless of architecture.

Why this matters: The downstream results corroborate the language modeling perplexity findings (Figure 5) with task-specific evidence. The concern with purely perplexity-based evaluation is that it might favor architectures that are good at next-token prediction but fail at more structured reasoning. The consistent 3+ point aggregate advantage on diverse reasoning tasks (commonsense, coreference, narrative understanding) strengthens the claim that RetNet's modeling quality is genuinely competitive, not just an artifact of the perplexity metric.

Caveats: The evaluation is limited to relatively small benchmarks (the largest, HellaSwag, is ~10k examples) and does not include more challenging tasks like MMLU, GSM8K, or HumanEval that would stress the model's reasoning capabilities. Additionally, the paper does not report results for the 1.3B and 2.7B models on downstream tasks — only the 6.7B model is evaluated. It is possible that the advantage is size-dependent (the paper itself notes that RetNet only begins outperforming Transformer above 2B parameters), and smaller RetNet models might underperform on downstream tasks even if the 6.7B model excels.

---

Training Cost Comparison (Memory and Throughput)

Table 4 reports training memory and throughput for Transformer, Transformer with FlashAttention, and RetNet at four model sizes (1.3B, 2.7B, 6.7B, 13B) with sequence length 8192. The headline: RetNet achieves lower or comparable memory and higher or comparable throughput across all sizes, even against the highly optimized FlashAttention baseline.

Memory (lower is better):

• 1.3B: RetNet 34.5 GB vs. Transformer 74.8 GB vs. Transformer+FlashAttn 38.8 GB. RetNet uses 53.9% less memory than vanilla Transformer and is 11% lower than FlashAttention.
• 2.7B: RetNet 42.0 GB vs. Transformer 69.6 GB vs. FlashAttn 42.1 GB. RetNet is essentially tied with FlashAttention (42.0 vs. 42.1), both using ~40% less than vanilla Transformer.
• 6.7B: RetNet 48.0 GB vs. Transformer 69.0 GB vs. FlashAttn 51.4 GB. RetNet uses 30.4% less than vanilla Transformer and is 6.6% lower than FlashAttention.
• 13B: RetNet 45.9 GB vs. Transformer 61.4 GB vs. FlashAttn 46.3 GB. RetNet uses 25.2% less than vanilla Transformer and is marginally lower than FlashAttention (45.9 vs. 46.3).

Throughput in words per second, wps (higher is better):

• 1.3B: RetNet 73,345 wps vs. Transformer 10,832 wps vs. FlashAttn 63,965 wps. RetNet is 6.8× faster than vanilla Transformer and 14.7% faster than FlashAttention.
• 2.7B: RetNet 38,921 wps vs. Transformer 5,186 wps vs. FlashAttn 34,990 wps. RetNet is 7.5× faster than vanilla Transformer and 11.2% faster than FlashAttention.
• 6.7B: RetNet 17,459 wps vs. Transformer 2,754 wps vs. FlashAttn 16,230 wps. RetNet is 6.3× faster than vanilla Transformer and 7.6% faster than FlashAttention.
• 13B: RetNet 8,642 wps vs. Transformer 1,209 wps vs. FlashAttn 7,945 wps. RetNet is 7.1× faster than vanilla Transformer and 8.8% faster than FlashAttention.

Why this matters: These results demonstrate that RetNet's training efficiency advantage is not merely theoretical — it translates to real wall-clock speedups on standard hardware. The comparison against FlashAttention is particularly important because FlashAttention represents the state of the art in efficient Transformer training. RetNet's ability to match or exceed FlashAttention using vanilla PyTorch (no custom CUDA kernels) suggests that the efficiency comes from algorithmic structure (the chunkwise O(NBd) complexity) rather than engineering optimization. The paper notes:

"without relying on specific kernels, it is easy to train RetNet on other platforms efficiently. For example, we train the RetNet models on an AMD MI200 cluster with decent throughput."

This platform portability is a practical advantage — FlashAttention's optimizations are specific to NVIDIA GPUs and require significant engineering effort to port.

The 13B result requires scrutiny: At 13B, RetNet's memory (45.9 GB) is lower than at 6.7B (48.0 GB). This is unusual and is likely an artifact of tensor parallelism configuration (which changes the per-GPU memory distribution) rather than an intrinsic property of the architecture. The paper enables tensor parallelism for 6.7B and 13B models, and the memory numbers reflect per-GPU consumption in the distributed setting — they are not directly comparable across sizes where parallelism strategies differ. The throughput numbers, however, are directly meaningful because they measure tokens processed per second across the entire system.

---

Inference Cost Comparison (Memory, Throughput, Latency)

Figure 6 presents the most practically impactful results: inference cost for the 6.7B model on a single A100-80GB GPU across varying sequence lengths and batch sizes. The headline numbers from Figure 1 (for 8k sequence length): 8.4× faster decoding, 70% less GPU memory, and 3.4× higher throughput compared to Transformer with KV cache.

GPU Memory (Figure 6a): Transformer memory grows approximately linearly from ~15 GB at sequence length 2048 to ~42 GB at 8192 — the KV cache dominates. RetNet memory stays flat at ~15 GB regardless of sequence length, with model weights occupying 97% of the total and the recurrent state contributing only ~3%. At 8192, RetNet uses approximately 15 GB vs. 42 GB for Transformer — roughly a 64% reduction (the paper claims 70% in the abstract, which is close). The key insight is that RetNet's memory footprint is determined by model size, not context length, making it predictable and bounded.

Throughput (Figure 6b): Transformer throughput drops sharply with sequence length: from roughly 280 wps at 2048 to roughly 80 wps at 8192 — a 3.5× degradation because each decoding step must attend over the growing KV cache. RetNet throughput is approximately 250–300 wps and is length-invariant — the recurrent computation per step is $O(d^2)$, independent of context length. At 8192, RetNet achieves roughly 270 wps vs. Transformer's 80 wps: a 3.4× advantage (matching the Figure 1 claim).

Latency (Figure 6c): This panel reveals a critical deployment consideration. Transformer latency increases with both sequence length and batch size — processing a batch of 8 requests at 8192 context takes ~350 ms per token. RetNet latency at 8192 context is approximately 50–60 ms and nearly flat across batch sizes 1–8. The paper emphasizes:

"In order to make latency acceptable, we have to restrict the batch size, which harms the overall inference throughput of Transformers. By contrast, RetNet's decoding latency outperforms Transformers and keeps almost the same across different batch sizes and input lengths."

This is significant for serving infrastructure: Transformer deployments often run at low batch sizes (1–4) to keep latency acceptable for interactive applications, which underutilizes GPU compute. RetNet can run at higher batch sizes without latency penalty, achieving higher overall throughput in production serving scenarios.

Why this matters: These results directly validate the "low-cost inference" axis of the impossible triangle. The memory and throughput advantages are not marginal — they are multi-fold at realistic deployment sequence lengths (4k–8k tokens). The latency insensitivity to batch size is particularly important because it changes the economics of LLM serving: you can batch more requests without user-perceived slowdown, increasing hardware utilization and reducing cost per query.

Caveats: The inference measurements use the 6.7B model, which has modest weight memory requirements (~13.4 GB in FP16). For larger models (70B, 175B), the weight memory dominates, and the relative advantage of eliminating the KV cache shrinks. If weights are 140 GB and KV cache is 10 GB, eliminating the cache saves only ~7% total memory — still beneficial but not transformative. The paper does not report inference results for the 1.3B or 2.7B models, nor for larger models. The inference measurements also appear to be for a single sequence (batch size 1 for memory and throughput; Figure 6c varies batch size only for latency). Multi-user serving throughput in a continuous batching setup may differ from the single-sequence measurements reported.

---

Comparison with Efficient Architecture Variants (200M Scale)

Table 5 reports perplexity on in-domain and four out-of-domain corpora for 200M-parameter models trained on 10k steps with 0.5M token batch size. RetNet achieves the best (lowest) perplexity on every corpus.

In-domain results:

• RetNet: 26.05
• H3: 29.97
• RWKV: 30.92
• Hyena: 32.08
• Linear Transformer: 40.24

The gap between RetNet and the next-best architecture (H3) is 3.92 PPL — a substantial margin at this scale. Linear Transformer is dramatically worse (40.24 vs. 26.05), confirming the paper's claim that kernel-based attention approximations sacrifice significant modeling capacity.

Out-of-domain generalization:

• PG22: RetNet 45.27 vs. H3 49.17 vs. RWKV 51.41 vs. Hyena 52.75 vs. Linear Transformer 63.86.
• QMSum: RetNet 21.33 vs. H3 24.29 vs. RWKV 28.17 vs. Hyena 28.18 vs. Linear Transformer 28.45.
• GovReport: RetNet 16.52 vs. H3 19.19 vs. RWKV 19.80 vs. Hyena 20.55 vs. Linear Transformer 25.33.
• SummScreen: RetNet 22.48 vs. H3 25.11 vs. RWKV 25.78 vs. Hyena 26.51 vs. Linear Transformer 32.02.

The ranking is strikingly consistent: RetNet > H3 > RWKV ~ Hyena ≫ Linear Transformer on every corpus. The out-of-domain results are important because they test whether RetNet's inductive biases (exponential decay with content-aware modulation) generalize to different text distributions, or whether the architecture overfits to characteristics of the training corpus. The consistent advantage across diverse domains — books (PG22), meeting transcripts (QMSum), government reports (GovReport), and screenplays (SummScreen) — suggests that the architecture captures general properties of language rather than corpus-specific artifacts.

Why this matters: This comparison directly addresses the "performance" axis of the impossible triangle. Prior architectures that achieved training parallelism and low-cost inference (H3, Hyena) sacrificed performance relative to Transformer. Table 5 shows that at 200M scale, RetNet does not make this sacrifice — it outperforms all efficient alternatives by meaningful margins. This is the evidence that breaking the impossible triangle requires exact dual-form equivalence (RetNet) rather than structural approximation (H3/Hyena) or sequential training (RWKV).

Training and inference complexity of compared methods: The paper includes a brief complexity analysis in Section 3.5. For training:

• RWKV: $O(dn)$ token-mixing complexity via element-wise operators — reduces FLOPs but limits capacity.
• Hyena: $O(dn \log n)$ via Fast Fourier Transform — efficient but still superlinear.
• RetNet: $O(dn(b+h))$ where $b=512$ (chunk size), $h=256$ (head dimension) — linear in $n$ with a constant factor.

For inference:

• Hyena has $O(n)$ per-step complexity (like Transformer) — it does not achieve $O(1)$ inference.
• RWKV, H3, and RetNet all achieve $O(1)$ per-step decoding.
• Linear Transformer achieves $O(1)$ but with severely degraded performance.

Caveats: The 200M-scale comparison uses a single training run per architecture with 10k steps (5B tokens), which is minimal training. Relative performance at this scale may not predict relative performance at the 1B–10B scale, as different architectures may have different scaling exponents. Additionally, the hyperparameters for compared architectures may not be optimal — the paper adjusts RWKV and H3 configurations for fair comparison (e.g., keeping FFN layers consistent, setting H3 head dimension to 8), but there may be undiscovered configurations where these architectures perform better. The Linear Transformer result (40.24 PPL) is so much worse than other methods that it raises questions about whether the implementation or hyperparameters were correctly tuned — the paper provides no details on Linear Transformer configuration.

---

Context Length Scaling (Appendix B)

Table 8 reports perplexity for the 6.7B RetNet and Transformer models evaluated with different context lengths (512, 1024, 2048), using 2048-token chunks and computing perplexity only on the last 128 tokens of each chunk. Key finding: RetNet benefits more from increased context length than Transformer, and holds a consistent advantage at each length.

• At context 512: RetNet 13.09 vs. Transformer 13.55 (gap: 0.46)
• At context 1024: RetNet 12.14 vs. Transformer 12.56 (gap: 0.42)
• At context 2048: RetNet 11.98 vs. Transformer 12.35 (gap: 0.37)

RetNet's perplexity improves by 1.11 points going from 512 to 2048 context (13.09 → 11.98), while Transformer improves by 1.20 points (13.55 → 12.35) — comparable absolute gains. The gap narrows slightly with longer context (0.46 → 0.37) but remains meaningful. Notably, RetNet at context 1024 (12.14) outperforms Transformer at context 2048 (12.35), suggesting that RetNet extracts more information per token of context.

Why this matters: This experiment addresses a potential concern about the exponential decay mechanism: does the fixed $\gamma$ decay cause the model to "forget" information beyond a certain distance, limiting its ability to use long contexts? The monotonic improvement with context length (both models improve) and RetNet's consistent advantage suggest that the multi-scale decay rates are sufficiently slow (the slowest heads have $\gamma \approx 0.9998$, retaining information over thousands of steps) to capture long-range dependencies in practice. The experiment also validates that the chunkwise training procedure (B=512) does not create a train-test mismatch when evaluating with full 2048-length attention.

Caveats: The maximum context length tested is 2048 tokens, which is modest by contemporary standards (many production models use 8k–32k context windows). The paper does not evaluate whether RetNet's advantage persists or degrades at very long contexts (8k, 16k, 32k) where the exponential decay might become more limiting compared to Transformer's exact attention. The chunkwise training with B=512 means the model never processes a full 2048-token attention matrix during training; it only sees chunk-local attention (512) plus recurrently summarized cross-chunk information. The inference-time evaluation at 2048 uses the parallel form (or recurrent form) without chunking, so there is a training-inference gap in how long-range dependencies are computed. The fact that this gap does not harm perplexity is reassuring but may become more significant at longer contexts where the recurrent summary loses fidelity.

---

Ablation Studies and Robustness Checks

All ablations use the 200M-scale configuration (16 layers, hidden 1024) and are evaluated on the same in-domain and out-of-domain corpora as Table 5. Results are reported in Table 6.

• Swish gate removal (−swish gate): Removing the gating mechanism (Equation 8, the Swish-activated element-wise multiplication) increases in-domain perplexity from 26.05 to 27.84 (+1.79) and degrades performance on all out-of-domain corpora (e.g., PG22: 45.27 → 49.44, GovReport: 16.52 → 17.45). This is the largest single-component degradation in the ablation suite, confirming that the gate provides essential nonlinearity. Without softmax in the retention mechanism, the gating is the primary source of nonlinearity in the MSR module — its removal leaves the module as a largely linear operation on values, substantially reducing modeling capacity.

• GroupNorm removal (−GroupNorm): Replacing GroupNorm with standard LayerNorm (or removing head-wise normalization) increases in-domain perplexity from 26.05 to 27.54 (+1.49). Out-of-domain degradation is also substantial (PG22: 45.27 → 46.95, GovReport: 16.52 → 17.59). This supports the paper's argument that separate normalization per head is important because different decay rates produce different variance statistics. The degradation is slightly smaller than gate removal, suggesting that while normalization helps, the gate is more critical.

• Gamma decay removal (−γ decay, i.e., γ = 1 for all heads): Setting all decay rates to 1.0 (no decay, equivalent to uniform attention over all past tokens) increases in-domain perplexity from 26.05 to 27.86 (+1.81). This is the second-largest degradation, nearly matching gate removal. Out-of-domain: PG22 45.27 → 47.85, GovReport 16.52 → 17.49. The result strongly supports the paper's claim that exponential decay is not just a computational convenience for enabling recurrence — it is a useful inductive bias for language modeling. Uniform attention (γ = 1) gives equal weight to all past tokens regardless of distance, which is almost certainly suboptimal for natural language where local context is more relevant than distant context on average.

• Multi-scale decay removal (−multi-scale decay, all heads use γ = 127/128): Using a single decay rate across all heads (γ = 127/128 ≈ 0.992) increases in-domain perplexity from 26.05 to 27.02 (+0.97). This is a smaller degradation than removing decay entirely (+1.81) or removing the gate (+1.79), suggesting that having some decay is more important than having multiple decay rates. However, the multi-scale version still outperforms the single-rate version by a meaningful margin, confirming that different temporal scales across heads are beneficial. Out-of-domain results show the same pattern (PG22: 45.27 → 47.18, GovReport: 16.52 → 17.17).

• Head dimension reduction (from 256 to 64): Reducing the per-head dimension from 256 to 64 (for Q and K; V reduces from 512 to 128) while increasing the number of heads to keep total parameters constant increases in-domain perplexity from 26.05 to 27.68 (+1.63). Out-of-domain: PG22 45.27 → 47.72, GovReport 16.52 → 17.46, QMSum 21.33 → 23.09. The substantial degradation supports the paper's argument that larger head dimension increases the memory capacity of the recurrent state $S_n \in \mathbb{R}^{d \times d}$. A 256×256 state matrix can store far more information than a 64×64 matrix, and this capacity matters for the recurrent form where the state is the only mechanism for retaining past information.

Non-obvious patterns in the ablation results:

The relative magnitudes of degradation tell an interesting story about which components matter most:

1. Gate removal (+1.79) and decay removal (+1.81) are the largest — these are the essential nonlinearity and temporal structure, respectively.
2. Head dimension reduction (+1.63) is next — state capacity matters substantially.
3. GroupNorm removal (+1.49) — normalization discipline is important but less critical than the core computational structure.
4. Multi-scale removal (+0.97) — having multiple temporal scales provides a modest but consistent benefit.

This ordering suggests an architecture design principle: structure before normalization, nonlinearity before scale diversity. The exponential decay structure and nonlinear gating are fundamental to the model's operation; removing either causes severe degradation. The multi-scale aspect and normalization are refinements that improve performance but are not essential for basic functioning.

The out-of-domain generalization pattern is also noteworthy. Every ablation degrades performance on every out-of-domain corpus — there is no case where removing a component improves in-domain perplexity at the expense of out-of-domain generalization, or vice versa. This suggests that the ablated components (gate, GroupNorm, multi-scale decay, large head dimension) improve the model's general language modeling capacity rather than overfitting to training corpus characteristics. The consistent out-of-domain degradation across diverse text types (books, meetings, government reports, screenplays) strengthens confidence that these are genuine architectural improvements, not training-set-specific optimizations.

What is NOT ablated:

• Chunk size: The paper uses B=512 for all experiments but never ablates this choice. The chunkwise representation's efficiency depends on B, and it is unclear whether perplexity is sensitive to this hyperparameter.
• Number of heads: The ablation changes head dimension but adjusts head count to keep parameters constant — there is no exploration of the optimal head count at fixed head dimension.
• Gamma initialization: Different gamma initialization schemes (e.g., linear spacing vs. exponential spacing, different ranges) are not compared.
• Value dimension doubling: The widened V dimension (2d) is a specific design choice motivated by parameter allocation matching — ablating whether this doubling is actually beneficial (vs. using d for V and keeping the extra parameters elsewhere) would clarify its importance.
• Learning gamma: The paper fixes gamma as constant and unlearned — ablating learned gamma would test whether the fixed exponential structure is genuinely superior to learned temporal weighting, or merely simpler to implement.

---

Critical Assessment

The experimental results in this paper demonstrate a coherent and largely convincing picture: RetNet matches or exceeds Transformer performance across language modeling perplexity and downstream task accuracy at scales up to 6.7B parameters, while providing substantial inference cost reductions (8.4× faster decoding, 70% less memory at 8k context) and competitive or superior training efficiency. The experiments directly test all three axes of the "impossible triangle" and provide evidence that RetNet simultaneously achieves training parallelism, low-cost inference, and competitive performance at the tested scales. However, a careful reading reveals several limitations in what the experiments actually demonstrate relative to the paper's strongest claims.

The performance parity claim holds at tested scales but scaling behavior beyond 6.7B is unknown

The paper's central empirical claim is that RetNet achieves "Transformer-comparable performance" (Section 1) while providing inference benefits. Figure 5 supports this at 1.3B, 2.7B, and 6.7B parameters, with RetNet even showing a slight advantage at the two larger scales. However, the phrase "favourable scaling results" in the abstract implies that the scaling trajectory favors RetNet — that as models get larger, RetNet will continue to match or exceed Transformer. The evidence for this is limited to three data points, all in the undertrained regime (100B tokens for models that would typically be trained on 300B–1T tokens at these sizes). It is entirely possible that at larger scales or longer training, the architectures diverge — RetNet could plateau earlier (due to state capacity limits in the recurrent form) or Transformer could pull ahead (due to the representational flexibility of softmax attention). The paper's own observation that RetNet "tends to outperform Transformer when the model size is larger than 2B" is based on only two data points above that threshold (2.7B and 6.7B). Extrapolating a scaling trend from two points is speculative. The paper acknowledges this implicitly by noting future work will "scale up RetNet in terms of model size and training steps," but the current experiments do not establish scaling parity — they establish that RetNet is not catastrophically worse at these scales, which is valuable but a weaker claim.

The inference cost advantage is measured at a single model size and on a single GPU architecture

The dramatic inference cost reductions (Figure 6: 8.4× faster, 70% less memory) are all measured on the 6.7B model running on a single A100-80GB GPU. For this configuration, model weights are ~13.4 GB (in FP16) and the KV cache at 8k context is roughly 32 GB — the KV cache dominates, making its elimination transformative. For a 70B model, weights alone would be ~140 GB, requiring multiple GPUs, and the KV cache would be proportionally larger. If the weight memory dominates (140 GB weights vs. 40 GB KV cache), eliminating the cache saves ~22% total memory, not 70%. The 8.4× speedup similarly depends on the KV cache being the computational bottleneck — if model computation dominates (as it does for very large models where matrix multiplies in FFN layers overshadow attention), the recurrent form's advantage shrinks. These numbers are impressive for the tested configuration but should not be extrapolated to "RetNet is always 8.4× faster" without model-size-specific analysis that the paper does not provide.

Additionally, the inference measurements appear to use a batch size of 1 for the memory and throughput panels (6a, 6b), with the latency panel (6c) varying batch size. In production serving, continuous batching and prefilling of prompts are the standard optimization techniques. The paper does not evaluate prefilling latency (processing the input prompt, which in RetNet can use the parallel or chunkwise form) vs. decoding latency (generating tokens one at a time, where the recurrent form is used). This distinction matters because user-perceived latency is dominated by the slower of the two phases, and RetNet's prefilling behavior (which involves computing over the full input sequence) may have different characteristics than its decoding behavior.

Training efficiency is compared against FlashAttention but the implementation quality gap cuts both ways

Table 4 shows RetNet (vanilla PyTorch) outperforming Transformer with FlashAttention (heavily optimized CUDA kernels) in both memory and throughput. The paper frames this as evidence that RetNet's algorithmic efficiency is so strong that it beats engineering optimization. However, this comparison is asymmetric: RetNet benefits from its algorithmic structure (linear complexity in sequence length), but FlashAttention is optimizing a fundamentally quadratic operation. If FlashAttention-like kernel fusion were applied to RetNet's chunkwise recurrent computation, the RetNet advantage might grow further — but it also might not, if the chunkwise form's memory access patterns are less amenable to fusion than standard attention. The paper claims this potential ("RetNet has the potential to further reduce cost via advanced implementation, such as kernel fusion") but without demonstrating it, the comparison to FlashAttention is partly a comparison of algorithmic efficiency and partly a comparison of implementation maturity. The takeaway that "RetNet is efficient enough that even unoptimized code beats highly optimized Transformers" is correct, but the quantitative gap (e.g., 7.6% faster at 6.7B) would change under different implementation tradeoffs.

The 200M-scale comparison lacks training parity and hyperparameter optimization

Table 5 compares RetNet against H3, RWKV, Hyena, and Linear Transformer at 200M parameters. The training protocol (10k steps, 0.5M token batch size = 5B tokens total) is extremely short — these models are severely undertrained. At 200M parameters and 5B tokens, the token-to-parameter ratio is only 25:1, far below typical practice. Different architectures may have different learning dynamics: some may converge faster early in training but plateau sooner, while others may be slower to start but benefit more from extended training. The 10k-step comparison captures only the early training dynamics, which may favor certain architectures. The paper provides no learning curves (perplexity vs. training step) for these architectures, making it impossible to assess whether the ranking would change with more training.

Additionally, the hyperparameters for compared architectures are partially harmonized (FFN layers kept consistent, H3 head dimension set to 8) but there is no systematic hyperparameter sweep reported for any of the baseline architectures. RWKV's TimeMix module has its own hyperparameters (e.g., decay rates, initialization schemes) that affect performance; H3's state-space model has dimensionality and initialization choices. It is possible that with architecture-specific tuning, the baselines would perform better. The Linear Transformer's abysmal performance (40.24 PPL vs. RetNet's 26.05, a 54% increase) is so extreme that it suggests either a fundamental failure of the architecture at language modeling, or a mistuned configuration. If it is the former, the paper could provide analysis of why it fails so badly (beyond "struggles to encode position information"); if the latter, the comparison is unfair.

Claims about the "impossible triangle" implicitly assume the triangle is a binary property, but performance is continuous

The paper's framing of the "impossible triangle" (Figure 2, Table 1) uses checkmarks (✔) and crosses (✘) to indicate whether each architecture achieves training parallelism, low-cost inference, and good performance. This binary representation obscures important continuous variation. H3 gets a ✔ for "Performance" in Table 1, yet Table 5 shows H3 (29.97 PPL) significantly underperforms RetNet (26.05) and the paper argues that H3's content-unaware formulation limits its modeling capacity. What threshold of performance constitutes "good"? If it is "matches Transformer," then H3's ✔ is misleading — the paper provides no Transformer baseline at 200M scale to compare against. If it is "better than random chance," the bar is too low to be meaningful. The binary framing makes the "triangle" diagnosis cleaner but papers over the reality that RetNet itself may not perfectly match Transformer at all scales, meaning the triangle is "broken" in degree, not in kind. A more precise claim would be: "RetNet substantially reduces the performance-efficiency tradeoff compared to prior architectures, bringing all three properties into a practically useful regime simultaneously, though with residual uncertainty about scaling to very large models."

Missing experiments that would strengthen the paper considerably

Several experiments are conspicuous by their absence:

1. Long-context language modeling beyond 2048 tokens. The evaluation in Appendix B stops at 2048 context, yet the inference experiments (Figure 6) use up to 8192 tokens. Perplexity at 4k, 8k, and 16k context would directly test whether the exponential decay mechanism adequately captures very long-range dependencies. This is a natural stress test for an architecture that replaces exact attention with decaying recurrence.

2. RetNet at 13B with downstream evaluation. Table 4 includes a 13B model for training cost but no perplexity or downstream results are reported. The largest model evaluated for performance is 6.7B. If the trend of RetNet outperforming Transformer above 2B continues, the 13B model should show a larger gap — this would strengthen the scaling claim considerably.

3. Training curves (perplexity vs. tokens). All results report final perplexity after 100B tokens. Learning curves would reveal whether RetNet and Transformer converge at different rates, whether the advantage emerges early or late in training, and whether either architecture shows signs of plateauing.

4. Ablation of the chunk size $B$. The chunkwise training uses $B=512$ throughout. Ablating this would clarify the tradeoff between training speed (larger B = more parallelism) and training-inference gap (larger B = less cross-chunk recurrence during training, potentially making the model less adapted to pure recurrent inference). This is a practically important hyperparameter that the paper treats as fixed without justification.

5. Comparison against a compute-matched Transformer at larger scale. The paper shows RetNet training is faster per step (Table 4), but does not answer: if you give Transformer the same total GPU-hours as RetNet, how do their final perplexities compare? The faster training could be reinvested into more tokens or larger models, and a FLOPs-matched comparison would reveal whether RetNet's efficiency translates to better end-model quality at fixed training budget.

6. Evaluation on tasks requiring very long-range reasoning. The downstream benchmarks in Table 3 (HellaSwag, PIQA, StoryCloze) primarily require local coherence and commonsense reasoning, not multi-thousand-token dependencies. Tasks like long-document QA, multi-hop reasoning across long contexts, or the SCROLLS benchmark (which the paper cites as Shaham et al., 2022 but does not evaluate on) would test whether the exponential decay mechanism loses information that exact attention preserves.

7. Measuring the train-inference equivalence gap empirically. The paper's central claim is that the parallel and recurrent forms are exactly equivalent, so switching between them incurs no performance loss. This is mathematically true for the retention computation itself, but in practice, the model is trained with the chunkwise form (parallel within chunks, recurrent across) and inferred with the pure recurrent form. It would be valuable to measure whether inference-time perplexity differs when using the parallel form (computed over the full context) vs. the recurrent form (computed step-by-step). Any gap would reveal accumulated numerical error or distribution shift from the chunkwise training approximation.

Conditions under which the claims most clearly hold

Based on the evidence presented, the paper's claims are best-supported under the following conditions:

• Model scales between 1.3B and 6.7B parameters — the full suite of results (perplexity, downstream, training cost, inference cost) exists only in this range. Extrapolation to smaller or larger scales is speculative.
• Training budgets of ~100B tokens — all scaling experiments use this budget. At larger token budgets (1T+), the relative performance of RetNet and Transformer may shift.
• Context lengths up to ~8k tokens — inference cost measurements and language modeling evaluation (grouped results in Appendix B) cover this range. Beyond 8k, the exponential decay may become more limiting.
• GPU memory-constrained inference scenarios — the 70% memory reduction is most impactful when the KV cache is a significant fraction of total memory (i.e., for moderate-size models with long contexts). For very large models (175B+) where weight memory dominates, the relative advantage shrinks.
• Throughput-sensitive serving with moderate batch sizes — the latency insensitivity to batch size (Figure 6c) is most valuable when batching is needed for throughput. At batch size 1, the absolute latency advantage is still significant (50–60 ms vs. ~100 ms) but less dramatic than the throughput advantage at higher batch sizes.

The experiments do not establish that RetNet is uniformly superior to Transformer — the paper's own claims are more measured than that ("RetNet is a strong successor to Transformer," not "RetNet is always better"). The experiments do establish that the architecture resolves the impossible triangle at the tested scales in the sense that no single property is catastrophically sacrificed — RetNet trains in parallel (Table 4), infers with $O(1)$ cost (Figure 6), and performs competitively (Figure 5, Table 3). Whether this holds at GPT-3 scale (175B parameters, 300B+ tokens) or GPT-4 scale remains an open question that the paper explicitly defers to future work.

6. Limitations and Trade-offs

No Validation at Large Language Model Scales Beyond 6.7B Parameters

The most significant limitation is the scale at which RetNet's performance parity with Transformer has been empirically validated. The largest RetNet model evaluated for language modeling quality and downstream tasks is 6.7B parameters (Figure 5, Table 3). A 13B model appears only in training cost measurements (Table 4), with no corresponding perplexity or downstream accuracy reported. The paper explicitly defers scaling validation to future work:

"In the future, we would like to scale up RetNet in terms of model size and training steps" (Section 4)

The consequence is that the central claim — that RetNet resolves the "impossible triangle" by simultaneously achieving training parallelism, low-cost inference, and competitive performance — is only demonstrated at a scale that is modest by contemporary production standards. The architecture could exhibit different scaling behavior at larger sizes where the representational differences between exact softmax attention and exponential-decay-based retention may become more consequential. Several specific concerns arise:

• State capacity bottleneck at scale. The recurrent state matrix $S_n \in \mathbb{R}^{d \times d}$ with head dimension $d = 256$ has a fixed information capacity of $256 \times 256 = 65{,}536$ elements per head. At larger model sizes where the model must track more complex dependencies across longer contexts, this fixed-capacity state may become a bottleneck that softmax attention — which can attend to arbitrarily many past tokens with arbitrarily sharp selectivity — does not face. The paper's head dimension ablation (Table 6) shows that reducing $d$ from 256 to 64 increases perplexity by 1.63 points at 200M scale, confirming that state capacity matters. However, whether the default $d=256$ capacity suffices at 70B or 175B scale is completely untested.

• Scaling law exponents are unknown. The paper shows three data points on the scaling curve (1.3B, 2.7B, 6.7B). Fitting a power law to determine whether RetNet's perplexity scaling exponent matches Transformer's is impossible with three points. The observation that RetNet "tends to outperform Transformer when the model size is larger than 2B" (Section 3.2) is based on only two data points above that threshold (2.7B and 6.7B). If RetNet's scaling exponent is slightly worse than Transformer's, the architectures could cross again at 70B or 175B, with Transformer pulling ahead. The paper provides no theoretical argument for why the scaling exponents should be identical.

• Training token budgets are far below compute-optimal. All models are trained on 100B tokens. At 6.7B parameters, the Chinchilla-optimal token budget is approximately 130B tokens, but production models at this scale are often trained on 1T+ tokens. The 100B budget means models are significantly undertrained, and the observed performance ordering may not hold at longer training durations. Different architectures can have different learning dynamics — faster initial convergence followed by earlier plateauing, or slower initial progress with better ultimate performance. Learning curves (perplexity vs. tokens processed) are not reported, making it impossible to assess whether the RetNet advantage is stable or shrinking with more training.

What evidence exists: The scaling curves in Figure 5 show RetNet matching or slightly exceeding Transformer at the three tested sizes, but the error bars or confidence intervals are not reported, and the narrow y-axis range (roughly 13.0–15.0 PPL) means small absolute differences are visually amplified. The downstream task advantage for the 6.7B model (Table 3: +3.44 points aggregate zero-shot) is consistent with the perplexity trend, but downstream results are not reported for 1.3B and 2.7B models, so it is impossible to verify whether the advantage grows with scale as the paper implies.

Mitigation status: Not addressed. The paper identifies scaling to larger sizes as future work but provides no theoretical analysis or preliminary evidence (e.g., scaling law fits from the existing three data points) to suggest the trend will continue. A practitioner deciding whether to adopt RetNet for a 70B+ model is making a bet without evidence.

---

Inference Cost Advantage Is Measured at a Scale Where the KV Cache Dominates, and Does Not Generalize to Very Large Models

The headline inference numbers — 8.4× faster decoding, 70% less GPU memory, 3.4× higher throughput for the 7B model at 8k context length (Figure 1) — are measured at a specific operating point where the Transformer's key-value cache is a dominant fraction of total memory and computation. For the 6.7B model at 8k context on a single A100-80GB GPU:

• Model weights (FP16): approximately 13.4 GB
• KV cache at 8k context: roughly 32 GB (the paper reports ~42 GB total for Transformer vs. ~15 GB for model weights alone, Figure 6a)
• The KV cache is approximately 2.4× larger than the model weights

Eliminating the KV cache therefore saves roughly 64% of total memory — close to the claimed 70%. However, this ratio changes dramatically with model scale. For a 70B model:

• Model weights (FP16): approximately 140 GB
• KV cache at 8k context: scales proportionally, roughly 320 GB
• Total Transformer memory: ~460 GB across multiple GPUs
• RetNet memory: ~140 GB (weights only)
• Memory reduction: ~70% (still proportional), BUT requires multiple GPUs in both cases

The more critical shift is in the computation advantage. For small models, attention computation (which scales with sequence length) can be a significant fraction of total FLOPs. For very large models, the feed-forward network layers (which scale with model dimension but are independent of sequence length) dominate the computational budget. The recurrent form's advantage — replacing $O(Nd)$ attention with $O(d^2)$ state updates — matters less when the $8d^2$ FFN computation per layer dwarfs the sequence-mixing cost. The 8.4× decoding speedup is measured at 6.7B; at 175B, the speedup could be substantially smaller because the per-token cost is dominated by weight multiplication, not context processing.

The paper does not discuss this scale-dependence of the inference advantage. The measurements in Figure 6 use a single model size (6.7B) on a single GPU architecture (A100-80GB). The paper does not provide inference cost numbers for the 1.3B or 2.7B models, nor does it estimate or project the advantage at larger scales. The automatic claim that RetNet's inference advantage is "length-invariant" (Figure 6b) is true about RetNet's own cost scaling but says nothing about how the relative advantage over Transformer changes with model size.

What evidence exists: Figure 6 thoroughly characterizes the 6.7B model at sequence lengths from 2048 to 8192 and batch sizes from 1 to 8. The results are internally consistent and clearly show RetNet's cost being sequence-length-invariant while Transformer's grows. Figure 6a shows that RetNet's additional memory beyond model weights is only approximately 3% (the recurrent state). The paper does not measure or discuss inference cost for the 1.3B, 2.7B, or 13B models.

Mitigation status: Partially addressed. The paper acknowledges future deployment work: "we are interested in deploying RetNet models on various edge devices, such as mobile phones" (Section 4). For edge deployment with small models, the memory advantage is maximized because total memory is severely constrained. For data center deployment of very large models, the relative advantage is likely smaller than the headline numbers suggest, and the paper does not discuss this regime.

---

The Chunkwise Training Procedure Creates a Training-Inference Discrepancy That Is Not Empirically Measured

The paper's central architectural claim is that the parallel (Equation 5), recurrent (Equation 6), and chunkwise recurrent (Equation 7) forms of retention are mathematically equivalent — they compute exactly the same function. However, during training, the chunkwise form (not the pure parallel or pure recurrent form) is used for efficiency:

"We use the parallel (Equation (5)) and chunkwise recurrent (Equation (7)) representations during the training process" (Section 2.3)

At inference, the pure recurrent form is used. This creates a structural asymmetry: the model is trained with a computation that processes local chunks with full attention (the inner-chunk term in Equation 7) and summarizes cross-chunk information through a recurrent state. At inference, the model processes all tokens through the recurrent state — there is no inner-chunk parallel computation.

The mathematical equivalence between chunkwise and pure recurrent forms holds only in exact arithmetic. In practice, two sources of discrepancy arise:

1. Numerical precision accumulation. The recurrent form computes $S_n = \gamma S_{n-1} + K_n^\top V_n$ repeatedly, accumulating numerical error in the state matrix. The chunkwise form resets this accumulation at chunk boundaries by recomputing inner-chunk interactions from scratch. Over very long sequences, the pure recurrent state may drift from the chunkwise-computed state due to floating-point error accumulation, particularly for heads with $\gamma$ very close to 1 (where the decay is very slow and errors persist longer).

2. Gradient flow differences. During training with the chunkwise form, gradients flow through the inner-chunk parallel computation (providing dense, simultaneous gradient signals from all token pairs within the chunk) and through the cross-chunk recurrent state (providing more attenuated gradient signals across chunks). At inference, there is no inner-chunk dense computation — all token interactions are mediated through the recurrent state. The model may learn to rely on within-chunk attention patterns that the recurrent form can only approximate, creating a subtle train-inference gap that does not exist for the pure parallel or pure recurrent forms individually.

The paper does not measure this gap. There is no experiment comparing inference-time perplexity using the parallel form (computed over the full context, which is the "true" computation) versus the recurrent form (computed step-by-step) versus the chunkwise form. Such a comparison would directly quantify whether the numerical equivalence holds in practice or whether error accumulation causes measurable degradation.

What evidence exists: The paper reports perplexity evaluated at different context lengths (Table 8) and inference cost (Figure 6), but these are separate measurements. Perplexity evaluation likely uses the parallel form (computing over the full context at once, since evaluation does not require autoregressive decoding), while inference cost measurements use the recurrent form. The two are never compared on the same input to verify equivalence. The training efficiency results (Table 4) use the chunkwise form with $B=512$, but there is no ablation of chunk size to test whether models trained with larger chunks (more parallel, less recurrent) perform differently at recurrent inference than models trained with smaller chunks (more recurrent, less parallel).

Mitigation status: Not addressed. The paper asserts mathematical equivalence but does not empirically verify it in the presence of finite-precision arithmetic and the specific training configuration (chunkwise training, recurrent inference). A practitioner deploying RetNet for very long contexts (e.g., 32k tokens) would want to know whether the recurrent state accumulates meaningful error after thousands of steps, particularly for heads with slow decay. The paper's maximum evaluated context length for language modeling is 2048 (Table 8), far short of the 8192 used in inference cost measurements, and even farther from the long-context scenarios (100k+ tokens) that the chunkwise form is theoretically well-suited for.

---

Training Data Efficiency and Optimization Dynamics Are Not Characterized

The paper compares RetNet and Transformer at fixed training token budgets (100B tokens for scaling experiments, 5B tokens for the 200M comparison) but provides no information about how quickly each architecture learns. Training curves (validation perplexity vs. tokens processed) are not reported for any experiment. This omission prevents answering several practically important questions:

• Does RetNet learn faster or slower than Transformer? If RetNet achieves the same perplexity with fewer training tokens, its effective advantage is larger than the final perplexity numbers suggest — the faster training per step (Table 4) compounds with faster learning per token. If RetNet learns slower but catches up, the advantage depends on total training budget.

• Does the performance gap widen or narrow over training? The scaling curves (Figure 5) are snapshots at 100B tokens. If the gap is widening (RetNet pulling ahead), the final advantage understates the benefit at longer training. If the gap is narrowing (Transformer catching up), RetNet might underperform at the 1T+ token budgets used for production models. A narrowing gap would suggest that Transformer's softmax attention provides a representational advantage that matters more as the easier patterns are learned and only hard long-range dependencies remain, while a widening gap would suggest that the exponential decay prior provides better inductive bias for language.

• Are there instability issues at specific points in training? The paper notes that "the RetNet training is quite stable in our experiments" (Section 3.2) and uses DeepNet initialization for stability, but without training curves, transient instabilities (loss spikes, perplexity regressions) that might occur early in training and require intervention are invisible. The fixed $\gamma$ decay rates and the retention score normalization tricks (Section 2) are designed to maintain stability, but their effectiveness throughout training is not demonstrated.

The absence of training curves also makes it difficult to assess whether the hyperparameter choices (learning rate schedule, warmup, batch size) are equally well-tuned for both architectures. The paper uses the same training hyperparameters for RetNet and Transformer (Table 7), which may favor one architecture over the other if their optimal learning dynamics differ. The chunkwise training of RetNet creates a different gradient signal structure than full-attention Transformer training, and the optimal learning rate or batch size might differ.

What evidence exists: Figure 5 provides final perplexity at three model sizes after 100B tokens. Table 5 provides final perplexity for the 200M comparison after 5B tokens. No intermediate checkpoints, learning curves, or training dynamics analysis is reported. The paper's claim that RetNet achieves "favorable scaling results" (abstract) is based entirely on endpoint comparisons, not on the trajectory of learning.

Mitigation status: Not addressed. The paper does not mention this as a limitation or suggest that training dynamics analysis is needed. For a practitioner deciding whether to invest in RetNet training infrastructure, the absence of learning curve data means the total cost to reach a target perplexity (combining per-step speed and per-token learning efficiency) cannot be estimated from the reported results alone. The faster per-step training (Table 4) is a known advantage; whether this translates to faster time-to-convergence depends on unmeasured learning dynamics.

---

The Fixed Exponential Decay Prior May Be Too Restrictive for Tasks Requiring Sharp, Content-Selective Attention Over Long Distances

The retention mechanism replaces softmax attention's content-dependent, competition-based selectivity with a fixed exponential decay prior modulated by content-aware query-key dot products. The softmax in standard attention creates a probability distribution over past tokens, enabling the model to attend very sharply to a small number of highly relevant tokens while nearly completely ignoring others — this is the "winner-take-all" dynamic that gives attention its ability to precisely route information. In retention, even if the query-key dot product $Q_n K_m^\top$ is very high for a distant token, the contribution is still dampened by $\gamma^{n-m}$.

For heads with fast decay ($\gamma \approx 0.97$), $\gamma^{100} \approx 0.05$ — a token 100 steps back is weighted at most 5% of its query-key similarity, regardless of how relevant it is. For heads with slow decay ($\gamma \approx 0.9998$), $\gamma^{1000} \approx 0.82$, so distant tokens can still contribute substantially, but these slow-decay heads have limited temporal resolution for recent tokens (since the decay is slow, they treat recent and moderately distant tokens similarly). The multi-scale design (different $\gamma$ per head) mitigates this by providing heads that specialize in different temporal ranges, but it does not change the fundamental constraint: a head's temporal selectivity is structurally coupled to its decay rate, and the decay is monotonic — closer tokens always receive higher base weight than farther tokens, regardless of content.

This structural constraint may become limiting for tasks that require:

• Exact retrieval of information from a specific distant location — e.g., "What was the name of the character introduced in the first paragraph?" at the end of a long document. Softmax attention can give near-100% weight to that specific token if the query-key match is strong enough, regardless of distance. Retention with decay will always multiply the attention weight by $\gamma^{\text{distance}}$, so even with the slowest decay head ($\gamma \approx 0.9998$), a token 10,000 steps back receives at most ~13.5% of the weight it would receive if it were adjacent ($0.9998^{10000} \approx 0.135$).
• Sharp, content-based filtering — e.g., in a legal document, attending only to clauses that contain specific terms while ignoring intervening text. Softmax attention naturally suppresses irrelevant tokens (their scores become near-zero after softmax), while retention with decay assigns at least $\gamma^{n-m}$ times the query-key weight to every token, potentially accumulating noise from many weakly-relevant tokens.
• Tasks where temporal proximity is anti-correlated with relevance — e.g., resolving a pronoun that refers to an entity mentioned three paragraphs ago. The exponential decay prior assumes that closer tokens are more relevant on average, which is true for most natural language but systematically fails for long-distance coreference resolution, where the most recent mention may be irrelevant and the relevant antecedent is far away.

The paper's language modeling experiments do not stress-test these limitations. The downstream benchmarks in Table 3 primarily require local coherence and commonsense reasoning. The context length ablation (Table 8) shows that RetNet benefits from longer context, but the maximum evaluated length is 2048 tokens — far short of the regimes (10k–100k tokens) where the exponential decay constraint would be most limiting. The paper does not evaluate on tasks like long-document question answering, multi-hop reasoning across long contexts, or needle-in-a-haystack retrieval that would directly probe whether retention can match attention's ability to precisely attend to specific distant tokens.

What evidence exists: The multi-scale decay ablation (Table 6) shows that having multiple decay rates (in-domain PPL 26.05) outperforms a single rate (27.02) and dramatically outperforms no decay (27.86), confirming that the exponential prior is beneficial on average for language modeling. However, this does not test whether the prior is ever harmful for specific token predictions. The out-of-domain results (Table 5) show RetNet outperforming alternatives on GovReport and SummScreen, which involve longer documents, but perplexity is an average metric that can mask occasional catastrophic failures on specific long-distance dependencies. The paper provides no qualitative analysis of cases where retention's decay prevents attending to the correct distant token.

Mitigation status: Not addressed as a limitation. The paper frames exponential decay as a feature (enabling recurrence) and provides ablation evidence that it helps on average. However, the structural tradeoff — that monotonic decay inherently limits sharp, distance-agnostic selective attention — is not discussed. A practitioner deciding whether to use RetNet for tasks requiring precise long-range retrieval (legal document analysis, codebase understanding where a function call must resolve to its definition 5000 tokens earlier) would want to evaluate this directly. The paper provides no such evaluation and does not propose mechanisms (e.g., learned decay rates that can become very close to 1, or a hybrid attention-retention mechanism) that could address this constraint.

---

Computational Cost of Difficulty Estimation Is Not Accounted for in the Practical Efficiency Picture

The chunkwise recurrent training procedure requires selecting a chunk size $B$, which the paper sets to 512 for all experiments without ablation or justification. While this is not as extreme as the 2048-sample difficulty estimation cost in the reference example paper, it represents a similar class of limitation: a hyperparameter that fundamentally affects both training efficiency and the train-inference gap is set once and never analyzed.

More significantly, the chunkwise form introduces a direct tradeoff that the paper does not characterize:

• Larger $B$ (e.g., 1024, 2048): More within-chunk parallelism, faster training, but the model sees more of its computation through the parallel form (full attention within the chunk) and less through the recurrent form (cross-chunk state passing). This could lead to the model learning patterns that depend on within-chunk parallel computation, which the pure recurrent inference form cannot replicate exactly. The train-inference gap grows.
• Smaller $B$ (e.g., 64, 128): Training is more recurrent-like (more chunks, more cross-chunk state passing), potentially better preparing the model for recurrent inference, but less GPU parallelism and slower training. The efficiency advantage over Transformer shrinks.

The paper's choice of $B=512$ is implicitly treating this tradeoff as resolved, but without ablation, a practitioner cannot assess whether $B=512$ is near-optimal, whether the model quality is sensitive to this choice, or whether different sequence lengths during training or inference would warrant different chunk sizes.

A related unaccounted cost: the paper uses complex-valued arithmetic for the $\Theta = e^{in\theta}$ rotations (Equations 5, 8). Complex multiplication requires 4 real multiplications and 2 additions (or 3 multiplications with the Karatsuba trick), compared to 1 multiplication for real-valued operations. The recurrent state stores complex-valued matrices, doubling the state memory. The paper does not discuss whether complex arithmetic introduces overhead in training or inference, whether standard deep learning frameworks handle complex tensors efficiently, or whether the complex rotation could be replaced with an equivalent real-valued rotation (as RoPE does) without loss of the dual-form property.

What evidence exists: Table 4 reports training memory and throughput for RetNet with $B=512$. The paper notes RetNet is implemented in "vanilla PyTorch code" and leaves "kernel fusion or FlashAttention-like acceleration for future work" (Section 3.3). No ablation of chunk size or analysis of complex arithmetic overhead is provided.

Mitigation status: Acknowledged for the kernel fusion aspect (the paper states it is left to future work). Not acknowledged for the chunk size tradeoff or complex arithmetic overhead. The practical cost of RetNet training relative to the headline numbers depends on these unanalyzed factors. A practitioner implementing RetNet would need to determine chunk size and handle complex arithmetic without guidance from the paper.

7. Implications and Future Directions

How This Work Changes the Landscape

RetNet represents a paradigm shift in the primitives of sequence modeling architecture design, not merely an incremental efficiency improvement over Transformers. The paper's central reframing — that the parallel and recurrent forms of a sequence-mixing operation should be algebraically equivalent manifestations of the same computation, rather than approximations of each other — changes what it means to design an architecture for sequence modeling. Before RetNet, the field implicitly accepted that training parallelism and low-cost inference were opposing objectives to be balanced through approximation: linear attention approximated softmax to gain recurrence; RWKV accepted sequential training to gain efficient inference; S4 accepted content-unaware dynamics to gain both. RetNet demonstrates that this tradeoff is not inherent to sequence modeling — it is an artifact of choosing a computational form (softmax attention) that cannot be factorized, and then attempting to patch around that limitation. The paradigm shifts from "how do we make attention cheaper?" to "what recurrent operations admit exact parallel unrolling, and how do we make those content-aware?"

This shift has several concrete consequences for the research landscape:

It redefines the architectural search space. The paper's derivation path — start with a linear recurrence with a diagonalizable transition matrix, ensure content-awareness through learned projections, and derive the parallel form algebraically — provides a template for designing future architectures. Rather than beginning from attention and seeking efficiency, researchers can begin from a recurrence and verify that it admits an exact parallel representation. The mathematical criterion is crisp: the recurrence must be linear in the state update (nonlinear recurrences like LSTMs cannot be unrolled into a single matrix multiplication) and the transition matrix must be diagonalizable (so that raising it to powers remains tractable). This narrows the search space to a well-defined class of operations, making architectural innovation more principled and less dependent on empirical trial-and-error.

It reconciles contradictory findings about the viability of recurrent models for large-scale language modeling. Prior work oscillated between "RNNs are dead; Transformers won" (the dominant view after 2017) and "RNNs can be revived with the right optimizations" (the RWKV and S4 lines of work). RetNet provides a synthesis: recurrent models can match Transformers, but only when the recurrence is designed to have an exact parallel dual — meaning the model trained in parallel is literally the same model deployed recurrently. The failures of prior recurrent approaches (element-wise operators harming capacity, content-unaware state-space models underperforming) are not failures of recurrence per se, but failures to achieve this exact dual-form property while maintaining content-awareness. This reframing explains why RWKV (which lacks a true parallel form) and S4 (which lacks content-awareness) underperform while RetNet succeeds, resolving the apparent contradiction between "RNNs can work" (RWKV showed they can be competitive) and "RNNs can't match Transformers" (prior recurrent models consistently underperformed at scale).

It redirects attention from engineering optimization to algebraic structure. The paper's comparison against FlashAttention (Table 4) is revealing: RetNet implemented in vanilla PyTorch matches or exceeds a heavily optimized Transformer implementation in both memory and throughput. The implication is that algorithmic structure (O(NBd) chunkwise complexity vs. O(N²d) full attention) provides efficiency gains that engineering optimization alone cannot match. This does not mean engineering optimization is irrelevant — RetNet could benefit from kernel fusion just as Transformers did — but it reframes the priority: get the algebraic structure right first, then optimize. A research program focused on finding operations with exact dual forms is likely to yield larger efficiency gains than one focused on optimizing operations with inherently unfavorable scaling.

It makes the "impossible triangle" a falsifiable diagnostic rather than a rhetorical device. Before RetNet, the triangle could have been dismissed as an oversimplification — perhaps the three properties simply could not coexist in any architecture, making the framing a description of an unavoidable tradeoff rather than a solvable problem. RetNet provides existence proof that they can coexist, at least at the tested scales. This transforms the triangle from a lament about architectural limitations into a concrete benchmark: any proposed successor architecture can be evaluated on whether it simultaneously achieves training parallelism (measured by throughput and memory at scale), low-cost inference (measured by per-token latency and memory growth with sequence length), and competitive performance (measured by scaling curves against a Transformer baseline). The paper's Table 1 provides a template that future architecture papers must fill in, with RetNet as the first entry to receive checkmarks in all three columns.

However, the magnitude of this shift should not be overstated. RetNet has been validated at up to 6.7B parameters and 100B training tokens. Transformer architectures have been validated at 540B+ parameters and trillions of tokens. The paradigm shift RetNet proposes — that exact dual-form equivalence is the path forward — depends on whether the scaling trends observed at 1.3B–6.7B continue to larger scales. If RetNet's performance degrades relative to Transformer at 70B+ parameters (due to state capacity limits, the restrictiveness of the exponential decay prior, or unobserved training instability), the shift would be partial: RetNet would be recognized as a superior architecture for moderate-scale, inference-constrained deployments, while Transformers would remain dominant for the largest models. The paper's contribution would still be significant, but the paradigm would be "dual-form architectures for efficiency-critical applications" rather than "the recurrence is the true primitive, and attention is a historical accident."

Which research directions become more attractive:

• Algebraic architecture design. Searching for other linear recurrences that admit exact parallel unrolling, perhaps with different decay structures (non-exponential, data-dependent), different state transition parameterizations, or different query-key-value formulations. The retention derivation provides a template: start from a recurrence, diagonalize the transition, add content-awareness through projections.
• State capacity optimization. Understanding how the recurrent state matrix dimension $d$ scales with model size and sequence length — is there a capacity law analogous to Chinchilla scaling laws that relates state dimension to optimal performance at a given compute budget?
• Training-inference co-design. The chunkwise form introduces a train-inference gap that the paper does not measure or mitigate. Understanding how training procedures (chunk size, loss functions, regularization) affect recurrent inference quality is a new research axis that Transformers did not require.
• Hybrid architectures. Combining retention heads with a small number of full-attention heads for tasks requiring sharp long-range retrieval, or using retention for most layers with attention only at specific layers where precise token-level routing is critical.

Which research directions become less attractive:

• Pure kernel-based attention approximations. The paper shows linearized attention dramatically underperforming (40.24 PPL vs. 26.05 for RetNet at 200M scale, Table 5). If exact dual-form equivalence is the path forward, research into better kernels to approximate softmax is attacking the wrong problem — the goal should be to find operations that don't need softmax in the first place.
• Element-wise sequence mixing for efficiency. RWKV's element-wise operators trade capacity for speed, and RetNet shows that this tradeoff may be unnecessary — a properly designed dual-form architecture can achieve efficiency without sacrificing the dense token-to-token interactions that give attention its power.
• Architecture-agnostic inference optimizations (KV-cache compression, speculative decoding) as primary research goals. While these techniques remain practically valuable, RetNet suggests that the fundamental solution to inference cost is architectural, not a post-hoc optimization. Research effort might be better directed at finding architectures with O(1) inference natively rather than engineering around the Transformer's O(N) limitation.

---

Follow-Up Research This Work Enables

Scaling law analysis for RetNet: determining whether the performance parity with Transformer holds beyond 6.7B parameters. The paper shows RetNet matching or slightly exceeding Transformer at 1.3B, 2.7B, and 6.7B parameters (Figure 5), with the gap appearing to widen above 2B. However, three data points in the undertrained regime (100B tokens for models that would typically receive 300B–1T tokens) are insufficient to establish a scaling law. A direct follow-up would train RetNet and Transformer at matched parameter counts across a wider range — e.g., 1.3B, 2.7B, 6.7B, 13B, 30B, 70B — each trained to Chinchilla-optimal token budgets (e.g., 20 tokens per parameter or more), and fit power laws to the resulting perplexity curves. The key question is whether the scaling exponents differ. If RetNet's exponent is worse, the architectures will cross at some scale and Transformer will pull ahead; if RetNet's exponent is equal or better, the "successor" claim is strengthened. This experiment would also test whether the state capacity limit (the fixed d×d recurrent state at each head) creates a bottleneck at larger scales where more information must be tracked — if perplexity plateaus earlier for RetNet than for Transformer as model size increases, it would indicate a fundamental capacity constraint. The training cost measurements in Table 4 already provide per-step throughput for the 13B model, but perplexity and downstream results for that model are not reported. Filling in that data point alone would extend the scaling evidence by one more order of magnitude in parameters.

Empirical measurement of the train-inference equivalence gap across sequence lengths. The paper's central theoretical claim is that the parallel, recurrent, and chunkwise forms compute exactly the same function. In practice, the model is trained with the chunkwise form (Equation 7, B=512) and inferred with the pure recurrent form (Equation 6). Two sources of discrepancy exist: numerical error accumulation in the recurrent state over long sequences (especially for slow-decay heads with γ ≈ 0.9998), and potential learned reliance on within-chunk parallel attention patterns that the recurrent form can only approximate. A direct experiment would measure: for a trained RetNet model, compute perplexity on a held-out set using (a) the pure parallel form on the full sequence (Equation 5), (b) the pure recurrent form processing tokens one at a time (Equation 6), and (c) the chunkwise form with varying chunk sizes (Equation 7). Compare these perplexities at sequence lengths of 512, 1024, 2048, 4096, 8192, and 16384. If the gap between parallel and recurrent perplexity grows with sequence length, it indicates numerical drift in the recurrent state. If the gap is larger for models trained with larger B (say, 1024 vs. 512), it indicates a chunk-size-dependent train-inference mismatch. This experiment would directly validate or refute the practical equivalence claim and provide guidance on whether recurrent inference at very long contexts (100k+ tokens) is reliable or requires periodic state resetting. The paper already has the infrastructure to run this experiment — it evaluates perplexity at different context lengths (Table 8) and runs recurrent inference (Figure 6) — but never compares the two on the same input.

Needle-in-a-haystack retrieval evaluation to stress-test the exponential decay prior. The retention mechanism weights past tokens by γ^(n−m), which inherently favors recent tokens regardless of content relevance. This structural prior is beneficial on average for language modeling (ablation: removing decay increases perplexity by 1.81, Table 6) but may fail catastrophically when a specific distant token must be retrieved with high precision. The classic needle-in-a-haystack test — insert a specific fact (e.g., "The special key is FLOWERS") at a fixed position in a long document (2k, 4k, 8k, 16k, 32k, 64k tokens), then query the model at the end ("What is the special key?") — would directly measure whether retention's decay prevents attending to the distant fact. A strong follow-up would compare RetNet against Transformer at multiple insertion positions (early, middle, late in the document) and multiple context lengths. The hypothesis: RetNet performs well when the "needle" is recent (within the effective window of the slowest-decay head) but degrades sharply when the needle is very distant, while Transformer's full attention retrieves it regardless of distance. This experiment would map the effective memory horizon of the multi-scale decay rates and determine whether tasks requiring precise long-range retrieval (legal document analysis, codebase understanding, multi-hop QA over long contexts) are within RetNet's capabilities or require architectural modifications (e.g., hybrid attention-retention, or a learned retrieval mechanism).

Learning dynamic decay rates: can γ be input-dependent while preserving the dual form? The paper uses fixed, unlearned decay rates per head, chosen to span a range of temporal scales via γ = 1 − 2^(−5−arange(0,h)). This ensures the dual form remains exact (since γ is a scalar constant per head), but it means the temporal weighting cannot adapt to the input — a head that decays slowly treats all inputs with the same temporal profile. An important extension would explore whether γ can be made content-dependent (e.g., γ_n = f(X_n) for some learned function f) while still preserving the exact parallel-recurrent equivalence. The challenge: if γ varies per token, the recurrence becomes S_n = γ_n S_{n−1} + K^⊤n V_n, and the unrolled form becomes a product of varying decay factors ∏{k=m+1}^{n} γ_k rather than a simple γ^(n−m). This product can still be computed in parallel (as cumulative products), but whether the resulting operation has an efficient chunkwise form and whether it improves performance is an open question. A concrete experiment: add a small MLP that predicts γ_n from the current token's representation, train RetNet models at 200M scale with and without learned γ, and compare perplexity and retrieval accuracy on the needle-in-a-haystack test. If learned γ improves long-range retrieval without destabilizing training (the fixed γ likely contributes to training stability, as the paper notes), it would be a significant extension. If learned γ causes training instability or the model collapses to γ ≈ 1 for all tokens (losing the temporal structure), it would establish a boundary condition on when fixed priors are preferable to learned ones.

Combining a small number of full-attention layers with retention layers: does hybrid architecture capture the best of both? Retention excels at efficient, broad-context modeling through multi-scale decay, but the absence of softmax means it cannot produce the sharp, highly selective attention patterns that softmax attention achieves for tasks like precise token-level routing (e.g., copying a specific entity name from earlier in the text). Full attention is expensive at scale but may only be needed at specific layers — perhaps the first few layers for local feature extraction, the last few for task-specific output preparation, or specific middle layers where content-based filtering is critical. A hybrid architecture could use retention for the majority of layers (gaining O(1) inference for those layers) and insert full attention at a small number of strategic positions (paying O(N) only for those specific layers). A concrete experiment: train a 6.7B model where layers 1, L/2, and L use full attention and all other layers use retention, compare against pure RetNet and pure Transformer on (a) language modeling perplexity, (b) the needle-in-a-haystack retrieval test, (c) long-document QA tasks from SCROLLS, and (d) inference cost at 8k context. The goal is to find the minimal number of attention layers needed to close any performance gap on tasks requiring precise long-range retrieval, while preserving most of RetNet's inference efficiency advantage. The paper's own related work discusses S4 and H3 in the context of replacing attention entirely — a hybrid approach would test whether retention's strengths (efficient broad-context modeling) and attention's strengths (precise content-selective routing) are complementary rather than competing.

Application of retention to encoder-decoder architectures and seq2seq tasks. The paper evaluates RetNet exclusively on decoder-only language modeling — next-token prediction and downstream classification tasks. Many important applications use encoder-decoder architectures (machine translation, summarization, speech recognition) where the encoder processes the full input sequence (potentially very long) and the decoder generates outputs autoregressively. In this setting, the encoder could use the parallel or chunkwise form of retention (since the full input is available at once), while the decoder uses the recurrent form for autoregressive generation. The encoder would benefit from linear-complexity processing of long inputs; the decoder would benefit from O(1) per-step generation cost without a cross-attention KV cache over the encoder output (since the encoder output would be summarized in the recurrent state). A concrete experiment: replace the self-attention and cross-attention layers in a standard Transformer encoder-decoder with retention variants, train on a long-document summarization task (e.g., GovReport, SummScreen — which the paper uses for perplexity evaluation in Table 5 but does not train on directly), and measure both generation quality (ROUGE scores) and inference efficiency (latency and memory for long input documents). This would test whether retention's advantages extend beyond language modeling to conditional generation, and whether the recurrent state can effectively summarize a long encoder input for cross-attention in the decoder.

On-device deployment benchmarking for small RetNet models. The paper's abstract and conclusion mention deployment on edge devices and mobile phones as a motivation, but no experiments are conducted in resource-constrained environments. A practical follow-up would: train a small RetNet model (e.g., 125M parameters, suitable for on-device deployment), quantize it to INT8 or INT4 precision, and benchmark inference latency, memory, and energy consumption on a mobile device (e.g., an iPhone or Android phone) for autoregressive text generation at varying context lengths (100 to 4096 tokens). Compare against an equivalently sized Transformer model with KV-cache quantization (a standard on-device optimization). The key metrics: (a) at what context length does the Transformer's KV cache exceed the device's memory budget, making generation impossible, while RetNet continues to function? (b) What is the energy-per-token difference, which directly impacts battery life for on-device assistants? The paper already provides the A100 inference benchmarks (Figure 6); replicating these on mobile hardware with small models would directly validate the deployment motivation and provide practical guidance for on-device LLM design. The memory analysis in Figure 6a suggests that RetNet's memory is dominated by model weights (97%), with the recurrent state contributing only ~3% — for a small quantized model, this means the total memory footprint could be small enough for always-on operation, a regime Transformers with growing KV caches cannot achieve.

---

Practical Applications and Downstream Use Cases

High-throughput LLM serving with stringent latency requirements. For serving infrastructure that must handle many concurrent user requests — chatbots, code completion APIs, real-time translation — RetNet's latency insensitivity to batch size (Figure 6c) is the decisive advantage. In a standard Transformer serving setup, increasing batch size improves GPU utilization and throughput (more requests processed per second) but degrades per-request latency because each request's KV cache grows and attention must scan it. Operators typically cap batch sizes at 1–4 to keep latency acceptable for interactive use, leaving GPU compute underutilized. RetNet's decoding latency at 8k context is approximately 50–60 ms and nearly flat from batch size 1 to 8 (Figure 6c), meaning operators can increase batch size to maximize throughput without latency penalties. The paper's numbers suggest a practical improvement: at batch size 8 and 8k context, Transformer latency is approximately 350 ms per token vs. RetNet's ~60 ms — a 5.8× improvement in user-perceived latency for the same throughput, or equivalently, the ability to serve 5–6× more users at the same latency. For a production deployment processing millions of requests per day, this directly reduces the number of GPUs needed and improves user experience simultaneously. The memory advantage (70% reduction at 8k) further compounds this by allowing larger models or longer contexts on the same hardware, reducing the need for model parallelism or request truncation.

Long-document processing and analysis at scale. For applications that must process entire documents — contract review, scientific literature analysis, long-form content generation with full-document context — the Transformer's quadratic memory scaling in input length forces compromises: documents must be chunked (losing cross-chunk context), summarized first (losing detail), or processed with sparse attention patterns (losing faithfulness). RetNet's chunkwise training (linear memory in sequence length, per Section 3.3) and recurrent inference (O(1) per-token memory regardless of context length, per Figure 6a) mean that processing a 100k-token document incurs the same per-token inference cost as processing a 1k-token document. The chunkwise form during training enables efficient processing of very long sequences without the O(N²) memory blowup that makes training Transformers on 100k-length sequences prohibitively expensive even with gradient checkpointing. Specific numbers: the paper shows RetNet at 6.7B training on 8192-length sequences with 48 GB GPU memory (Table 4). A Transformer without FlashAttention requires 69 GB for the same sequence length — at 32k length, the Transformer would require approximately 4× more memory for the attention matrix alone (~96 GB additional for KV cache), likely exceeding even multi-GPU configurations, while RetNet's memory would remain approximately constant. For a legal tech company processing 50,000-word contracts, RetNet enables end-to-end processing without chunking artifacts, directly improving the accuracy of clause extraction, obligation identification, and cross-reference resolution that depend on long-range context.

On-device language models for privacy-sensitive applications. Deploying LLMs on users' devices — for email composition, text prediction, document summarization, or accessibility features — requires models that operate within severe memory and energy constraints without network connectivity. The paper's memory analysis (Figure 6a: RetNet's memory is 97% model weights, 3% recurrent state, independent of context length) is directly relevant: for a small 125M-parameter model quantized to 4 bits (~62 MB for weights), the total memory footprint is approximately 64 MB regardless of conversation length. An equivalent Transformer with 8k context would require an additional ~32 MB for the KV cache (in FP16), roughly 50% more memory — and this grows with every token generated. For an always-on keyboard assistant that processes everything the user types, a Transformer's growing cache eventually exhausts the device's memory budget, forcing cache eviction and context loss. RetNet's fixed memory footprint means the assistant can maintain full conversation history indefinitely — an entire day's typing — without memory growth or context truncation. The latency advantage (50–60 ms per token vs. 100+ ms for Transformer, Figure 6c for batch size 1) also matters for on-device use where user-perceived responsiveness is critical. The paper's explicit mention of edge deployment in the conclusion suggests the authors see this use case as central; the benchmark results, while measured on A100 GPUs, provide architectural evidence that the advantage would carry to mobile hardware.

Self-improving training pipelines with efficient data generation. When using LLMs to generate training data for iterative self-improvement (a technique used in methods like STaR, ReST, and constitutional AI), the inference cost of generating millions of training examples can dominate the total compute budget. If each training example requires generating a 2k-token response, and the pipeline generates 10 million examples, the total inference cost is 20 billion tokens of generation. With Transformer at 6.7B and 2k context, generating 2k tokens takes approximately 2k × (latency per token at batch size 1, roughly 100 ms from Figure 6c) ≈ 200 seconds per example (serial, single request). With RetNet at comparable throughput (Figure 6b: ~280 wps at 2k context), the same generation takes approximately 7 seconds per example — a 28× speedup that translates to generating the 10 million examples in ~81 GPU-days vs. ~2,300 GPU-days for Transformer. For research labs and companies running iterative training pipelines (generating data, training on it, generating better data, retraining), this efficiency difference determines whether such pipelines are economically feasible. RetNet's training efficiency advantage (Table 4: 6.8× faster throughput than vanilla Transformer at 1.3B, 7.5× at 2.7B) further compounds this, as the training phase between generation rounds is also faster. The paper does not evaluate self-improvement pipelines directly, but the architectural properties make RetNet an attractive backbone for any workflow where inference cost is the bottleneck.

---

When to Prefer This Method

The paper does not articulate an explicit tradeoff framework or decision rule for choosing RetNet over Transformer (or vice versa) under specific conditions. The positioning is broadly "RetNet is a successor to Transformer" — a replacement, not an alternative to be selected situationally. The experimental comparisons (Figures 5, 6; Tables 3, 4, 5) consistently show RetNet matching or exceeding Transformer on all measured axes at the tested scales. The paper does not identify any regime where Transformer is preferable, nor does it discuss failure modes or conditions under which RetNet underperforms.

However, the limitations analysis (Section 6 of this document) identifies practical constraints that a deployer should consider, even if the paper does not frame them as explicit tradeoffs. Based on what the paper demonstrates versus what remains unvalidated:

Deploy RetNet today (strongest evidence) when:

• Model size is 1.3B–6.7B parameters — the full suite of performance, training cost, and inference cost results exists in this range.
• Inference cost (latency, throughput, memory) is the primary constraint — the 8.4× faster decoding, 70% less memory, and batch-size-insensitive latency (Figure 6) are the paper's most robustly demonstrated advantages.
• Training budget is approximately 100B tokens — all scaling experiments use this budget, and RetNet's training dynamics at 1T+ tokens are unknown.
• Context lengths are 2k–8k tokens — inference cost measurements and language modeling evaluation cover this range.
• The application is decoder-only autoregressive generation — the paper evaluates only this architecture type.

Exercise caution (weaker or no evidence) when:

• Model size exceeds 13B parameters — no perplexity or downstream results are reported beyond 6.7B; the 13B model appears only in training cost measurements (Table 4).
• Training budgets are very large (500B+ tokens) — learning curves are not reported, and it is unknown whether RetNet and Transformer converge at different rates or to different asymptotes.
• The task requires sharp, precise retrieval of specific tokens from very long contexts (10k+ tokens) — the exponential decay prior may limit such retrieval, and no stress-test evaluation exists.
• The architecture is encoder-decoder or requires cross-attention — only decoder-only models are evaluated.
• The task involves modalities beyond text — the paper mentions multimodal LLMs as future work but provides no results.