Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

🎯 Pitch

Finch (RWKV-6) achieves up to 4.2× faster training than Flash Attention at 16K sequence length with 40% less memory, proving that RNNs can decisively beat transformer efficiency when inference demands O(1) per-token cost. It achieves this via a simple but powerful innovation: replacing static per-channel decay with data-dependent recurrence that lets the model dynamically decide what to forget.

1. Executive Summary

This paper introduces two new recurrent architectures — Eagle (RWKV-5) and Finch (RWKV-6) — that advance the RWKV family of RNN-based language models through multi-headed matrix-valued states (replacing the scalar-valued states of RWKV-4 with per-head $K^T V$ matrices, expanding from $\mathbb{R}^D$ to $\mathbb{R}^{(D/h) \times (D/h)}$ ) and a dynamic recurrence mechanism (replacing static learned per-channel decay rates with data-dependent time-varying decay $w_t$ computed via LoRA-augmented linear interpolations). Across models scaling from 0.46B to 7.5B parameters trained on a new 1.12-trillion-token multilingual corpus, Eagle and Finch achieve competitive performance on English benchmarks while substantially advancing the multilingual Pareto frontier — Finch-3B reaches 57.1% average accuracy on multilingual tasks versus 53.9% for RWKV-4-3B — and Finch demonstrates 4.2× faster training than Flash Attention at 16K sequence length while using 40% less memory, establishing that RNN architectures can match or exceed transformer efficiency only when inference-time constraints demand $O(1)$ per-token memory and unbounded context handling.

2. Context and Motivation

The Core Problem: The Transformer's Quadratic Inference Cost

The fundamental tension this paper addresses is deceptively simple: the Transformer architecture — despite dominating generative sequence modeling — imposes a fundamental cost structure that makes it unsuitable for many deployment scenarios. Specifically, the standard multi-headed self-attention mechanism requires $O(N)$ time and $O(N)$ memory per inference step when handling a sequence of length $N$ , because attention computes pairwise interactions between the current token and every token in the preceding context. This quadratic cost manifests in two practical pathologies:

KV cache memory explosion: At inference time, Transformers must store the key and value projections of every token processed so far. For a model with $L$ layers, hidden dimension $D$ , and sequence length $N$ , the KV cache consumes $O(L \cdot N \cdot D)$ memory. This means that a 7B-parameter model serving a single request with 128K tokens can require over 12 GB of memory just for the KV cache, independent of model weights. For batched serving with hundreds of concurrent requests, this becomes the dominant infrastructure cost, dictating deployment hardware requirements.
Unbounded per-step compute growth: While the first token of a sequence requires only $O(1)$ computation, each subsequent token must attend to all preceding tokens. By the $N$ -th token, a single forward pass requires $O(N)$ operations. This means latency grows linearly with the number of tokens already processed, making Transformers inherently less efficient for long conversations, streaming applications, or any scenario where the same model processes growing context over time.

These costs are not merely asymptotic concerns — they directly determine whether models can run on consumer devices, how many concurrent users a single GPU can serve, and whether a given application is economically viable at scale.

Why This Problem Matters

The paper's motivation extends beyond academic interest in architectural novelty. Three concrete deployment patterns make the Transformer's inference characteristics particularly painful:

On-device and edge deployment: Running LLMs on phones, laptops, or embedded devices requires both memory efficiency (devices have 8–16 GB total RAM, shared across the OS and other applications) and predictable latency (users expect real-time responses regardless of how long the conversation has been going). A Transformer's growing KV cache eventually exhausts device memory, and its per-step latency creep degrades user experience. RNN architectures, by maintaining a fixed-size hidden state regardless of context length, solve both problems: memory is constant and latency is constant per step.

High-throughput batch inference: In production serving environments, the primary cost driver is memory bandwidth rather than raw FLOPs. The KV cache must reside in GPU HBM (high-bandwidth memory), and as batch size or sequence length grows, the cache footprint limits how many requests can be processed simultaneously. The paper's speed and memory benchmarks (Figures 6 and 7) demonstrate that Finch uses 40% less memory than Flash Attention at equivalent sequence lengths — directly translating to higher throughput per GPU and lower per-token serving costs.

Unbounded context applications: Tasks like processing entire codebases, analyzing full-length books, or maintaining long-running agent conversations require models that can handle context lengths of 100K+ tokens. While techniques like sliding window attention (Mistral's approach) and Flash Attention (Dao et al., 2022) reduce the practical costs of Transformers, they do not fundamentally change the $O(N)$ memory scaling — they only improve the constant factors. An architecture with truly $O(1)$ per-token memory, like an RNN, imposes no fundamental limit on context length beyond what the hidden state can effectively represent.

Prior Approaches and Their Shortcomings

The paper situates itself within a broad landscape of attempts to reconcile the triangle of desiderata that has proven frustratingly elusive: high modeling quality, efficient training, and cheap inference.

Transformers with approximate attention. A large body of work has attempted to reduce the Transformer's quadratic cost by approximating or sparsifying self-attention. Representative approaches include:

Sparse attention patterns (Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020): Restrict each token to attend only to a subset of previous tokens (e.g., a sliding window, dilated windows, or global+local combinations). These reduce inference time and memory but introduce inductive biases that may not align with task requirements — the model cannot attend to arbitrary positions, limiting its ability to form long-range dependencies.
Low-rank approximations (Wang et al., 2020; Xiong et al., 2021): Project the attention matrix into a lower-dimensional space, reducing the $O(N^2)$ attention computation to $O(Nk)$ for some rank $k$ . While effective in theory, these often degrade modeling quality because the low-rank assumption does not always hold for real attention patterns.
Kernel-based approximations (Choromanski et al., 2020; Katharopoulos et al., 2020a): Replace the softmax with a kernel function that admits a linearized form, enabling the associative reordering $\phi(Q)(\phi(K)^T V)$ . This achieves linear time in principle, but in practice the choice of kernel function significantly impacts performance, and many kernel-based approaches underperform standard attention on challenging tasks.

The paper's Table 1 provides a comparative analysis that highlights the limitations of these approaches: while many achieve sub-quadratic or even linear training time, most either sacrifice parallel training (LSTM-style RNNs), require $O(N \log N)$ training time (H3/S4, Hyena), or retain $O(N)$ inference memory (Hyena, linear Transformer variants without careful implementation). The paper frames RWKV-4 as the first RNN to genuinely rival Transformer performance while maintaining $O(1)$ inference time and memory and providing fully parallelizable $O(N)$ training — a combination that prior work had not achieved.

State Space Models (SSMs). A separate lineage of work, originating in signal processing, models sequences via a continuous-time state space representation $\dot{h}(t) = Ah(t) + Bx(t)$ , $y(t) = Ch(t)$ . Early SSMs (Gu et al., 2020, 2022) achieved strong results on long-range sequence modeling benchmarks but were historically computed via $O(N \log N)$ long convolutions, making them expensive to train for very long sequences. Recent work (Smith et al., 2023; Gu & Dao, 2023) introduced associative scan techniques that enable parallelized $O(N)$ training, and data-dependent variants (Mamba) added input-dependent $B$ and $C$ matrices (and later $A$ matrices) that allow the model to selectively filter information based on input content — a crucial capability that static SSMs lacked for tasks like language modeling, where different tokens need different retention periods (e.g., function names should persist for the duration of a function; filler words should be forgotten quickly).

The paper explicitly positions Finch's dynamic recurrence as conceptually related to these data-dependent SSMs, particularly Mamba's selective state mechanism (Section 2):

"A new class of SSMs has also emerged concurrently with our work (Katsch, 2023; Gu & Dao, 2023) that feature data-dependent $A$ and $B$ terms, which function similarly to the data-dependent dynamic recurrence used in Finch."

However, the paper distinguishes Finch by how the data-dependence is implemented: through LoRA-augmented learned vectors rather than through a discretized continuous-time system. This architectural difference produces different computational characteristics, as demonstrated in the associative recall experiments (Section 8.2) and speed benchmarks (Section 9), where Finch and Mamba exhibit distinct scaling behaviors despite their superficial similarity as data-dependent RNNs.

Linear Attention and the AFT Precedent. The paper traces a direct intellectual lineage from prior work that the reader needs to understand to appreciate the architectural innovations. Linear Attention (Katharopoulos et al., 2020a) observed that if the softmax in standard attention is replaced with a separable kernel — specifically $\text{softmax}(QK^T)V \approx \phi(Q)(\phi(K)^T V)$ where $\phi$ is a non-negative feature map — then the computation can be reordered via associativity. The term $\phi(K)^T V$ can be accumulated into a recurrent state of fixed size, enabling $O(1)$ per-step inference while maintaining parallelizable training. However, naive linear attention consistently underperformed standard attention because the kernel approximation introduced errors that compounded over long sequences.

The Attention Free Transformer (AFT) (Zhai et al., 2021) addressed some of these quality issues by introducing learned pairwise positional biases $w_{i,t}$ that provided the model with explicit information about the relative positions of tokens. The paper's Equation 1 reproduces AFT's formulation, which replaces the query-key interaction with a simpler element-wise gating mechanism while retaining learned positional biases as a proxy for attention weights:

$\text{AFTAtt}_t = \sigma_q(q_t) \odot \frac{\sum_{i=1}^t \exp(k_i + w_{i,t}) \odot v_i}{\sum_{i=1}^t \exp(k_i + w_{i,t})}$

This equation is worth unpacking because RWKV's entire design philosophy flows from it. Unlike attention, which computes query-key dot products, AFT uses a learned per-position bias $w_{i,t}$ that depends only on positions $i$ and $t$ . The key $k_i$ acts as a content-dependent modifier on this positional bias. The result is that each position's value $v_i$ contributes to the output with a weight determined by both its content ( $k_i$ ) and its position ( $w_{i,t}$ ), but without the quadratic $QK^T$ operation. This is the key insight that enables linear complexity while retaining some form of content-position interaction.

RWKV-4's reformulation and its remaining limitations. RWKV-4 (Peng et al., 2023) refined AFT's formulation in several critical ways, as shown in the paper's Equation 2:

$\text{wkv}_t = \frac{\sum_{i=1}^{t-1} \exp(-(t-1-i)w + k_i) \odot v_i + \exp(u + k_t) \odot v_t}{\sum_{i=1}^{t-1} \exp(-(t-1-i)w + k_i) + \exp(u + k_t)}$

The key changes from AFT are:

Channel-wise decay $w$ replaces the full pairwise positional bias matrix. Instead of learning a separate bias for every $(i, t)$ pair, RWKV-4 learns a single decay rate per channel that geometrically diminishes the contribution of past tokens: token $i$ is weighted by $\exp(-(t-1-i)w)$ when computing the output at time $t$ .
Token shift (a learnable linear interpolation between $x_t$ and $x_{t-1}$ ) is applied to the inputs before computing $r$ , $k$ , $v$ , allowing the model to blend current and previous token information in a channel-specific way.
A bonus term $u$ gives the current token's $k_t \odot v_t$ a separate, learnable boost, allowing it to be treated differently from the decaying sum of past tokens.

These innovations made RWKV-4 the first RNN to genuinely compete with Transformers on language modeling benchmarks. However, the paper identifies specific architectural limitations that motivate the Eagle and Finch upgrades:

Scalar-valued states: In RWKV-4, the recurrent state is a vector of dimension $D$ . Each channel independently decays its accumulated value. However, the interactions between different channels within the state are limited — there is no mechanism for one channel's value to influence another's. Matrix-valued states (Eagle's innovation) introduce per-head $K^T V \in \mathbb{R}^{(D/h) \times (D/h)}$ matrices, where each entry $(i, j)$ represents the interaction between the $i$ -th dimension of the key and the $j$ -th dimension of the value. This fundamentally increases the expressivity of the state: a vector state of size $D$ can store $D$ independent scalar accumulations, while a matrix state of size $(D/h) \times (D/h)$ can store $(D/h)^2$ interaction terms. For typical head sizes of 64, this means the state's representational capacity grows quadratically with heads.
Static decay rates: RWKV-4's $w$ is a fixed learned vector — the same decay pattern applies to every token regardless of content. This means the model cannot decide to retain a particular piece of information longer based on its importance. Finch's dynamic recurrence introduces $w_t$ , a data-dependent decay rate that can vary at each time step based on the input. Intuitively, when the model encounters a noun that will be referenced later, it can set low decay rates for the channels representing that noun, effectively "remembering" it. When it encounters a filler word, it can set high decay rates, quickly forgetting. This content-dependent memory management is strikingly similar to the theory behind induction heads in Transformers, where attention heads learn to copy information from earlier positions based on matching query-key patterns.
Sigmoid-activated receptance: RWKV-4 applies $\sigma(r_t)$ as a gate on the final attention output. Eagle replaces this with a SiLU-gated mechanism (Equation 7: $\text{SiLU}(g_t) \odot \text{LayerNorm}(r_t \cdot \text{wkv}_t)$ ) where $g_t$ is a separate learned gate and the receptance $r_t$ acts as a query vector multiplying the state directly, without the sigmoid bottleneck.

How This Paper Positions Itself

The paper positions Eagle and Finch not as competitors to Transformers in the absolute sense — Mistral-7B still leads on English benchmarks — but as advancing the state of the art for RNN-based architectures to the point where the performance gap is small enough that the inference advantages become decisive in many practical scenarios.

This positioning is important because it reframes the evaluation criteria. A Transformer researcher might look at Table 4 and note that Eagle-7B trails Mistral-7B by 4.3 percentage points on average English benchmark accuracy. The RWKV authors would respond that this comparison misses the point: the relevant comparison is not "which architecture wins at a given parameter count" but rather "given a specific deployment constraint — say, serving 1000 concurrent users with 32K context windows on a single A100 — which architecture delivers the best quality?" Under memory constraints that make Transformers' KV caches infeasible, the comparison is between Eagle/Finch and other $O(1)$ -memory architectures (Mamba, HGRN, earlier RWKV versions), not between Eagle and Transformers with unlimited memory.

The paper also positions itself within the open science movement, emphasizing in Table 2 that Eagle-7B is the only 7B+ model with publicly available weights, training code, inference code, and training dataset — all under Apache 2.0 license. This is in contrast to models like Mistral-7B (weights available, training data undisclosed), Llama-2 (weights available under a restrictive license, training data undisclosed), and GPT-4 (nothing disclosed). The authors implicitly argue that architectural innovation requires full reproducibility — without knowing what data a model was trained on, it is impossible to determine whether performance differences stem from architectural choices or data quality.

Finally, the paper introduces Finch at a specific moment in the evolution of RNN architectures. Concurrent work — Mamba, GLA, Griffin, HGRN — had demonstrated that data-dependent recurrences could close much of the remaining gap to Transformers. Finch enters this conversation with a specific hypothesis: that LoRA-augmented token shift and data-dependent decay rates, applied within the RWKV framework of channel-wise geometric decay and token mixing, can match or exceed competing data-dependent RNNs while maintaining RWKV's implementation simplicity and parallel training efficiency. The associative recall experiments (Figure 4) are the key evidence for this claim, showing Finch achieving near-perfect accuracy on MQAR tasks where previous non-Transformer architectures, including RWKV-4, struggled.

The paper does not claim to have definitively demonstrated that RNNs are "better" than Transformers. Rather, it claims — and the experimental results support — that the architectural innovations in Eagle and Finch represent a significant step toward RNN-based LLMs that are competitive enough with Transformers on quality that their inference cost advantages make them the pragmatic choice for a growing range of deployment scenarios.

3. Technical Approach

This is primarily an architectural design paper whose core idea is that two specific innovations — matrix-valued recurrent states and data-dependent dynamic decay — can lift the performance of an RNN-based language model from "competitive with Transformers under ideal conditions" (RWKV-4) to "competitive broadly" while preserving the $O(1)$ per-token inference cost that makes RNNs attractive for deployment. The paper does not introduce a new training algorithm or objective; rather, it evolves the RWKV-4 architecture through two sequential refinements, each validated at scale, and measures the resulting models against contemporary Transformers and other sub-quadratic architectures.

3.1 Reader Orientation

What is being built: A sequence of two RNN-based language model architectures — Eagle (RWKV-5) and Finch (RWKV-6) — that process text autoregressively by maintaining a fixed-size hidden state that is updated at each time step, enabling inference with $O(1)$ memory and $O(1)$ time per token regardless of context length.

What problem it solves and the shape of the solution: The problem is that existing RNN architectures, including RWKV-4, underperform Transformers on language modeling benchmarks because their recurrent states lack the representational richness to capture the complex token interactions that attention mechanisms provide. The solution is two architectural upgrades applied sequentially: first, expand the state from vector-valued to matrix-valued (Eagle), giving each head a $(D/h) \times (D/h)$ state that explicitly captures key-value interactions across feature dimensions; second, make the decay rates data-dependent (Finch), so the model can decide at each time step how much of the past to retain based on the content of the current and previous tokens, rather than using a fixed learned decay schedule.

3.2 Big-Picture Architecture (Diagram in Words)

The overall architecture follows the same block-stacked residual structure as RWKV-4 and Transformers, but the internal mechanics of the time-mixing sub-layer differ fundamentally:

Token Shift module: Before computing the main operations, each input token $x_t$ is blended with the previous token $x_{t-1}$ via a learned linear interpolation (Eagle) or a data-dependent interpolation (Finch). This produces the "shifted" inputs for the current time step. The module is applied independently to the inputs that will become $r$ , $k$ , $v$ , $g$ , and (in Finch) $d$ (the raw decay input).
Time-Mixing sub-layer: This is the RNN core, replacing the self-attention of Transformers. It takes the token-shifted inputs, computes key-value outer products $k_t^T v_t$ , accumulates them into a multiply-headed matrix-valued state $s$ with exponential decay, applies a learned per-channel boost $u$ to the current token's contribution, and then uses the receptance vector $r_t$ as a query to read from the combined state. The output passes through a SiLU gate and LayerNorm before projection.
Channel-Mixing sub-layer: This is the analog of the Transformer's feed-forward network. It takes the same input $x'_t$ , applies token shift, projects up to a hidden dimension of $3.5D$ , applies squared ReLU activation, projects back down to $D$ , and applies a sigmoid gate. This sub-layer is identical in Eagle and Finch (and nearly identical to RWKV-4), handling per-token non-linear transformations without recurrence.
Residual connections and Pre-LayerNorm: Each sub-layer is wrapped with a residual connection and preceded by LayerNorm, following the standard Pre-LN Transformer convention.
RWKV World Tokenizer: A Trie-based greedy-matching tokenizer with a manually curated vocabulary of 65,536 tokens, designed to give underrepresented languages more balanced token coverage than BPE tokenizers trained on English-heavy corpora.

Information flows as: raw text → RWKV World Tokenizer → token IDs → embedding layer → sequence of stacked residual blocks (each containing Time-Mixing → Channel-Mixing) → final LayerNorm → output projection to vocabulary logits → next-token prediction.

3.3 Roadmap for the Deep Dive

The technical explanation proceeds in the following order, which build from the shared foundation up to the most novel features:

First, the Token Shift mechanism, since it is the first operation applied in every block and its evolution from Eagle to Finch introduces the data-dependence pattern that defines Finch's contributions. Understanding why token shift exists and what it accomplishes is essential background for every subsequent equation.
Second, Eagle Time Mixing, which introduces matrix-valued states, the reformulated receptance, SiLU gating, and LayerNorm over heads. This is the core computational engine of Eagle and the foundation upon which Finch builds.
Third, Finch's data-dependent innovations, beginning with the data-dependent token shift (ddlerp) and the LoRA augmentation mechanism, then the data-dependent decay $w_t$ . These are the paper's most novel contributions and the technical heart of Finch.
Fourth, Channel Mixing, which is shared across Eagle and Finch. Explained briefly since it is identical to RWKV-4 except for a reduced hidden dimension.
Fifth, the RWKV World Tokenizer, since tokenization choices affect multilingual performance and the paper makes specific claims about the tokenizer's design philosophy.

The rationale for this ordering is that each component depends on the previous one: Token Shift feeds into Time Mixing; Eagle Time Mixing is the baseline that Finch's dynamic recurrence modifies; Channel Mixing is independent of the time-mixing innovations; and the tokenizer is an orthogonal data-preprocessing choice.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architectural evolution paper that takes the RWKV-4 design and makes two successive refinements, each validated through pretraining at scale. The core idea is that RWKV-4's primary limitations — scalar-valued states and static decay rates — can be addressed through specific, carefully motivated design changes that preserve the $O(1)$ inference property while substantially improving benchmark performance.

The analysis below walks through each architectural mechanism in detail, explaining not just what each equation computes, but why that particular form was chosen over alternatives, what physical/computational process it models, and how it connects to the broader design philosophy of the RWKV family.

Token Shift: Learning How Much Past vs. Present to Use

The Token Shift mechanism is the first operation in every time-mixing and channel-mixing block. It has no direct analog in standard Transformers — the closest comparison would be a learned 1D causal convolution with kernel size 2, applied channel-wise before any attention or feed-forward computation. Understanding token shift is critical because it establishes a design pattern (learnable linear interpolation between current and previous token) that Finch later makes data-dependent.

The basic lerp operation (Eagle and earlier)

The paper defines the linear interpolation used in RWKV-4 and Eagle Token Shift via Equation 3:

$\text{lerp}_{\square}(a, b) = a + (b - a) \odot \mu_{\square}$

where $a$ and $b$ are vectors of dimension $D$ (or $D/h$ per head in multi-headed contexts), $\mu_{\square} \in \mathbb{R}^D$ is a learnable vector specific to the operation subscript $\square$ , and $\odot$ denotes element-wise (Hadamard) multiplication.

When used as $\text{lerp}_{\square}(x_t, x_{t-1})$ , the operation computes a channel-wise blend between the current token's representation and the previous token's representation. Each channel $i$ independently mixes the two inputs according to its learned parameter $\mu_{\square}[i]$ .

What this operation computes operationally: For a given vector channel, if $\mu = 0$ , the output is purely $x_t$ (current token). If $\mu = 1$ , the output is purely $x_{t-1}$ (previous token). If $\mu = 0.5$ , the output is the mean. Critically, $\mu$ is learned per channel, so the model can learn that some feature dimensions should emphasize the current token (e.g., for detecting new entities) while others should emphasize the previous token (e.g., for maintaining syntactic continuity). The subtraction $(b-a)$ isolates the change between consecutive tokens, and $\mu$ controls how much of that change to incorporate into the output.

Why this form: The lerp formulation provides three properties that matter for the RWKV design:

Channel-wise independence: Each feature dimension makes its own decision about the current-vs-past tradeoff. This is important because different semantic features have different natural timescales — part-of-speech information might change with every token, while topic information might persist for paragraphs. Channel-wise $\mu$ allows a single token shift operation to simultaneously handle features at different timescales.
Smooth interpolation rather than discrete gating: A discrete choice between $x_t$ and $x_{t-1}$ (e.g., via a sigmoid gate) would prevent gradients from flowing through the "deselected" branch. The lerp formulation always includes contributions from both tokens, with $\mu$ controlling the relative weight. This means the model can learn to partially attend to the past even when primarily focused on the present, and gradients always flow through both paths.
Computational minimalism: The lerp requires only one element-wise subtraction, one element-wise multiplication, and one element-wise addition — three $O(D)$ operations total. This is crucial because the token shift is applied five times per time-mixing block (for $r$ , $k$ , $v$ , $g$ , and $d$ in Finch) — any increase in token shift cost would multiply across all these applications.

What token shift enables: The paper explicitly states that token shift "makes it possible to form induction heads within a single layer since even a single head can directly accumulate both past and current token data into separate subspaces within these vectors." Induction heads — the mechanism by which Transformers perform in-context learning by matching a previous occurrence of a token and predicting what followed it — require the model to simultaneously represent "what token am I looking at now" and "what token was the previous one" so that pattern matching can occur. Without token shift, a single recurrent layer would only have access to the current token's embedding; the previous token's information would be buried in the recurrent state, mixed with all other past tokens. Token shift gives the layer direct access to both $x_t$ and $x_{t-1}$ simultaneously, enabling the kind of two-token pattern detection that induction heads perform.

Multiple independent shifts: The token shift is applied separately to each of $r$ , $k$ , $v$ , and $g$ (and $d$ in Finch), meaning there are different learned $\mu_r$ , $\mu_k$ , $\mu_v$ , $\mu_g$ (and $\mu_d$ ). This allows the model to use different current-vs-past blends for different roles: the key might emphasize the current token (to encode what is being stored), while the value might emphasize the previous token (to encode the context in which the storage occurs).

Eagle Time Mixing: Matrix-Valued States and Reformulated Receptance

Eagle Time Mixing is the core recurrent computation that replaces the Transformer's self-attention. It processes a sequence of tokens by maintaining a matrix-valued hidden state per head and updates it at each time step via a decay-based accumulation of key-value outer products. This section walks through each equation in the Eagle time-mixing block, explaining what it computes, why that specific form was chosen, and how it improves upon RWKV-4.

Step 1: Token-shifted projections (Equation 4)

$\square_t = \text{lerp}_{\square}(x_t, x_{t-1}) \mathbf{W}_{\square}, \quad \square \in \{r, k, v, g\}$

where $x_t \in \mathbb{R}^D$ is the input at time $t$ , $\text{lerp}_{\square}$ is the token shift operation defined in Equation 3 with learned parameter $\mu_{\square}$ , and $\mathbf{W}_{\square} \in \mathbb{R}^{D \times D}$ is a learned projection matrix (one for each of receptance $r$ , key $k$ , value $v$ , and gate $g$ ). All operations are computed per head, so effectively each $\mathbf{W}_{\square}$ projects from $D$ to $D/h$ for each head, but the paper elides the head index for notational simplicity.

What this computes: For each role ( $r$ , $k$ , $v$ , $g$ ), the operation first blends $x_t$ with $x_{t-1}$ using the role-specific token shift parameter $\mu_{\square}$ , then projects the blended vector through a learned linear transformation. The output is four vectors (per head) — $r_t$ , $k_t$ , $v_t$ , $g_t$ — each of dimension $D/h$ , representing the receptance (query analog), key (storage address analog), value (stored content analog), and gate (output modulation), respectively.

Why four separate projections: This mirrors the query-key-value decomposition in attention but adds an explicit gate $g_t$ . In attention, the query determines "what am I looking for," the key determines "what does each position offer," and the value determines "what content should be retrieved." Receptance $r_t$ plays the query role — it will multiply with the accumulated state to extract information. Key $k_t$ plays the addressing role — its outer product with $v_t$ determines where in the state matrix the value gets stored. Value $v_t$ is the content being stored. Gate $g_t$ is an additional learned filter applied after the state readout, allowing the model to suppress or amplify the time-mixing output on a per-channel basis.

Step 2: Decay rate parameterization (Equation 5)

$w = \exp(-\exp(\omega))$

where $\omega \in \mathbb{R}^{D/h}$ is the actual headwise trainable parameter, and $w \in (0, 1)^{D/h}$ is the derived per-channel decay rate.

What this computes: This is a parameterization trick. The model learns unconstrained real-valued parameters $\omega$ , but the double-exponential transformation $\exp(-\exp(\omega))$ maps any real value to the interval $(0, 1)$ . When $\omega$ is a large negative number, $-\exp(\omega)$ is close to zero, so $w \approx 1$ (slow decay — information persists). When $\omega$ is a large positive number, $-\exp(\omega)$ is a large negative number, so $w \approx 0$ (fast decay — information is quickly forgotten).

Why this form: The double-exponential parameterization addresses two practical concerns:

Guaranteed contraction: Because $w \in (0, 1)$ , the matrix $\text{diag}(w)$ is strictly a contraction mapping — its eigenvalues are all in $(0, 1)$ . This means that when the state is multiplied by $\text{diag}(w)$ at each time step (Equation 9), the contribution of past tokens geometrically decays to zero. Without this guarantee, the state could grow unboundedly, leading to numerical instability during long sequences.
Unconstrained optimization: The parameter $\omega$ can take any real value, meaning the optimizer can freely move it up or down without worrying about domain constraints. The transformation handles the mapping to $(0, 1)$ differentiably. This is the same pattern used in variational autoencoders for parameterizing standard deviations (softplus) and in many neural parameterizations where a parameter must be constrained to a specific range.
Channel-wise decay: Each channel has its own learned $\omega$ , and therefore its own $w$ . This means the model can learn that certain feature dimensions should retain information for hundreds of tokens (small $w$ ) while others should operate on a much shorter timescale (large $w$ ).

Step 3: The WKV attention computation (Equation 6)

$\mathbf{wkv}_t = \text{diag}(u) \cdot k_t^T \cdot v_t + \sum_{i=1}^{t-1} \text{diag}(w)^{t-1-i} \cdot k_i^T \cdot v_i \in \mathbb{R}^{(D/h) \times (D/h)}$

where $u \in \mathbb{R}^{D/h}$ is a learned per-channel boost parameter, $\text{diag}(u)$ is a diagonal matrix with $u$ on the diagonal, $k_t \in \mathbb{R}^{D/h}$ is the key vector at time $t$ , $v_t \in \mathbb{R}^{D/h}$ is the value vector, $w \in (0, 1)^{D/h}$ is the decay vector from Equation 5, and $\text{diag}(w)^{t-1-i}$ denotes the diagonal matrix whose $j$ -th diagonal entry is $w_j^{t-1-i}$ (the per-channel decay rate raised to the power of the time difference).

What this computes: The $\mathbf{wkv}_t$ matrix is the time-mixed state at position $t$ , encoding the entire history up to and including the current token in a fixed-size matrix representation. It is a sum of outer products $k_i^T v_i$ for all past positions $i$ , where each position's contribution is scaled by a per-channel decay factor that depends on how far in the past position $i$ is. The first term is the current token's contribution, scaled by a learned boost vector $u$ rather than a decay factor. The second term is the sum over all past tokens, where token $i$ (which occurred $t-1-i$ steps ago) has its outer product decayed element-wise by $w_j^{t-1-i}$ for each channel $j$ .

Specifically, in the outer product $k_i^T v_i$ , the entry at row $a$ , column $b$ is $k_i[a] \cdot v_i[b]$ . After decay by $\text{diag}(w)^{t-1-i}$ , the entry becomes $w_a^{t-1-i} \cdot k_i[a] \cdot v_i[b]$ . So each row of the state matrix decays at its own rate — row $a$ , which corresponds to the $a$ -th dimension of the key, decays by $w_a$ per time step. This means different key dimensions can have different memory timescales.

Why this form — the matrix-valued state: This is the most consequential change from RWKV-4. In RWKV-4, the state was a vector: $\sum_i \text{decay} \cdot k_i \odot v_i$ , where $\odot$ is element-wise multiplication. The state at each channel $j$ was a scalar $\sum_i \text{decay}_j \cdot k_i[j] \cdot v_i[j]$ — notice that channel $j$ of the state only interacted with channel $j$ of the key and value. There was no cross-channel interaction.

In Eagle, the state is a matrix: $\sum_i \text{decay} \cdot k_i^T v_i$ . The entry at $(a, b)$ stores interactions between key dimension $a$ and value dimension $b$ . When the receptance $r_t$ multiplies this matrix (Equation 7), it produces a vector whose $b$ -th component is $\sum_a r_t[a] \cdot \text{state}[a, b]$ . This means the receptance can query across key dimensions to retrieve value information — exactly the kind of cross-dimensional interaction that makes attention powerful. The matrix state has $(D/h)^2$ independent entries per head versus $D/h$ entries for a vector state, representing a substantial increase in representational capacity.

The matrix-valued state can also be understood as a memory bank where:

Each row corresponds to a "memory slot" indexed by the key dimensions
Each column corresponds to a "content dimension" of the stored values
Writing to memory: $k_i^T v_i$ adds the value $v_i$ to rows weighted by the key $k_i$ — dimensions of $k_i$ that are large get more of $v_i$ written to their corresponding rows
Reading from memory: $r_t \cdot \text{state}$ computes, for each content dimension, a weighted sum over all memory slots, where the weights are given by the receptance $r_t$

This is functionally similar to linear attention $\phi(Q)(\phi(K)^T V)$ , but with exponential decay replacing the kernel function $\phi$ , and with the learned boost $u$ giving the current token special treatment.

The role of the boost term $u$ : Without $u$ , the current token's contribution would be treated identically to past tokens — it would enter the state and only influence the output through the receptance query. The vector $u$ gives the current token an additional, learnable per-channel boost. If a channel's $u$ is large, the current token's key-value pair dominates the output; if $u$ is small, the model relies more on the accumulated history. This allows the model to learn a different treatment for "what is happening right now" versus "what has been building up over time."

Step 4: The recurrent formulation (Equations 8-9)

The paper provides an equivalent recurrent formulation that makes the per-step update explicit:

$\mathbf{wkv}' = s + \text{diag}(u) \cdot k^T \cdot v$

$s' = \text{diag}(w) \cdot s + k^T \cdot v$

where $s \in \mathbb{R}^{(D/h) \times (D/h)}$ is the recurrent state carried from the previous time step, and $s'$ is the new state to be passed to the next time step.

What this computes: At each time step, two things happen:

State update (Equation 9): The old state $s$ is decayed element-wise by $w$ (different decay rates for different rows), and the current token's key-value outer product $k^T v$ is added. This is exactly an exponentially-weighted moving average of $k_i^T v_i$ over time, with per-channel decay rates.
Output computation (Equation 8): The current token's key-value outer product, boosted by $u$ , is added to the (pre-update) state to produce the $\mathbf{wkv}$ matrix for the current time step. Note that this uses the old state $s$ (from before incorporating the current token), plus the current token's contribution with the boost. This ordering — reading from the state before updating it — is standard RNN practice and prevents the current token from attending to itself (beyond the explicit boost term).

Why this recurrent form matters for inference: Equations 8-9 require $O((D/h)^2)$ operations per time step — the cost of the outer product $k^T v$ and the matrix multiplication $\text{diag}(w) \cdot s$ . Critically, this cost is independent of sequence length. At inference time, the model processes one token at a time, updating the fixed-size state $s$ and producing a fixed-size output, with no need to store or attend to any previous tokens beyond what is compressed in $s$ . This is what gives Eagle (and Finch) the $O(1)$ per-token inference property.

Step 5: Output gating and projection (Equation 7)

$o_t = \text{concat}(\text{SiLU}(g_t) \odot \text{LayerNorm}(r_t \cdot \mathbf{wkv}_t)) \mathbf{W}_o \in \mathbb{R}^D$

where $g_t \in \mathbb{R}^{D/h}$ is the gate vector from Equation 4, $r_t \in \mathbb{R}^{D/h}$ is the receptance vector, $\mathbf{wkv}_t \in \mathbb{R}^{(D/h) \times (D/h)}$ is the time-mixed state from Equation 6, $\text{LayerNorm}$ operates on each of the $h$ heads separately (equivalent to GroupNorm with $h$ groups), $\text{concat}$ concatenates the outputs across all heads, and $\mathbf{W}_o \in \mathbb{R}^{D \times D}$ is the output projection matrix.

What this computes — step by step:

Receptance query ( $r_t \cdot \mathbf{wkv}_t$ ): The receptance vector $r_t$ (dimension $D/h$ ) multiplies the matrix state $\mathbf{wkv}_t$ (dimensions $(D/h) \times (D/h)$ ), producing a vector of dimension $D/h$ . This is the "read" operation: for each column (value dimension) of the state, compute a weighted sum over rows (key dimensions), where the weights are given by $r_t$ . The resulting vector is a content-dependent summary of the accumulated history.
LayerNorm: Each head's output vector is LayerNorm-ed independently. This normalizes across the feature dimensions within each head, preventing any head from dominating due to scale and stabilizing training. The paper notes this is equivalent to GroupNorm with $h$ groups. This is not present in RWKV-4 and addresses the fact that matrix-valued states can have much larger dynamic range than vector states.
SiLU gating ( $\text{SiLU}(g_t) \odot \dots$ ): The gate vector $g_t$ is passed through the SiLU (Sigmoid Linear Unit) activation — $\text{SiLU}(x) = x \cdot \sigma(x)$ — and multiplied element-wise with the LayerNorm-ed output. This replaces RWKV-4's approach of applying $\sigma(r_t)$ as a gate. SiLU gating has two advantages: (a) it separates the gating mechanism from the receptance, so $r_t$ can focus purely on querying the state while $g_t$ handles output modulation; (b) SiLU is a smooth, non-monotonic function that can both amplify and suppress (unlike sigmoid which only suppresses to $[0, 1]$ ), giving the model more flexibility in output scaling.
Concatenation and output projection: The per-head outputs are concatenated back into a vector of dimension $D$ (since there are $h$ heads, each producing $D/h$ dimensions), and the concatenated vector is projected through $\mathbf{W}_o$ to produce the final time-mixing output $o_t \in \mathbb{R}^D$ . This projection allows cross-head interaction, since heads operate independently until this point.

Why this gating change matters: In RWKV-4, the receptance vector $r_t$ was passed through a sigmoid and used as both the query and the gate. This conflated two distinct functions: determining what to read from the state (the query role) and determining how much of the output to let through (the gate role). Eagle separates these into $r_t$ (query, no activation) and $g_t$ (gate, SiLU-activated). The SiLU gate can take values greater than 1 (unlike sigmoid which is bounded at 1), allowing the model to amplify important signals beyond the range of the raw state readout.

Channel Mixing: The Feed-Forward Analog

Channel Mixing is the second sub-layer in each residual block and is identical in Eagle and Finch, with only a minor dimensionality change from RWKV-4. It plays the same architectural role as the feed-forward network in Transformers: per-token non-linear transformation without recurrence.

The Channel Mixing equations (Equations 10-13) are restated from RWKV-4 for notational consistency:

Step 1: Token-shifted projections (Equations 10-11)

$r'_t = \text{lerp}_{r'}(x'_t, x'_{t-1}) \mathbf{W}_{r'} \in \mathbb{R}^D$

$k'_t = \text{lerp}_{k'}(x'_t, x'_{t-1}) \mathbf{W}_{k'} \in \mathbb{R}^{3.5D}$

where $x'_t \in \mathbb{R}^D$ is the input to the Channel Mixing sub-layer (the output of the preceding Time Mixing sub-layer plus residual), $\mathbf{W}_{r'} \in \mathbb{R}^{D \times D}$ projects to a dimension- $D$ receptance vector, and $\mathbf{W}_{k'} \in \mathbb{R}^{D \times 3.5D}$ projects to a dimension- $3.5D$ key vector.

What's different from RWKV-4: The hidden dimension is reduced from $4D$ to $3.5D$ . The paper explains this explicitly: "This reduction accounts for new gating weights in Eagle Time Mixing to ensure an equi-parameter relation with the prior model at the same number of layers and embedding dimension." Since Eagle Time Mixing adds new parameters (the separate gate $g_t$ and its projection matrix), the Channel Mixing hidden dimension is reduced to keep the total parameter count comparable when comparing architectures at the same layer count and embedding dimension. This is a practical engineering choice, not a statement that $3.5D$ is inherently better than $4D$ .

Step 2: Gated activation (Equations 12-13)

$v'_t = \text{ReLU}(k'_t)^2 \mathbf{W}_{v'} \in \mathbb{R}^D$

$o'_t = \sigma(r'_t) \odot v'_t \in \mathbb{R}^D$

where $\text{ReLU}(k'_t)^2$ applies ReLU activation followed by element-wise squaring, and $\mathbf{W}_{v'} \in \mathbb{R}^{3.5D \times D}$ projects back to the model dimension.

What this computes: This is a gated linear unit with squared ReLU activation. The $3.5D$ -dimensional key $k'_t$ is passed through $\text{ReLU}(\cdot)^2$ — which zeros out negative values and squares positive values — creating a sparse, non-negative representation. This is then projected down to $D$ dimensions via $\mathbf{W}_{v'}$ . The $D$ -dimensional receptance $r'_t$ is passed through a sigmoid to produce per-channel gate values in $(0, 1)$ , which are element-wise multiplied with $v'_t$ .

Why squared ReLU: The paper inherits this from RWKV-4 without further justification, but the effect is to make the activation sparser and more sharply peaked than standard ReLU. Values near zero after ReLU stay near zero after squaring; values far from zero become even larger after squaring. This creates a "sharpening" effect that may help the model make more decisive feature selections.

The residual connection: The paper notes that each sub-layer is wrapped with a residual connection following the Pre-LN convention — LayerNorm is applied to the input before the sub-layer, and the sub-layer's output is added to its input. This means $x_{t+1} = x_t + \text{TimeMixing}(\text{LayerNorm}(x_t))$ and similarly for Channel Mixing.

Finch: Data-Dependent Token Shift and Dynamic Decay

Finch extends Eagle by making two previously static mechanisms data-dependent: the token shift interpolation weights and the per-channel decay rates. Both changes are implemented through a LoRA-inspired augmentation pattern that the paper uses consistently. This section explains the LoRA mechanism first, then shows how it is applied to token shift and decay respectively.

The LoRA augmentation pattern (Equation 14)

$\text{lor}_{\square}(x) = \lambda_{\square} + \tanh(x \mathbf{A}_{\square}) \mathbf{B}_{\square}$

where $x \in \mathbb{R}^D$ is an input vector, $\lambda_{\square} \in \mathbb{R}^D$ is a learned base value (analogous to the static $\mu$ parameters in Eagle), $\mathbf{A}_{\square} \in \mathbb{R}^{D \times 32}$ is a learned down-projection matrix, $\mathbf{B}_{\square} \in \mathbb{R}^{32 \times D}$ is a learned up-projection matrix, and $\tanh$ is the hyperbolic tangent activation applied element-wise.

What this computes: This function takes a learned static vector $\lambda_{\square}$ and adds a data-dependent offset to it. The offset is computed by projecting the input $x$ through a bottleneck: first down to 32 dimensions via $\mathbf{A}_{\square}$ , then through $\tanh$ (which squashes values to $[-1, 1]$ ), then back up to $D$ dimensions via $\mathbf{B}_{\square}$ . The bottleneck dimension of 32 is fixed (64 for $\omega$ specifically), making this a low-rank augmentation — the $D \times 32$ and $32 \times D$ matrices contain $2 \cdot 32D$ parameters, which is much smaller than a full $D \times D$ matrix (which would be $D^2$ ).

Why this pattern — the LoRA inspiration: The paper explicitly references LoRA (Low Rank Adaptation, Hu et al., 2022), which was originally developed for fine-tuning: instead of updating a full weight matrix, LoRA learns a low-rank update $BA$ where $B$ and $A$ are small. Finch inverts this idea — instead of using LoRA to fine-tune a frozen model, Finch uses the LoRA structure during pretraining to inexpensively make learned vectors context-dependent. A full $D \times D$ weight matrix would add $D^2$ parameters per augmentation; the LoRA approach adds only $2 \cdot 32D$ parameters. For $D = 4096$ , this is roughly 260K parameters versus 16.8M — a ~65× reduction.

Where $\tanh$ matters: The $\tanh$ activation bounds the bottleneck representation to $[-1, 1]$ , which means the offset added to $\lambda_{\square}$ is bounded. This prevents the data-dependent component from drowning out the learned base value. Without $\tanh$ , the offset could be arbitrarily large for unusual inputs, potentially destabilizing the model.

Data-dependent token shift — ddlerp (Equation 15)

$\text{ddlerp}_{\square}(a, b) = a + (b - a) \odot \text{lor}_{\square}(a + (b - a) \odot \mu_x)$

where $a$ and $b$ are the vectors to interpolate (typically $x_t$ and $x_{t-1}$ ), $\mu_x \in \mathbb{R}^D$ is a learned vector controlling a preliminary static blend (analogous to Eagle's $\mu$ ), and $\text{lor}_{\square}$ is the LoRA augmentation from Equation 14.

What this computes — operationally:

First, compute a preliminary blend: $a + (b - a) \odot \mu_x$ , which is exactly the Eagle-style static lerp between $a$ and $b$ using $\mu_x$ .
Pass this blended vector through the LoRA function $\text{lor}_{\square}$ to produce a data-dependent interpolation weight vector.
Use this data-dependent weight vector (instead of a static $\mu$ ) to perform the final interpolation between $a$ and $b$ .

The result is that the interpolation weight for each channel now depends on the content of the blended input. If a particular pattern in the data suggests that the previous token's information is important for this channel, the LoRA mechanism can output a value closer to 1 (favoring $b$ , the previous token). If the pattern suggests the current token is sufficient, it can output a value closer to 0 (favoring $a$ ).

Why this form — second-order token shift: The Eagle token shift uses a single static parameter $\mu$ per channel. Finch's ddlerp uses $\mu_x$ to compute a static preliminary blend, then modulates that blend based on the content of the blend itself. The paper describes this as a "second-order variant of Token-Shifting, allowing each channel of $w_t$ to vary based on a mix of the current and prior tokens, with the mix itself determined by aspects of both tokens." The intuition is that the model first forms a rough combined representation (via $\mu_x$ ), then inspects that representation to decide how much of the past to incorporate. This is analogous to a two-step decision process: "what changed?" (the difference $b - a$ ) and then "given what changed, how much of the past matters?"

Finch Token Shift applied (Equation 16)

$\square_t = \text{ddlerp}_{\square}(x_t, x_{t-1}) W_{\square}, \quad \square \in \{r, k, v, g\}$

This is identical in structure to Eagle's Equation 4, but with $\text{lerp}$ replaced by $\text{ddlerp}$ (each with its own $\mu_x$ , $\lambda_{\square}$ , $\mathbf{A}_{\square}$ , and $\mathbf{B}_{\square}$ parameters). The result is that the inputs to the key, value, receptance, and gate projections are now blended in a context-dependent way.

Data-dependent decay (Equations 17-18)

$d_t = \text{lora}_d(\text{ddlerp}_d(x_t, x_{t-1}))$

$w_t = \exp(-\exp(d_t))$

where $\text{ddlerp}_d$ is the data-dependent token shift for the decay input, $\text{lora}_d$ is the LoRA augmentation applied to the output of ddlerp (using the same pattern as Equation 14 but with a doubled bottleneck size of 64: $\mathbf{A}_\omega \in \mathbb{R}^{D \times 64}$ , $\mathbf{B}_\omega \in \mathbb{R}^{64 \times D}$ ), and $w_t \in (0, 1)^{D/h}$ is the time-varying decay vector.

What this computes: This is the core innovation of Finch. In Eagle, $w$ is a static learned vector (Equation 5). In Finch, $w_t$ is computed dynamically at each time step as a function of the current and previous token. The computation chain is:

Blend $x_t$ and $x_{t-1}$ using the data-dependent token shift (ddlerp $_d$ ) to produce an intermediate representation.
Pass this representation through the LoRA augmentation ( $\text{lora}_d$ ) to produce a raw decay parameter $d_t$ — the $\omega$ analog.
Apply the same double-exponential transformation as Eagle (Equation 5) to map $d_t$ to $(0, 1)$ , producing $w_t$ .

The learned static component (the $\lambda_d$ inside $\text{lora}_d$ ) provides a baseline decay pattern, while the data-dependent component (the $\tanh(x A_d) B_d$ term) adds context-dependent modulation. This means the model can learn that certain linguistic patterns — for example, the start of a new sentence or the introduction of a named entity — should trigger lower decay rates (better retention) for certain channels, while other patterns — filler words, punctuation — should trigger higher decay rates.

Why a doubled bottleneck for decay: The paper notes that $\mathbf{A}_\omega \in \mathbb{R}^{D \times 64}$ , $\mathbf{B}_\omega \in \mathbb{R}^{64 \times D}$ uses a bottleneck of 64 rather than 32, and states that "future 7B and larger Finch models are expected to further increase the size of these weight matrices by double or more." The larger bottleneck for the decay computation reflects the fact that the decay rate directly controls the memory timescale — arguably the most important parameter in the architecture — and benefits from greater representational capacity.

Finch Time Mixing (Equations 19-22)

The time-mixing equations for Finch are structurally identical to Eagle's but with the static $w$ replaced by the time-varying $w_t$ . The key difference appears in how past tokens are decayed:

$\mathbf{wkv}_t = \text{diag}(u) \cdot k_t^\top \cdot v_t + \sum_{i=1}^{t-1} \text{diag}\left(\bigodot_{j=i+1}^{t-1} w_j\right) \cdot k_i^\top \cdot v_i$

where $\bigodot_{j=i+1}^{t-1} w_j$ denotes element-wise multiplication of all decay vectors $w_j$ for $j$ from $i+1$ to $t-1$ .

What this computes — the critical change: In Eagle, the decay factor for token $i$ at time $t$ was simply $w^{t-1-i}$ — the static decay rate raised to the power of the time difference. In Finch, the decay factor is the element-wise product of all intermediate decay vectors $w_{i+1}, w_{i+2}, \ldots, w_{t-1}$ . Each time step between when token $i$ was stored and the current time $t$ applies its own per-channel decay. If all $w_j$ are equal to some constant $w$ , this reduces to Eagle's formulation. But because each $w_j$ can be different, the effective decay between time $i$ and time $t$ depends on what happened in between.

This has a powerful intuitive interpretation: the model can "protect" important information by setting low decay rates in subsequent time steps when that information is likely to be needed, and can "flush" irrelevant information by setting high decay rates. For example, when processing a long paragraph about a specific topic, the model might set low decay rates for topic-relevant channels; when the paragraph ends and a new topic begins, it might raise decay rates to clear out the old topic.

Recurrent formulation (Equations 21-22):

$\mathbf{wkv}' = s + \text{diag}(u) \cdot k^\top \cdot v$

$s' = \text{diag}(w) \cdot s + k^\top \cdot v$

These are identical in form to Eagle's Equations 8-9, but $w$ in Equation 22 is now $w_t$ (time-varying) rather than a static vector. This means the state update at each time step applies a different decay pattern. The $O(1)$ per-step cost is preserved because $\text{diag}(w_t) \cdot s$ is still an element-wise (row-wise) scaling of the state matrix, not a full matrix multiplication.

The RWKV World Tokenizer: Vocabulary Engineering for Multilinguality

The RWKV World Tokenizer is a manually curated vocabulary of 65,536 tokens implemented via Trie-based greedy matching. It departs fundamentally from the standard BPE (Byte-Pair Encoding) approach used by most LLM tokenizers.

Vocabulary construction: Rather than learning merge rules from data, the vocabulary is assembled by merging and manually selecting tokens from five existing tokenizers: GPT-NeoX-20B, GPT2, OpenAI's cl100k_base (tiktoken), Llama2, and Bloom. The goal is to ensure that non-European languages (Chinese, Korean, Japanese, Arabic, etc.) receive adequate token coverage that a purely data-driven BPE tokenizer would not provide, since BPE merges tokens proportional to their frequency in the training corpus, which is dominated by English and European languages.

Vocabulary structure (65,536 tokens):

Token 0: The document boundary marker <EOS> / <SOS>.
Tokens 1-256: Byte encodings — token $k$ encodes byte $k-1$ . Tokens 1-128 correspond to standard ASCII characters. This means every byte sequence can be represented, providing a fallback for any text.
Tokens 257-65,529: Multi-byte UTF-8 tokens with minimum length 2 bytes — words, subwords, accented characters, CJK characters, Hangul, Hiragana, Katakana, emoji. Chinese characters are allocated in the range 10,250 to 18,493.
Tokens 65,530-65,535: Reserved for future use.

Tokenization algorithm: Trie-based greedy matching. A Trie (prefix tree) is built from the vocabulary. Encoding proceeds left-to-right through the input string: at each position, the longest prefix matching a vocabulary entry is selected as the next token, and the process advances past that match. This guarantees deterministic, fast tokenization — $O(L)$ where $L$ is the input string length, with the maximum match length bounded by the longest vocabulary entry.

Why Trie-based greedy matching over BPE: The paper explicitly states that the tokenizer is designed to "mitigate undue burden, which naive BPE and related methods cause, on minor languages." In a BPE tokenizer trained on an English-heavy corpus, common English words become single tokens while words in underrepresented languages may require many tokens to encode. For example, a Korean word might require 5-6 tokens in a BPE tokenizer trained on primarily English text, while "the" requires 1 token. The RWKV World Tokenizer manually allocates token slots to characters and subwords from underrepresented languages, ensuring more balanced token-to-semantic-unit ratios across languages. The Trie-based greedy matching is simpler and faster than BPE's merge-rule application, and does not require the training corpus and merge table that BPE needs.

4. Key Insights and Innovations

Innovation 1: Matrix-Valued States as a Principled Capacity Upgrade for Linear Recurrence

The dominant assumption in RNN design prior to Eagle was that the recurrent state should be a vector — a flat list of scalar accumulators, one per channel, each decaying independently. This assumption was so pervasive that it persisted across architectural paradigms: classical RNNs (LSTM, GRU), early linear attention implementations, state space models, and RWKV-4 all maintained vector-valued hidden states. The reasoning was straightforward: a vector state of size $D$ requires $O(D)$ memory and $O(D)$ computation per step, which is already the lower bound for any operation touching all dimensions. Adding capacity meant increasing $D$ .

Eagle's matrix-valued state breaks this assumption by observing that cross-channel interactions — the ability for one dimension of the key to influence how a different dimension of the value is stored and retrieved — are the core mechanism that makes attention powerful, and that vector states fundamentally cannot capture these interactions. In a vector state $s \in \mathbb{R}^D$ , channel $j$ accumulates $k[j] \cdot v[j]$ independently. There is no term involving $k[a] \cdot v[b]$ for $a \neq b$ . The vector state is, mathematically, a diagonal matrix state — only the diagonal entries of the $K^T V$ outer product are retained, and all off-diagonal interactions are discarded.

By upgrading to matrix-valued states $s \in \mathbb{R}^{(D/h) \times (D/h)}$ per head, Eagle captures the full outer product. For a head dimension of 64, this means the state contains 4,096 interaction terms where the vector state contained only 64. The per-step cost increases from $O(D/h)$ to $O((D/h)^2)$ , but the paper demonstrates that this tradeoff — quadratic cost in the head dimension rather than in the sequence length — is acceptable because head dimensions are typically small (64) and fixed regardless of context length.

Why this is conceptually distinctive: The paper reframes RNN capacity not as a question of "how many scalars can we store" but as "what algebraic structure does our state have?" A matrix state is not just a bigger vector — it supports fundamentally different operations. The receptance $r_t$ can query across key dimensions to retrieve value information: $r_t \cdot (K^T V)$ computes $\sum_a r_t[a] \cdot (k[a] \cdot v[b])$ for each output dimension $b$ , meaning the contribution of value dimension $b$ depends on the match between $r_t$ and all key dimensions simultaneously. This is the same "query-key compatibility determines value retrieval" pattern that defines attention, but realized through structured accumulation rather than pairwise token comparisons.

The Paper's Table 8 makes the algebraic progression explicit: RWKV-4's state is a scalar per channel, Eagle's is a matrix, and Finch's is a matrix with time-varying decay. Each step preserves the $D/h$ size of the vectors ( $k_t$ , $v_t$ , $r_t$ ) but changes the algebraic richness of what accumulates.

Comparison to prior work: Prior linear attention implementations (Katharopoulos et al., 2020a) also used matrix-valued states — $K^T V$ is inherent to the linear attention formulation — but these approaches treated the matrix state as an implementation detail of the kernel approximation. The contribution of Eagle is to recognize matrix-valued states as an architectural design choice separable from the kernel trick, and to build an entire RNN around them with specifically designed supporting mechanisms: the learned boost $u$ (giving the current token special treatment relative to the accumulated matrix), per-head LayerNorm (stabilizing the larger dynamic range of matrix operations), and SiLU gating (replacing the sigmoid bottleneck on receptance). The associative recall experiments (Figure 4) provide the key evidence: Eagle dramatically outperforms RWKV-4 on MQAR, a task specifically designed to require cross-dimensional information binding of the kind that matrix states enable but vector states cannot express.

Significance beyond performance: This is a fundamental reframing, not merely an incremental capacity upgrade. It establishes that the algebraic type of the recurrent state — vector vs. matrix vs. potentially higher-order tensor — is a first-class architectural hyperparameter with predictable effects on expressivity. Future work on RNN architectures can now ask not just "how big should the state be" but "what tensor shape should the state have, and what operations should it support?"

Evidence anchor: The progression from RWKV-4 to Eagle to Finch in Table 8 visually encodes this innovation, and the MQAR results in Figure 4 quantify its impact. Eagle's matrix state is the enabling mechanism that makes Finch's data-dependent decay effective — without the cross-dimensional interactions in the state, data-dependent decay would only modulate how quickly each independent channel forgets, not how channels interact.

Innovation 2: Dynamic Recurrence via Learned Modulation of Decay Rates

Prior to Finch, the dominant paradigm for recurrent architectures — including Eagle, RWKV-4, Mamba (at the time of this paper), and most SSMs — used static learned decay/forgetting rates. Each channel had a fixed parameter controlling how quickly it discounted past information, learned during training and then frozen at inference. This meant the model's "memory policy" was uniform across all inputs: a channel that decayed slowly would do so whether processing a crucial named entity or an irrelevant preposition.

Finch introduces data-dependent decay: at each time step, the vector $w_t$ that controls how much of the previous state is retained is computed as a function of the current and previous tokens (via the ddlerp and LoRA mechanisms described in Section 3). This means the model can, at inference time, decide: "this token is important — I should turn down the decay on channels encoding its information so it persists" or "this is just punctuation — decay can proceed at the baseline rate."

Why this is conceptually distinctive: The innovation is not the mechanism for making decay data-dependent (LoRA-augmented learned vectors) — that is an engineering choice that could be swapped out. The conceptual contribution is the diagnosis that static recurrence is the primary bottleneck preventing RNNs from matching Transformer performance on in-context learning and long-range dependency tasks. This diagnosis has specific theoretical content: Transformers' attention mechanism achieves content-dependent memory by design — each token's influence on future tokens depends on query-key compatibility, which is inherently input-dependent. An RNN with static decay cannot replicate this because its memory policy is input-independent. By making decay rates content-dependent, Finch closes this specific expressivity gap without sacrificing the $O(1)$ inference property.

The diagnostic precision matters: the paper is not saying "RNNs need to be more powerful in general." It is identifying a specific capability — content-dependent memory management — that Transformers possess, that prior RNNs lacked, and that is causally responsible for the performance gap on tasks like associative recall and in-context learning (where the model must remember and retrieve information from earlier in the sequence based on content matches). The MQAR experiments (Figure 4) validate this diagnosis directly: Finch achieves near-perfect accuracy on associative recall tasks where RWKV-4 and Eagle (with static decay) struggle, despite Eagle having matrix-valued states. The data-dependence of decay, not the matrix state itself, is what unlocks MQAR performance.

Comparison to prior work: The paper explicitly acknowledges concurrent work on data-dependent SSMs (Mamba, GateLoop) that introduced content-dependent $A$ and $B$ matrices. Finch's contribution is distinct in two ways:

The specific mechanism differs: Finch uses LoRA-augmented learned vectors within the RWKV framework of channel-wise geometric decay and token shift, rather than discretizing a continuous-time state space system. This matters because it means Finch's data-dependence operates on decay rates directly (as multiplicative factors on the state), not on the continuous-time dynamics matrices. The computational characteristics differ — Section 9 shows Finch using less memory than Mamba at equivalent sequence lengths — and the associative recall results show Finch outperforming Mamba on MQAR tasks, suggesting that the specific form of data-dependence matters.
The framing as memory management, not signal processing: SSMs originate from signal processing theory (the HiPPO framework for online function approximation), and their data-dependent variants are motivated by the need to selectively filter input signals. Finch frames data-dependent decay as content-aware memory management — the model learns to decide what to remember based on what it is reading. This framing connects directly to the cognitive science concept of adaptive forgetting (the idea that forgetting is not a failure of memory but an active, context-dependent process that prevents interference) and to the mechanistic interpretability literature on induction heads (where content-based retrieval depends on selective retention of key information).

Significance beyond performance: This innovation establishes a new axis for RNN design: the degree and form of data-dependence in the recurrence. Future architectures can ask not just "should recurrence be data-dependent" (Finch's answer: yes) but "which parameters should be data-dependent" (decay rates? the boost vector $u$ ? the token shift weights?) and "what form should the data-dependence take" (LoRA augmentation? direct linear projection? discretized continuous-time dynamics?). The paper's ablation in Appendix J showing that LoRA-augmented decay outperforms simpler parameterizations provides initial evidence on these questions, but the conceptual space opened is much larger than the specific mechanism.

Evidence anchor: Figure 4 (MQAR accuracy across architectures) is the primary evidence — Finch achieves dramatically higher MQAR accuracy than static-decay models at equivalent sequence lengths and model dimensions. Figure 5 (PG19 long-context loss) provides complementary evidence in a more naturalistic setting: Finch's loss continues dropping further into long sequences than Eagle's, consistent with the hypothesis that data-dependent decay enables better retention of relevant information over long contexts.

Innovation 3: Token Shift as a First-Class Architectural Primitive, and Its Evolution to Data-Dependence

The "token shift" operation — blending $x_t$ with $x_{t-1}$ via a learned interpolation before computing projections — is a seemingly minor detail that prior work might have dismissed as an implementation trick. The RWKV-4 paper introduced it alongside several other innovations, and it could easily be mistaken for a simple smoothing operation or a poor man's 1D convolution.

This paper elevates token shift to a first-class architectural primitive with specific, articulated functions, and then extends it to data-dependence in a way that reveals its deeper role. The key conceptual move is the paper's explicit claim (Section 4.1.1) that token shift "makes it possible to form induction heads within a single layer since even a single head can directly accumulate both past and current token data into separate subspaces within these vectors."

Why this is conceptually distinctive: This claim reframes token shift from a representational convenience to a necessary condition for in-context learning in recurrent architectures. Induction heads — the mechanism by which Transformers perform pattern completion ("saw A, then B; now saw A again; predict B") — require comparing the current token with the previous token to detect repetition and retrieve what followed the previous occurrence. In a Transformer, attention provides this comparison natively: the query at position $t$ can directly match against keys at all previous positions. In an RNN without token shift, the only information available at time $t$ is the current token embedding and the recursively accumulated state. The previous token $x_{t-1}$ is already mixed into the state along with all earlier tokens. Token shift ensures that $x_{t-1}$ is separately available in a controlled form, allowing the model to detect $[x_{t-1}, x_t]$ patterns without needing to isolate $x_{t-1}$ from the state.

The paper's extension of token shift to data-dependence (ddlerp in Finch) reveals an even deeper role: token shift is not just about making $x_{t-1}$ available at time $t$ , but about learning what aspects of $x_{t-1}$ versus $x_t$ are relevant in a context-dependent way. The data-dependent token shift in Equation 15 computes interpolation weights that depend on the content of the blended input. This means the model can decide, at each time step: "given the pattern of change between the previous and current token, should I emphasize the past or the present for this specific channel?" This is a form of content-based gating of information flow that operates before the recurrent state update — it filters what even enters the recurrent computation, rather than filtering what is retrieved from it.

Comparison to prior work: In Transformers, positional encoding and the attention mechanism handle the roles that token shift handles in RWKV. Positional encoding tells the model where tokens are; attention decides which tokens to combine. Token shift collapses both functions into a single, computationally cheap operation: the $x_t$ vs. $x_{t-1}$ blending provides relative position information (the model knows "this is what just happened" vs. "this is happening now"), and the learned per-channel $\mu$ parameters (or data-dependent LoRA weights in Finch) decide how to combine them. The computational cost is $O(D)$ per application, versus $O(N^2 D)$ for full attention or $O(ND)$ for linear attention — a difference in asymptotic complexity class. The paper does not claim token shift is as powerful as full attention; rather, it claims (and the benchmark results support) that token shift provides enough positional and comparative information to achieve competitive performance at dramatically lower cost.

The Finch ddlerp is also conceptually related to gating mechanisms in LSTMs and GRUs, but with a crucial difference: those architectures use learned gates that depend on the current input and the previous hidden state. Finch's ddlerp depends on the current and previous inputs directly, not on the recurrent state. This means the gating decision is based on surface-level token change patterns rather than on accumulated history — a simpler, more local computation that the paper argues is sufficient when combined with the richer matrix-valued state.

Significance beyond performance: Token shift represents a design philosophy for RNNs: provide the model with explicit, cheap access to local temporal differences, and let the recurrent state handle long-range dependencies. This philosophy contrasts with Transformers, which handle all temporal relationships — local and long-range — through the same expensive attention mechanism. It suggests a broader architectural principle: not all temporal interactions are equally complex, and architectures can be more efficient by using different mechanisms for different timescales. Token shift handles adjacent-token interactions; decay-based recurrence handles longer-range accumulation; and (in Finch) data-dependent decay provides content-based modulation of the long-range memory. This layered approach to temporal modeling is a conceptual contribution that could inform future architectures beyond the RWKV family.

Evidence anchor: The paper's architecture ablations (Appendix J, though specific numbers are not detailed in the provided text) presumably quantify the contribution of token shift. The primary conceptual evidence, however, is the explicit connection to induction heads (Section 4.1.1) and the demonstration that Finch models — with data-dependent token shift — achieve competitive in-context learning performance on benchmarks that require the kind of pattern detection token shift enables.

Innovation 4: The Diagnostic Use of Associative Recall as an Architectural Differentiator

The paper's use of the Multi-Query Associative Recall (MQAR) benchmark (Section 8.2, Figure 4) is not just another evaluation — it is a diagnostic tool that reveals why architectural differences matter in a way that aggregate benchmark scores cannot. This represents a methodological contribution to how architectures should be compared and analyzed.

What makes this distinctive: Aggregate benchmark scores (Tables 3 and 4) show that Finch outperforms Eagle, which outperforms RWKV-4, which outperforms Pythia. But these scores do not reveal which specific capability each architectural innovation enables. MQAR isolates a single capability — the ability to store key-value pairs in context and retrieve the correct value when the key reappears — that is known from mechanistic interpretability research (Elhage et al., 2021; Olsson et al., 2022) to be the core operation underlying in-context learning. By evaluating on MQAR across model dimensions and sequence lengths, the paper demonstrates:

Static decay with vector states (RWKV-4) fails catastrophically on MQAR: As sequence length increases — meaning more key-value pairs must be stored and distinguished — RWKV-4's accuracy drops sharply. This is explained by the vector state's inability to represent cross-dimensional key-value bindings: each channel independently accumulates, so there is no way to store "key A maps to value B" as a distinct association from "key C maps to value D" if the keys and values share feature dimensions.
Matrix-valued states (Eagle) substantially improve but do not fully solve MQAR: The matrix state can represent key-value bindings across dimensions, but with static decay, all stored associations decay at the same rate regardless of their importance. At longer sequence lengths, interference between stored associations still degrades performance.
Data-dependent decay (Finch) achieves near-perfect MQAR: By modulating decay rates based on content, the model can selectively retain associations that are likely to be queried and rapidly forget those that are irrelevant. This is exactly the capability that Transformers achieve through attention — content-based memory management — but implemented through a recurrent state.

Comparison to prior work: The paper explicitly cites Arora et al. (2023), who "benchmarked a range of models for multi-query associative recall (MQAR) and identified a performance gap between various linear transformer architectures and the transformer with attention." The paper positions its MQAR experiments as showing that Finch has closed this gap — at least on the MQAR task itself — and that the specific combination of matrix-valued states and data-dependent decay is what closes it. The comparison with Mamba is particularly telling: both Mamba and Finch have data-dependent recurrence and matrix-valued states, but they perform differently on MQAR, suggesting that "different combinations of these elements result in superior performance" (Section 8.2).

This is a diagnostic contribution, not just a benchmark result. It provides evidence for a causal claim: data-dependent decay is the specific mechanism that enables content-based memory management in recurrent architectures, and this capability is what was missing from prior RNNs that caused their in-context learning deficits. This claim is testable and falsifiable — future work could manipulate the form of data-dependence (e.g., making it more or less expressive, applying it to different parameters) and observe the effect on MQAR to map out the relationship between recurrence design and associative recall capability.

Significance beyond performance: The MQAR analysis establishes a methodology for architectural research: identify the computational primitive that underlying a high-level capability (in-context learning), design a synthetic benchmark that isolates that primitive, trace how architectural choices enable or prevent the primitive, and use this understanding to predict and explain aggregate performance. This is more scientifically rigorous than the common practice of training models, running standard benchmarks, and speculating about why one architecture outperforms another. The paper does not fully execute this methodology — it could have included ablations showing, for example, that removing data-dependent decay from Finch causes MQAR performance to revert to Eagle levels — but the conceptual framework is clearly present and distinguishes this work from purely empirical architecture comparisons.

Evidence anchor: Figure 4, showing MQAR accuracy for RWKV-4, Eagle, and Finch across model dimensions and sequence lengths. The text in Section 8.2 explicitly frames this as a capability analysis: "an increase in sequence length correlates with increased task difficulty" and the differential performance across architectures reveals which design elements address this difficulty. The paper also states that "Finch achieves extremely high accuracy in MQAR in our tests, and outperforms all well-known non-transformer architectures previously used to train large language models" — a claim that positions the MQAR result as the key evidence for Finch's architectural superiority in content-based memory management.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training corpus is the newly introduced RWKV World v2 Dataset, a 1.12 trillion token multilingual corpus drawn from publicly available sources. The composition is approximately 70% English, 15% multilingual (non-English languages), and 15% code. Appendix D provides detailed breakdown of constituent data sources, which include curated web text, books, code repositories, subtitles, and conversations. The dataset was specifically designed to improve representation of non-European languages that are typically undertokenized by BPE tokenizers. For evaluation, the paper uses a broad collection of standard benchmarks:

English-focused benchmarks (Table 4): LAMBADA (OpenAI), HellaSwag, PIQA, AI2 ARC (Easy and Challenge), GLUE, Winogrande, SciQ, and COPA. Evaluations are conducted using lm_evaluation_harness (Gao et al., 2023).
Multilingual benchmarks (Table 3): LAMBADA Multilingual, XCOPA, XNLI (cross-lingual natural language inference), PAWS-X (cross-lingual paraphrase detection), XStoryCloze, and xWinogrande.
Associative recall (Section 8.2): Multi-Query Associative Recall (MQAR), a synthetic benchmark designed to test a model's ability to store and retrieve key-value pairs from earlier in context. The difficulty scales with sequence length and model dimension.
Long-context language modeling (Section 8.3): The PG19 test set of books (Rae et al., 2019), measuring loss as a function of sequence position beyond the training context length.
Bamboo benchmark (Section 8.4, Table 5): A comprehensive long-context evaluation covering question answering, hallucination detection, text sorting, language modeling, and code completion. The paper evaluates on the 4K context version.
Multimodal benchmarks (Section 10, Table 6): GQA (visual question answering), ScienceQA-IMG (multimodal science reasoning), TextVQA (text-in-image question answering), and POPE (object hallucination evaluation).
Music modeling (Section 10.1): The Irishman ABC music sheet dataset (Wu et al., 2023), evaluated via byte-level next-token prediction loss.
Audio classification (Section 11, Table 7): AudioSet (Gemmeke et al., 2017), evaluated via mean Average Precision (mAP).

Base model(s). The paper trains four Eagle models (0.46B, 1.5B, 3B, 7.5B parameters) and two Finch models (1.6B, 3B parameters). All models are trained from scratch on the RWKV World v2 Dataset with a context length of 4096 tokens. The models span approximately two orders of magnitude in parameter count, enabling scaling analysis. The paper also references and compares against RWKV-4 models (1.5B, 3B, 7B) trained by the same team, providing a direct architectural lineage comparison. For external baselines, the paper uses publicly available models from other families (Pythia, Mamba, Llama-2, Mistral, Falcon, MPT, BLOOM) as reported by lm_evaluation_harness or from original publications.

Metrics. The primary metrics differ by task type:

Language modeling benchmarks: Accuracy (acc) for multiple-choice tasks, perplexity (ppl) for LAMBADA Multilingual, and F1 scores for hallucination detection. All are standard metrics computed by lm_evaluation_harness.
Associative recall (MQAR): Accuracy (%) at retrieving the correct value given a key, measured across varying sequence lengths and model dimensions.
Long-context PG19: Token-level cross-entropy loss as a function of sequence position, measured from position 2048 onward to test beyond-training-distribution context utilization.
Speed and memory benchmarks (Section 9, Figures 6-7): Wall-clock time (milliseconds) and GPU memory usage (MB), measured on A100 80GB GPUs at batch size 8 with model dimension 4096 and head size 64.
Multimodal/Audio: Standard task-specific metrics — accuracy for ScienceQA-IMG and TextVQA, F1 for POPE, mAP for AudioSet.
Training compute: FLOPs are estimated using standard formulas (see Appendix E), enabling FLOPs-matched comparisons in the Pareto frontier plots (Figures 2-3).

Baselines. The paper compares Eagle and Finch against a comprehensive set of existing models, grouped by parameter scale for fair comparison:

At the ~1.4B–1.6B scale (Tables 3 and 4):

Pythia-1.4B (Biderman et al., 2023) — a Transformer trained on the Pile dataset
Mamba-1.4B (Gu & Dao, 2023) — a state space model with data-dependent recurrence
RWKV-4-1.5B (Peng et al., 2023) — the direct architectural predecessor

At the ~2.8B–3B scale:

Pythia-2.8B
Mamba-2.8B
RWKV-4-3B
Mamba-2.8B-Hermes (a fine-tuned variant on the Hermes dataset)

At the ~7B scale:

Pythia-6.9B
MPT-7B (MosaicML, 2023)
Llama-2-7B (Touvron et al., 2023)
Falcon-7B (Almazrouei et al., 2023)
Mistral-7B-v0.1 (Jiang et al., 2023)
RWKV-4-7B (the RWKV-4 "World" variant)
Eagle-7B (the largest Eagle model)
LLaMA2-Chat-7B and Mistral-Instruct-7B (instruction-tuned variants, used only on the Bamboo benchmark)

Additional multimodal baselines (Table 6): BLIP-2 (with Vicuna-13B and Flan-T5-11B), InstructBLIP (with Vicuna-7B and Vicuna-13B), IDEFICS-9B and IDEFICS-80B, and TinyGPT-V.

Audio baselines (Table 7): DeepRes (26M parameters), PANNs (81M), HTS-AT (28.8M).

The selection of baselines is designed to compare against both the direct RWKV lineage (RWKV-4) to isolate the effect of architectural innovations, and against the broader field of Transformers and alternative architectures (Mamba, Falcon, Llama-2, Mistral) to establish competitive positioning.

Generation budget / compute accounting. For the core language modeling evaluations (Tables 3 and 4), all models are evaluated in zero-shot or few-shot settings as specified by lm_evaluation_harness, using the same evaluation protocol to ensure comparability. The paper does not apply test-time compute scaling strategies; all models are evaluated with a single forward pass per example (greedy decoding or constrained multiple-choice scoring). For the FLOPs-matched scaling analysis (Figures 2 and 3), training FLOPs are estimated based on model parameters and training tokens using the standard formula $6ND$ , where $N$ is parameter count and $D$ is training tokens. For speed and memory benchmarks (Section 9), compute is measured in milliseconds of execution time and megabytes of GPU memory on identical A100 hardware. For the Pareto frontier plots, FLOPs are reported on a logarithmic scale to accommodate models spanning from 0.4B parameters to 7B parameters trained on tokens ranging from 300B to 2T.

Cross-validation / statistical protocol. The paper does not report formal statistical significance testing or cross-validation on the evaluation benchmarks. Results in Tables 3-7 are point estimates from a single evaluation run per model per benchmark, using the standard lm_evaluation_harness settings. The primary statistical control is the use of identical evaluation pipelines across all models — all results are obtained through the same harness, with the same prompts and scoring methods — which reduces evaluation variance but does not eliminate it. For the associative recall experiments, multiple model dimensions and sequence lengths are swept to characterize the scaling behavior, providing a form of reliability through systematic variation rather than repeated sampling. The paper does not report confidence intervals, standard deviations, or test-retest reliability for any benchmark result, which limits the ability to determine whether small differences (1–2 percentage points) between models are statistically meaningful.

Main Quantitative Results

Multilingual Benchmark Performance (Table 3)

The most decisive advantage for Eagle and Finch emerges on multilingual benchmarks, where the architectural improvements demonstrate their largest gains over both prior RWKV models and competing architectures. At every comparable parameter scale, Eagle and Finch achieve the highest average multilingual accuracy in their size class:

At the ~1.5B scale: Finch-1.6B leads with an average multilingual accuracy of 55.0%, followed by Eagle-1.5B at 54.3%, Mamba-1.4B at 51.8%, RWKV-4-1.5B at 51.8%, and Pythia-1.4B at 49.7%. The 3.2 percentage point gap between Finch-1.6B and Mamba-1.4B (the strongest non-RWKV competitor at this scale) is driven primarily by Finch's advantage on xWinogrande (74.9% vs. 72.4%) and LAMBADA Multilingual accuracy (46.9% vs. 40.4%). The LAMBADA Multilingual perplexity difference is particularly stark: Finch-1.6B achieves 37.5 ppl versus Mamba-1.4B's 73.1 ppl, indicating substantially better calibration on multilingual next-word prediction.

At the ~3B scale: Finch-3B achieves 57.1% average multilingual accuracy, compared to Eagle-3B at 56.5%, Mamba-2.8B at 52.7%, RWKV-4-3B at 53.9%, and Pythia-2.8B at 51.1%. The gap between Finch-3B and the next-best competitor (Eagle-3B) narrows to 0.6 percentage points, suggesting that the data-dependent innovations in Finch provide diminishing architectural returns over Eagle as scale increases within this range. However, both Eagle-3B and Finch-3B substantially outperform RWKV-4-3B (by 2.6 and 3.2 points respectively), confirming that the matrix-valued state is the primary driver of multilingual improvement at this scale.

At the 7B scale: Eagle-7B achieves 58.2% average multilingual accuracy, outperforming RWKV-4-7B (56.4%), Falcon-7B (54.7%), Mistral-7B-v0.1 (55.5%), and Llama-2-7B (54.3%). Notably, Eagle-7B's multilingual performance exceeds Mistral-7B's by 2.7 percentage points despite Mistral-7B's stronger English benchmark performance (Table 4: 75.8% vs. 71.5% average). This inversion — Eagle leads on multilingual, Mistral leads on English — provides strong evidence that the RWKV World Tokenizer and the multilingual-focused RWKV World v2 dataset are contributing substantially to Eagle's multilingual advantage, independent of (and perhaps even despite) architectural differences. Eagle-7B's standout individual results include 62.2% on XCOPA (beating Mistral's 55.9%), 63.3% on XStoryCloze (beating Mistral's 59.2%), and 53.7% on LAMBADA Multilingual accuracy (beating Mistral's 51.9%).

Core observations: The multilingual results reveal a consistent pattern: Eagle and Finch models dominate on tasks requiring cross-lingual knowledge transfer (XCOPA, XStoryCloze, xWinogrande) and multilingual next-word prediction (LAMBADA Multilingual), while showing more mixed results on tasks that may depend on English-centric linguistic patterns (PAWS-X paraphrase detection, where RWKV-4-7B's 52.1% exceeds Eagle-7B's 45.6%). The dramatic improvement in LAMBADA Multilingual perplexity from RWKV-4 to Eagle to Finch (e.g., at 1.5B: 72.5 → 43.2 → 37.5) suggests that both the matrix-valued state and data-dependent decay contribute to better multilingual modeling, with data-dependent decay providing the additional benefit of content-aware memory retention that may be particularly valuable for languages with different word order patterns than English.

English-Focused Benchmark Performance (Table 4)

Eagle and Finch demonstrate competitive but not dominant performance on English benchmarks. The key results are:

At the ~1.5B scale: Finch-1.6B achieves 62.9% average English accuracy, Eagle-1.5B 62.4%, Mamba-1.4B 63.1%, RWKV-4-1.5B 59.2%, and Pythia-1.4B 59.2%. Mamba-1.4B holds a narrow lead (0.2 points over Finch), primarily driven by higher HellaSwag (59.0% vs. 57.3%) and PIQA (74.2% vs. 72.6%). Finch-1.6B leads on LAMBADA (66.8% vs. 64.5%) and SciQ (89.6% vs. 87.1%), demonstrating strength in factual knowledge and narrative comprehension even at this small scale. The 3.7 percentage point gap between Finch and RWKV-4 at 1.5B represents a substantial generation-over-generation improvement for the RWKV family.

At the ~3B scale: Finch-3B achieves 67.5% average, Mamba-2.8B 66.2%, Eagle-3B 66.0%, RWKV-4-3B 64.1%, and Pythia-2.8B 62.5%. The 1.3 point gap between Finch-3B and Mamba-2.8B reverses the ordering seen at the 1.5B scale. Finch-3B's advantage is concentrated in LAMBADA (70.8% vs. 68.1%) and GLUE (58.2% vs. 46.3%), while Mamba-2.8B leads on HellaSwag (65.9% vs. 64.8%) and ARC Easy (69.7% vs. 66.5%). The 19.6-point GLUE gap is the largest single-task difference at this scale and reflects a broader pattern: RWKV-family models consistently achieve higher GLUE scores than comparably-sized competitors (RWKV-4-3B: 53.6%, Eagle-3B: 46.3% — note the Eagle-3B GLUE result appears anomalous and may represent a measurement issue since Eagle-1.5B scores 54.1% and RWKV-4-3B scores 53.6%).

At the 7B scale: Eagle-7B achieves 71.5% average, trailing Mistral-7B-v0.1 (75.8%), Falcon-7B (71.2%), and Llama-2-7B (71.1%), but leading RWKV-4-7B (67.3%), MPT-7B (70.9%), and Pythia-6.9B (63.8%). The 4.3-point gap between Eagle-7B and Mistral-7B is the central result for English benchmarks. Breaking this down by task:

Eagle-7B wins on SciQ (95.5% vs. 95.9% — essentially tied), and GLUE (57.5% vs. 51.5% — a reversal of the typical pattern where larger Transformers underperform smaller RWKV models on GLUE)
Eagle-7B is competitive on LAMBADA (74.2% vs. 75.5%), though trailing
Eagle-7B shows its largest deficits on HellaSwag (70.9% vs. 81.0%, a 10.1-point gap) and ARC Challenge (39.5% vs. 50.1%, a 10.6-point gap)

HellaSwag and ARC Challenge both require complex commonsense reasoning — HellaSwag asks models to choose the most plausible continuation of a narrative, while ARC Challenge requires scientific reasoning and world knowledge. The large gaps on these tasks suggest that Eagle-7B, despite its architectural improvements, underperforms Transformer architectures on tasks requiring integration of broad world knowledge with multi-step inference. This is consistent with the hypothesis that recurrent architectures may be less effective than attention at retrieving and combining disparate pieces of information from the pretraining corpus, even when they excel at maintaining coherent representations within a single context (as evidenced by the strong LAMBADA and SciQ results).

Pareto frontier analysis (Figures 2 and 3): The FLOPs-vs-accuracy plots provide a complementary perspective that accounts for training efficiency. On multilingual benchmarks (Figure 2), Eagle and Finch represent points that lie clearly above the Pareto frontier established by other models — i.e., for their training FLOPs, they achieve higher multilingual accuracy than any other model architecture tested. At approximately 6 × 10^21 FLOPs (the Eagle-7B training budget), Eagle achieves roughly 58% average multilingual accuracy, while the nearest Transformer at similar FLOPs (likely Falcon-7B or Llama-2-7B) achieves approximately 54–55%. This is a significant Pareto improvement. On English benchmarks (Figure 3), the picture is more nuanced: Eagle and Finch are competitive with the Pareto frontier but do not clearly dominate it. At similar FLOPs, several Transformer models achieve higher English accuracy, though the gap is smaller than the raw parameter-matched comparisons in Table 4 might suggest because Eagle and Finch achieve their performance with fewer training FLOPs than many comparably-sized Transformers (due to training on only 1.1T tokens versus 2T+ for Llama-2).

Associative Recall (MQAR) — Figure 4

The MQAR experiments provide the most direct evidence for the paper's central claim that data-dependent decay enables content-based memory management. The key findings:

RWKV-4 fails catastrophically at long sequence lengths. Across the tested configurations (model dimension 64–256, sequence lengths 32–256), RWKV-4's MQAR accuracy drops sharply as sequence length increases, with near-zero accuracy at sequence length 256 even at dimension 256. This is predicted by the theory: vector-valued states with static decay cannot disambiguate multiple stored key-value pairs that share feature dimensions, and the accumulated interference grows with the number of stored associations.

Eagle substantially improves but does not fully solve MQAR. The matrix-valued state provides enough cross-dimensional representational capacity to handle moderate numbers of key-value pairs, but with static decay all stored associations decay at the same rate regardless of their importance or frequency. At longer sequence lengths (128–256), Eagle's accuracy still degrades, though significantly less than RWKV-4's. For example, at dimension 128 and sequence length 128, Eagle maintains substantial accuracy while RWKV-4 is near floor.

Finch achieves near-perfect MQAR accuracy. Across the tested configurations, Finch maintains high accuracy even at the longest sequence lengths tested. This is the paper's headline architectural result: data-dependent decay, when combined with matrix-valued states, enables an RNN to perform the content-based storage and retrieval that was previously thought to require attention. The paper explicitly frames this as Finch "outperform[ing] all well-known non-transformer architectures previously used to train large language models" on this task (Section 8.2).

Comparison with Mamba: The paper notes that Mamba, despite sharing data-dependent recurrence and matrix-valued states with Finch, "performed much worse on this task" (specific Mamba MQAR numbers are not provided in the excerpt, but the statement is clear). This differential performance despite surface-level architectural similarity is the paper's key evidence that the specific form of data-dependence — LoRA-augmented decay within the RWKV token-shift framework — matters, not just the presence of data-dependence. The authors hypothesize that "different combinations of these elements result in superior performance."

Scaling behavior: The figure caption notes that "an increase in sequence length correlates with increased task difficulty" across all architectures. The question the MQAR experiments answer is not whether difficulty increases with sequence length (it does for all models) but how quickly it increases — and whether any architecture achieves near-constant accuracy independent of sequence length. Finch comes closest to this ideal among the tested architectures.

Long Context Experiments (Section 8.3, Figure 5)

The PG19 experiments test whether the architectural improvements translate to better language modeling on real text beyond the training context length. All tested 3B models (RWKV-4 World, Eagle, and Finch) were trained with context length 4096. The evaluation measures cross-entropy loss from token position 2048 onward to focus on long-range context utilization rather than short-range prediction.

Eagle dramatically improves over RWKV-4. The loss curves in Figure 5 show Eagle achieving substantially lower loss than RWKV-4 World across all positions tested. Since both models were trained on similar data with the same context length, this improvement can be attributed to the matrix-valued state providing richer representations of long-range dependencies.

Finch further improves on this beyond Eagle. The loss for Finch continues to drop further into the sequence compared to Eagle, indicating that data-dependent decay enables more effective retention of relevant information over very long contexts. This is consistent with the MQAR findings: content-based memory management matters for real-world language modeling, not just synthetic tasks. The continuing loss decrease for Finch at positions beyond 3500 suggests that the model is successfully leveraging context from much earlier in the sequence, whereas RWKV-4's loss flattens or increases, indicating it has effectively "forgotten" earlier context.

Quantitative magnitude: The paper does not provide exact loss values in the text, but the visual difference in Figure 5 between RWKV-4 and Finch at position 3500+ appears substantial (roughly 0.1–0.2 nats based on the y-axis scaling typical for such plots). In language modeling, differences of this magnitude at long context are meaningful and translate to measurable improvements in downstream task performance on long-document benchmarks.

Interpretation: The long-context results, combined with the MQAR findings, paint a coherent picture: Eagle's matrix-valued states provide the capacity to represent complex long-range dependencies, while Finch's data-dependent decay provides the control to manage which dependencies are retained. The PG19 improvement from Eagle to Finch is the empirical signature of this control — Finch is not just capable of storing more information (Eagle already had matrix-valued states) but is better at deciding what to store and what to forget based on content.

Bamboo Long-Context Benchmark (Section 8.4, Table 5)

The Bamboo benchmark evaluates nine long-context reasoning tasks spanning question answering (MeetingQA, PaperQA, AltQA), prediction (MeetingPred, ShowsPred), text sorting (ReportSumSort, ShowsSort), and hallucination detection (SenHallu, AbsHallu). All tasks use a maximum context window of 4K tokens.

At the 1.5B scale: Eagle-1.5B achieves 9.2% average score and Finch-1.6B achieves 8.9%, both dramatically outperforming Pythia-1.4B (2.1%) and Mamba-1.4B (2.1%). The near-zero scores for Pythia and Mamba (zero on 7 of 9 tasks) indicate that these smaller Transformer and Mamba models essentially fail at long-context reasoning, while Eagle and Finch demonstrate non-trivial capability, particularly on hallucination detection (SenHallu: Eagle 13.2%, Finch 10.7%) and question answering (PaperQA: Eagle 19.0%, Finch 22.0%).

At the 3B scale: Finch-3B achieves 11.3% average, Eagle-3B 9.9%, Mamba-2.8B-Hermes 11.9%, Mamba-2.8B 2.4%, and Pythia-2.8B 2.2%. The Mamba-2.8B-Hermes result is notable — this model was fine-tuned on the Hermes instruction dataset and achieves the highest average at this scale, suggesting that instruction tuning provides benefits for long-context reasoning that base pretraining alone does not capture. Among base models, Finch-3B leads. The 9.5-point gap between base Mamba-2.8B (2.4%) and base Finch-3B (11.3%) is striking, indicating that architectural differences in how recurrence handles long contexts produce massive differences in downstream reasoning capability, not just in language modeling loss.

At the 7B scale: Eagle-7B-Hermes (fine-tuned) achieves 16.8% average, compared to LLaMA2-Chat-7B at 24.1% and Mistral-Instruct-7B at 39.3%. The instruction-tuned Transformers hold clear advantages, though Eagle-7B-Hermes substantially outperforms Pythia-6.9B (3.3%), and is competitive with LLaMA2-Chat-7B on several individual tasks (MeetingQA: 31.0% vs. 6.0% for LLaMA2-Chat — a notable reversal). The hallucination detection tasks (SenHallu, AbsHallu) show Eagle-7B-Hermes at 50.3% and 46.9% F1 respectively, versus LLaMA2-Chat's 64.7% and 63.4% — substantial gaps but not orders of magnitude.

Interpretation: The Bamboo results demonstrate that Eagle and Finch's long-context capabilities, while substantially improved over prior RWKV models, still trail Transformer-based architectures on structured reasoning over long documents. The gap is particularly evident on tasks requiring aggregation across long contexts (ReportSumSort, ShowsSort) where all non-instruction-tuned models score zero. For deployment scenarios requiring long-context reasoning, these results suggest that instruction tuning — which the RWKV family had not yet fully explored at the time of this paper — may be as important as architectural improvements. The strong performance of Mamba-2.8B-Hermes supports this interpretation.

Speed and Memory Benchmarks (Section 9, Figures 6 and 7)

The speed and memory benchmarks directly quantify the inference advantages that motivate the RWKV architectural approach. Results are measured on A100 80GB GPUs with batch size 8, model dimension 4096, and head size 64.

Memory scaling (Figure 6): Finch consistently uses less memory than both Flash Attention and Mamba across all sequence lengths tested (1K to 16K). At 16K sequence length, Finch uses approximately 40% less memory than Flash Attention and approximately 17% less than Mamba. The memory curves remain approximately flat for Finch as sequence length increases — confirming the $O(1)$ memory property of the recurrent formulation — while Flash Attention memory grows roughly linearly with sequence length (consistent with KV cache requirements). Mamba memory also grows, suggesting that Mamba's recurrent implementation may retain some sequence-length-dependent memory allocations in practice despite its theoretical $O(1)$ property.

Training speed (Figure 7): Finch's training time scales linearly with sequence length, exhibiting similar scaling to Mamba (both approximately $O(N)$ , as expected for parallelizable recurrences). Flash Attention training time scales approximately quadratically, as expected for standard attention. The crossover point where Finch becomes faster than Flash Attention occurs at approximately 4K sequence length. At 16K sequence length, Finch is approximately 4.2x faster than Flash Attention.

Practical implications: For training with 8K+ context lengths — increasingly common in frontier LLM training — Finch offers ~3–5x training speed advantages over Flash Attention while using ~40% less memory. This translates directly to reduced training cost and the ability to use larger batch sizes or longer sequences on the same hardware. The paper notes that "further optimization of our Finch CUDA implementation, including algorithmic improvements, are possible, and could lead to speed increases and greater parallelization," suggesting these results may be conservative relative to what a fully optimized implementation could achieve.

Multimodal and Audio Results (Sections 10–11)

VisualRWKV (Table 6): VisualRWKV-Eagle-3B achieves 49.7% on GQA, 58.3% on ScienceQA-IMG, 46.4% on TextVQA, and 81.4% average F1 on POPE. These results are competitive with larger Transformer-based multimodal models: BLIP-2 with Vicuna-13B (a ~13B parameter LLM + 1B parameter vision encoder) achieves 41.0%, 61.0%, and 42.5% respectively; InstructBLIP with Vicuna-7B achieves 49.2%, 60.5%, and 50.1%. VisualRWKV achieves comparable or better GQA performance with a much smaller language model (3B vs. 7–13B), suggesting that the RWKV architecture's recurrent state may be particularly effective at the kind of spatial-relational reasoning that GQA tests (scene graph question answering).

AudioRWKV (Table 7): AudioRWKV-Tiny (8.7M parameters) achieves 0.435 mAP on AudioSet, comparable to HTS-AT (28.8M parameters, 0.437 mAP) — i.e., achieving essentially equivalent performance with 70% fewer parameters. AudioRWKV-S (28.4M parameters) achieves 0.452 mAP, outperforming all listed baselines including PANNs (81M parameters, 0.434 mAP). The paper attributes this to the quad-directional shift (Q-Shift) operation that extends token shift to 2D spectrograms, an architectural adaptation specific to the audio domain.

Music modeling (Figure 8): The RWKV-5-Music model achieves approximately 2% lower loss than the RWKV-4-Music model on the Irishman ABC dataset, with improvements concentrated in the musical score portions (positions 30–100+) rather than the file header (positions 0–30). This domain-specific result demonstrates that the Eagle architectural improvements (matrix-valued states, SiLU gating, improved initialization) generalize beyond text to structured sequence modeling.

Ablation Studies and Robustness Checks

Token shift parameter ablation (Appendix J — specific figure/table not provided): The paper references an ablation study in Appendix J comparing different token shift configurations. Without the specific figure or table available in the excerpt, the key claim from the main text is that both the static token shift (Eagle) and data-dependent token shift (Finch) provide meaningful improvements over no token shift, and that the Finch ddlerp mechanism specifically enables better induction head formation by allowing context-dependent blending of past and present information.

LoRA bottleneck dimension (Appendix K): The paper reports ablations on the LoRA bottleneck size used for data-dependent operations in Finch. The decay computation uses a bottleneck dimension of 64 rather than the 32 used for other LoRA-augmented parameters, and the text states that "future 7B and larger Finch models are expected to further increase the size of these weight matrices by double or more." This suggests that the bottleneck dimension is a capacity hyperparameter that likely yields diminishing returns — the current setting represents a practical tradeoff point rather than a saturation point. The exact performance impact of different bottleneck sizes is not quantified in the provided excerpt.

Channel Mixing hidden dimension (Section 4.1.3): The reduction of the Channel Mixing hidden dimension from $4D$ to $3.5D$ in Eagle is explicitly framed as a parameter-count equalization, not a performance optimization: "This reduction accounts for new gating weights in Eagle Time Mixing to ensure an equi-parameter relation with the prior model at the same number of layers and embedding dimension." The paper does not provide an ablation showing whether $3.5D$ performs better or worse than $4D$ when total parameter count is not controlled — it is an engineering choice to enable fair architectural comparison, not a claim about optimality.

LayerNorm over heads (Equation 7, implicit ablation): Eagle adds LayerNorm (equivalently GroupNorm with h groups) to each head's output before the SiLU gate. The paper does not provide an explicit ablation removing this LayerNorm, but the recurrence from RWKV-4 (which lacked head-wise normalization) to Eagle (which adds it) can be interpreted as an implicit ablation: the combination of matrix-valued states, SiLU gating, and head-wise LayerNorm together produce the observed improvements. Isolating the contribution of LayerNorm specifically would require a controlled ablation that the paper does not report.

PRM aggregation strategy: This terminology from the example template does not apply to this paper. The RWKV work does not involve process reward models or any variant of verifier-guided search.

Eagle as embedding model (Section 12, Limitations): A notable negative result: the authors attempted to use Eagle as an embedding model on the Massive Text Embedding Benchmark (MTEB) but "were not able to get strong embedding performance." The paper hypothesizes that Eagle's recurrent state "is a very high-quality embedding of the context but an appropriate method is required to aggregate the information content." This is an informative failure: the fixed-size recurrent state that enables $O(1)$ inference may be ill-suited for tasks requiring per-token representations (like embedding individual sentences or tokens) because the state is optimized for sequential accumulation, not for representing individual positions in isolation.

ReST $^{EM}$ training: This is referenced in the example template but does not apply to this paper. The RWKV work does not involve reinforcement learning from human feedback or any RL-based training pipeline.

Multilingual vs. English tradeoff: While not presented as a formal ablation, the comparison between multilingual benchmarks (Table 3) where Eagle dominates and English benchmarks (Table 4) where Eagle is merely competitive constitutes an implicit investigation of the tokenizer and dataset contributions. Eagle-7B's 58.2% multilingual average (beating Mistral-7B's 55.5%) versus 71.5% English average (trailing Mistral-7B's 75.8%) suggests that the RWKV World Tokenizer and multilingual training data contribute substantially to the multilingual advantage, potentially more than the architectural innovations themselves. A controlled experiment training Eagle with a standard BPE tokenizer on English-heavy data would isolate the architectural contribution, but such an experiment is not reported. This is a significant inferential gap: the paper cannot fully attribute Eagle's multilingual strength to architecture versus data/tokenization choices.

Context length extrapolation (Section 8.3, Figure 5): The long-context experiments themselves serve as a robustness check on the architecture's ability to handle sequence lengths beyond the training context. All models were trained with context length 4096 but evaluated out to sequence positions 4000+. The fact that Eagle and Finch maintain decreasing loss curves beyond the training context length (while RWKV-4 degrades) confirms that the architectural improvements enable length extrapolation, not just better within-distribution modeling. The paper states that Eagle-7B was "pretrained with context length 4096, but no fundamental context length limitation or relationship to speed" — the PG19 experiment provides evidence for the first half of this claim (no fundamental limitation) and the speed benchmarks (Figure 7) provide evidence for the second (no relationship to speed — the per-token cost remains constant regardless of context length).

Vision encoder ablations (Table 6): VisualRWKV uses CLIP-L (0.4B parameters) as the vision encoder, while many competing models use larger encoders (CLIP-G at 1.0B, EVA01-CLIP-G at ~1.0B). The fact that VisualRWKV achieves competitive results with a smaller vision encoder suggests that the Eagle language model's architectural efficiency allows it to extract more from limited visual features, though a formal ablation keeping the vision encoder fixed while varying the language model architecture is not performed.

AudioRWKV Q-Shift ablation: The paper introduces Q-Shift as an audio-specific extension of token shift, but does not provide an ablation comparing AudioRWKV with and without Q-Shift on AudioSet. This makes it impossible to determine whether the Q-Shift innovation specifically, or the Eagle architecture more generally, drives the parameter efficiency gains over baselines.

Critical Assessment

Claim 1: "Eagle and Finch significantly improve over RWKV-4 on benchmarks for LLMs"

What the experiments demonstrate: This claim is clearly supported by the language modeling benchmark results. At every comparable parameter scale, Eagle outperforms RWKV-4 on both English and multilingual benchmarks. At 1.5B, Eagle achieves 62.4% English average vs. RWKV-4's 59.2% (Table 4) and 54.3% multilingual average vs. RWKV-4's 51.8% (Table 3). At 7B, Eagle achieves 71.5% English vs. RWKV-4's 67.3% and 58.2% multilingual vs. RWKV-4's 56.4%. Finch further improves on Eagle at the 1.6B and 3B scales (e.g., 62.9% vs. 62.4% English at 1.5B; 57.1% vs. 56.5% multilingual at 3B).

What is not demonstrated: The paper does not establish whether the improvements come primarily from architectural innovations (matrix-valued states, data-dependent decay) or from improved data (the RWKV World v2 dataset, which at 1.12T tokens differs from whatever RWKV-4 was trained on) and tokenization (the RWKV World Tokenizer, which is new in this work). Since RWKV-4 models used in comparison are the "World" variants that may have been trained on different data, the architectural versus data contribution cannot be isolated. Additionally, the improvements from Finch over Eagle are modest at some scales (0.5 points English at 1.5B, 1.5 points at 3B), raising the question of whether data-dependent decay contributes incrementally beyond what matrix-valued states already provide for standard benchmarks, even though it proves crucial for associative recall.

Claim 2: "The advancements provide significant progress toward developing more efficient AI models"

What the experiments demonstrate: The speed and memory benchmarks (Figures 6 and 7) provide strong evidence for efficiency advantages. Finch uses 40% less memory than Flash Attention at 16K sequence length and is 4.2x faster. The $O(1)$ per-token inference property is clearly demonstrated and has practical significance for deployment. The Pareto frontier plots (Figures 2 and 3) show that Eagle and Finch achieve competitive or superior accuracy for their training FLOPs, particularly on multilingual tasks.

What is not demonstrated: The paper does not compare total cost of ownership — training cost plus inference cost — in a realistic deployment scenario. A model that is 4.2x faster at training but requires 2x more parameters to achieve the same accuracy as a Transformer may not have lower total cost when amortized over millions of inference queries. The assertion of "more efficient" depends on the deployment scenario: for on-device inference with strict memory constraints, the efficiency advantage is clear; for high-throughput cloud serving where memory is abundant, the advantage may be marginal if Transformer models with Flash Attention and KV caching can serve similar throughput. The paper also does not benchmark inference throughput (tokens/second) in a realistic serving configuration with batching, continuous batching, and varying sequence lengths — measures that would matter more for production deployment than the training speed benchmarks reported.

Claim 3: "Finch achieves extremely high accuracy in MQAR and outperforms all well-known non-transformer architectures"

What the experiments demonstrate: Figure 4 shows Finch achieving near-perfect MQAR accuracy across tested configurations, dramatically outperforming RWKV-4 and Eagle at longer sequence lengths where both fail. The claim explicitly references "non-transformer architectures" and the paper positions Finch above Mamba on this task.

What is not demonstrated: The paper does not report Mamba's MQAR accuracy numbers explicitly, making it impossible to verify the magnitude of Finch's claimed advantage. The statement that Mamba "performed much worse on this task" is qualitative and underspecified — how much worse, at what sequence length, at what model dimension? Additionally, MQAR is a synthetic task designed to isolate associative recall. Superior MQAR performance does not guarantee superior performance on real-world in-context learning, which the paper's own benchmark results acknowledge: despite near-perfect MQAR, Finch-3B trails Mamba-2.8B on English average accuracy (67.5% vs. 66.2%, a small gap but a gap nonetheless). The relationship between MQAR performance and practical in-context learning capability is hypothesized but not demonstrated.

Claim 4: "Eagle and Finch models perform competitively with existing models under a wide variety of sequence modeling domains"

What the experiments demonstrate: The paper evaluates on English benchmarks, multilingual benchmarks, long-context language modeling (PG19), long-context reasoning (Bamboo), associative recall (MQAR), music modeling, vision-language tasks, and audio classification. This is genuinely broad coverage.

What is not demonstrated: "Competitively" is doing substantial work here. On English benchmarks at the 7B scale, Eagle-7B trails Mistral-7B-v0.1 by 4.3 percentage points on average accuracy (Table 4: 71.5% vs. 75.8%). On individual tasks, the gaps are larger: 10.1 points on HellaSwag, 10.6 points on ARC Challenge. Whether a 4-point average gap constitutes "competitive" depends on the use case — for many applications, this difference is material. The paper's framing of "competitive" relies on the multilingual results (where Eagle leads) and the inference efficiency (where Eagle's $O(1)$ memory provides a capability advantage). A fairer characterization might be: "Eagle and Finch achieve strong performance on multilingual tasks and competitive-within-striking-distance performance on English tasks, with inference efficiency advantages that may outweigh accuracy differences depending on deployment requirements."

Claim 5: "Matrix-valued states and data-dependent recurrence are the key innovations driving improvement"

What the experiments demonstrate: Ablation is largely implicit in the architectural progression: RWKV-4 → Eagle adds matrix-valued states (plus SiLU gating, head LayerNorm), Eagle → Finch adds data-dependent decay and token shift. The progression in benchmark scores (RWKV-4 < Eagle < Finch on multilingual and long-context tasks, RWKV-4 < Eagle ≈ Finch on English) and the dramatic differences on MQAR (RWKV-4 fails, Eagle partially succeeds, Finch fully succeeds) support the causal attribution.

What is not demonstrated: No clean ablation isolates individual mechanisms. For example, we cannot tell from the reported results whether matrix-valued states without SiLU gating would work as well; whether data-dependent token shift matters independently of data-dependent decay; whether the LoRA bottleneck dimension of 32 or 64 is optimal; whether head-wise LayerNorm is necessary or incidental. The paper's architectural evolution approach is valid for demonstrating that the combination of changes produces improvement, but it cannot establish which specific changes matter most or whether some are redundant. For instance, if Eagle's improvements over RWKV-4 come primarily from better initialization (mentioned in Section 3) rather than matrix-valued states, the paper provides no evidence that would reveal this.

Genuine Weaknesses and Gaps

Dataset and tokenizer confounds: The RWKV World v2 Dataset and RWKV World Tokenizer are introduced alongside Eagle and Finch, and the comparison models (Mamba, Pythia) were trained on different data. The multilingual advantage of Eagle over Mamba and Pythia could be entirely attributable to better tokenization and more multilingual training data, independent of architecture. The paper acknowledges the dataset contribution but does not attempt to control for it. A training run of Eagle or Finch with a standard BPE tokenizer on the Pile (or conversely, training a Transformer baseline on the RWKV World v2 data) would address this but is not performed.

No 7B Finch model: Despite Finch being presented as the more advanced architecture, the largest Finch model trained is 3B parameters. At the 7B scale, only Eagle is compared against Transformer baselines. This is a significant gap: the scaling behavior of Finch's data-dependent mechanisms may differ at larger model sizes, and the claim that Finch improves over Eagle is only validated up to 3B parameters. The paper notes this as future work ("We also plan to train and release larger versions of Finch such as 7B and 14B parameters"), but the absence limits the strength of Finch-specific conclusions.

Single training run per model: Each reported model (Eagle-0.4B, Eagle-1.5B, etc.) represents a single training run. There are no replicates, no sweeps over hyperparameters, and no characterization of training instability or seed sensitivity. This is standard practice in large-scale LLM research (training multiple 7B models from scratch is prohibitively expensive), but it means the reported benchmark scores could be affected by training noise or lucky/unlucky initialization. The 46.3% GLUE score for Eagle-3B — lower than Eagle-1.5B's 54.1% and RWKV-4-3B's 53.6% — may be an example of such noise.

Limited instruction tuning experiments: The Bamboo benchmark (Table 5) shows that instruction-tuned models (Mamba-2.8B-Hermes, Eagle-7B-Hermes, LLaMA2-Chat-7B) substantially outperform base models on long-context reasoning. The paper does not systematically explore instruction tuning for Eagle/Finch models, leaving unclear whether the architectural advantages persist after fine-tuning or whether Transformers benefit more from instruction tuning, potentially erasing the recurrent architecture's advantages.

Missing comparisons to concurrent architectures: The paper does not compare against GLA (Yang et al., 2023), Griffin (De et al., 2024), or HGRN (Qin et al., 2023) despite discussing them in the related work (Section 2). All were developed concurrently and represent alternative approaches to data-dependent recurrence. Including these comparisons (even if only on MQAR or a subset of benchmarks) would strengthen the claim that Finch's specific form of data-dependence is superior.

Associative recall as a necessary but not sufficient proxy: The paper makes a strong theoretical claim that MQAR performance predicts in-context learning capability. While the literature supports this connection (Olsson et al., 2022; Elhage et al., 2021), the paper does not directly evaluate in-context learning capability (e.g., few-shot accuracy scaling with number of examples, or tasks specifically designed to require pattern matching from context). The jump from synthetic MQAR to real-world in-context learning is a logical leap that the paper's own benchmark results do not fully validate — Finch excels at MQAR but does not clearly excel at the benchmarks that should benefit from better in-context learning (the English-focused benchmarks in Table 4 show Mamba-1.4B and Mamba-2.8B competitive with or slightly ahead of Finch in their respective size classes on average).

Training data contamination: The paper does not discuss data decontamination — whether evaluation benchmarks overlap with the RWKV World v2 training data. This is especially relevant for SciQ, where Eagle-7B achieves 95.5% accuracy (Table 4), and for LAMBADA, where Finch and Eagle show large improvements over prior models. Without decontamination analysis, it is impossible to know whether these strong results reflect genuine language understanding improvements or memorization of benchmark examples.

Missing latency benchmarks for inference: The speed benchmarks (Figure 7) measure training time, not inference latency. For a paper whose primary motivation is improved inference efficiency (Section 2 emphasizes on-device deployment, batch serving cost, and unbounded context applications), the absence of inference latency measurements — especially time-to-first-token and per-token generation time at varying batch sizes and sequence lengths — is a notable gap. The $O(1)$ per-token property guarantees constant per-step cost, but the absolute cost (in milliseconds per token) relative to a Transformer with KV cache, Flash Attention, and optimized serving infrastructure is what would determine practical adoption.

6. Limitations and Trade-offs

6.1 The Contribution of Architecture vs. Data and Tokenization Cannot Be Isolated

The assumption or constraint: The paper introduces three changes simultaneously — new architectures (Eagle/Finch), a new tokenizer (RWKV World Tokenizer), and a new training dataset (RWKV World v2) — and evaluates the resulting models against baselines trained with different tokenizers on different data distributions. The paper does not perform a controlled experiment holding tokenization and data constant while varying only the architecture. Section 6 acknowledges the dataset is "designed to go beyond the English-heavy focus of many datasets widely used to train LLMs today," and the tokenizer is explicitly built to "mitigate undue burden, which naive BPE and related methods cause, on minor languages" (Section 5).

The consequence: The paper's strongest empirical result — Eagle and Finch dominating multilingual benchmarks (Table 3) — cannot be attributed to architectural innovation with any confidence. A substantial portion of the multilingual performance advantage over Mamba, Pythia, and even Mistral-7B could stem from the RWKV World Tokenizer providing better token coverage for non-English languages (giving them more balanced token-to-semantic-unit ratios) and the RWKV World v2 dataset containing proportionally more multilingual data (15% of 1.12T tokens). Consider a concrete counterfactual: if Mamba were trained with the same tokenizer on the same dataset, would its multilingual performance match or exceed Eagle's? The paper provides no evidence either way. Conversely, if Eagle were trained with a standard BPE tokenizer on an English-heavy corpus (e.g., the Pile), would its English benchmark performance improve or decline? The 4.3-point English average gap between Eagle-7B and Mistral-7B (Table 4: 71.5% vs. 75.8%) might be attributable partly to dataset differences — Mistral was trained on a larger, potentially higher-quality dataset with undisclosed composition — rather than to architectural limitations of the RWKV recurrent design.

This confound also affects the scaling analysis. The Pareto frontier plots (Figures 2 and 3) plot accuracy vs. training FLOPs for models trained on different datasets. A model that achieves higher accuracy per FLOP might do so because its training data is more informative, not because its architecture extracts more learning per token. The paper's central scaling claim — that Eagle and Finch advance the multilingual Pareto frontier — conflates architectural efficiency with data efficiency.

What evidence exists in the paper: The confound is visible in the pattern of results. Eagle-7B achieves 58.2% multilingual average vs. Mistral-7B's 55.5% (Table 3) — a domain where the tokenizer and dataset are designed to help. But on English benchmarks (Table 4), Eagle-7B trails Mistral-7B by 4.3 points — a domain where the tokenizer/dataset advantage is neutral or possibly negative (since 15% of training tokens were code and 15% were multilingual, reducing the effective English token budget compared to an English-heavy training mix). This asymmetry is consistent with a data/tokenization explanation and also consistent with an architectural explanation (Eagle's recurrence being better at cross-lingual transfer but worse at English-specific commonsense reasoning). The paper cannot distinguish these explanations.

Mitigation status: The paper does not address this confound. It treats the tokenizer and dataset as contributions in their own right (listed as contributions 2 and 3 in the introduction), which they are, but does not acknowledge that this bundling prevents clean architectural attribution. Section 12 (Future Work) mentions expanding the training corpus but does not mention controlled tokenizer or data ablation studies. The only related acknowledgement is the note in Limitations that "our training corpus contains some synthetic data from GPT-3.5 and ChatGPT" which "will mimic ChatGPT's conversation style" — a separate data contamination concern, not a confound concern.

6.2 No Finch Model at 7B Parameters Exists to Validate Scaling of the Data-Dependent Mechanisms

The assumption or constraint: Finch, the paper's most advanced architecture combining matrix-valued states with data-dependent decay and token shift, is only trained and evaluated at 1.6B and 3B parameter scales. The largest RWKV model compared against 7B-class Transformers is Eagle-7B, which lacks Finch's data-dependent recurrence. The paper states in Section 12: "We also plan to train and release larger versions of Finch such as 7B and 14B parameters," explicitly acknowledging this as future work.

The consequence: All conclusions about Finch's advantages over Transformers at production-relevant scales are extrapolations from sub-4B models. This matters for several reasons:

First, the associative recall results (Figure 4) — Finch's strongest single-result claim — are demonstrated at model dimensions 64–256 (corresponding to very small models by LLM standards, likely well under 100M parameters). Whether Finch's near-perfect MQAR accuracy at dimension 256 translates to improved in-context learning in a 7B model with dimension 4096 is unknown. Scaling laws for in-context learning are not necessarily linear — capabilities can emerge discontinuously at certain scales (Wei et al., 2022), and the relationship between MQAR accuracy and real-world few-shot performance may change with model size.

Second, the data-dependent mechanisms in Finch add parameters and computation relative to Eagle. The LoRA augmentations ( $A_\square \in \mathbb{R}^{D \times 32}$ , $B_\square \in \mathbb{R}^{32 \times D}$ for each of $\mu_x$ , $\lambda_\square$ across multiple operations, plus doubled bottlenecks for the decay computation) add approximately $2 \cdot 32D \cdot 5 + 2 \cdot 64D \cdot 1 = 448D$ parameters per layer for the data-dependent components. As model dimension $D$ grows, these costs scale linearly with $D$ but the benefit relative to static mechanisms may not scale proportionally. It is possible that at larger scales, the capacity provided by standard pretraining alone reduces the marginal value of data-dependent recurrence — or, conversely, that data-dependence becomes more important at larger scales where the model has the capacity to learn sophisticated memory management policies. The paper provides no evidence in either direction.

Third, the speed and memory benchmarks (Figures 6 and 7) that demonstrate Finch's efficiency advantages are measured at model dimension 4096 — relevant for 7B models — but the Finch implementation used for those benchmarks may differ from what would be required for a full 7B Finch training run. The memory and speed advantages demonstrated at the kernel level may not fully materialize at the full model level when all components (multiple heads, channel mixing, embedding, output projection) are included.

What evidence exists in the paper: The gap is directly observable: Table 4 contains no Finch row at the 7B scale. The speed benchmarks (Section 9) test Finch kernels at dimension 4096, providing some evidence that the mechanisms scale computationally, but no evidence that they scale in terms of modeling quality. The MQAR experiments (Figure 4) show Finch outperforming Eagle across all tested model dimensions, but these dimensions (64–256) are an order of magnitude below the 4096 used in the speed benchmarks and the ~4096 used in Eagle-7B.

Mitigation status: The paper explicitly defers this to future work (Section 12). There is no 7B Finch training run, no FLOPs-matched comparison at 7B between Eagle and Finch, and no extrapolation analysis predicting Finch-7B performance based on 1.6B → 3B scaling trends. The statement in Section 4.2.1 that "future 7B and larger Finch models are expected to further increase the size of these weight matrices by double or more" implies that even the LoRA bottleneck dimensions tested at 1.6B/3B are not the final design — further reducing confidence that current Finch results predict future Finch scaling.

6.3 Inference Latency Under Realistic Serving Conditions Is Not Characterized

The assumption or constraint: The paper benchmarks training speed (Figure 7) but does not measure inference latency — specifically time-to-first-token and per-token generation time — in a realistic serving configuration with batching, variable sequence lengths, and continuous request arrival. The paper's central motivation for the RWKV architecture is inference efficiency (Section 2 discusses on-device deployment, high-throughput serving, and unbounded context), yet the only deployment-relevant metric reported is memory usage (Figure 6).

The consequence: The $O(1)$ per-token inference property guarantees that per-step cost is constant with respect to context length, but it does not guarantee that this constant is small. A Finch model might have $O(1)$ per-token cost that is, in absolute terms, $3\times$ the per-token cost of a Transformer with a modestly-sized KV cache at typical sequence lengths. Users deploying models for real-time applications (chatbots, code completion, live translation) care about milliseconds per token, not asymptotic complexity class. If Finch's recurrent state update requires a matrix multiplication $\text{diag}(w) \cdot s$ where $s \in \mathbb{R}^{(D/h) \times (D/h)}$ , and $D/h = 64$ (typical head size), the per-step state update involves $64^2 = 4096$ multiply-adds per head, times $h$ heads. For Eagle-7B with $D = 4096$ and $h = 64$ heads, this is $64 \times 64 \times 64 = 262,144$ operations per layer for the state update alone, plus the outer product $k^T v$ , the receptance multiplication, and all other projections. Whether this is faster or slower than a Transformer's attention computation at a given context length depends on implementation details, hardware characteristics, and batch size — none of which the paper explores for inference.

Additionally, the paper's training speed advantage (4.2× over Flash Attention at 16K sequence length) is measured on the time-parallel training implementation. Inference operates in strictly sequential recurrent mode (token by token, state updated each step), which has different computational characteristics than the parallelized training mode. The training implementation leverages parallelism across the time dimension (Equations 8-9 are computed simultaneously for all time steps during training), while inference must compute them sequentially. The paper's training speed numbers do not bound inference speed.

What evidence exists in the paper: Figure 7 measures training time, not inference time. Figure 6 measures memory usage — Finch uses 40% less memory than Flash Attention and 17% less than Mamba at 16K sequence length — which is a genuine deployment advantage but only addresses the memory half of the latency-memory tradeoff. Section 8.3 demonstrates that Eagle and Finch can handle sequence lengths beyond the training context (PG19 evaluation) and Section 8.4 demonstrates long-context reasoning capability (Bamboo), but neither measures the time required to process those long contexts at inference. Table 1 reports theoretical time complexity ( $O(1)$ for RWKV inference), but theoretical complexity and wall-clock latency can diverge substantially due to constant factors.

Mitigation status: The paper does not acknowledge this gap. Section 9, titled "Speed and Memory Benchmarks," reports only training speed. There is no discussion of inference latency, no measurement of time-to-first-token or tokens-per-second generation rate, and no comparison against optimized Transformer inference engines (vLLM, TensorRT-LLM, etc.) that use techniques like paged attention, continuous batching, and KV cache quantization to mitigate Transformers' memory costs in production. The note that "further optimization of our Finch CUDA implementation, including algorithmic improvements, are possible" applies to training kernels but does not mention inference optimization.

6.4 No Evidence That Architectural Improvements Transfer to Instruction-Tuned or Aligned Models

The assumption or constraint: All Eagle and Finch models evaluated in the main results (Tables 3 and 4, Figures 2–7) are base pretrained models without instruction tuning, RLHF, or other alignment. The paper does not train or evaluate instruction-tuned Eagle/Finch models on standard chat/instruction benchmarks (MT-Bench, AlpacaEval, Chatbot Arena). The only instruction-tuned variants mentioned are Eagle-7B-Hermes (Table 5, Bamboo benchmark) and a brief mention in Appendix G.2 (MT-Bench, specific numbers not provided in excerpt), but these are not systematically compared against instruction-tuned Transformers at scale.

The consequence: A growing fraction of LLM deployment involves instruction-tuned models — users interact with chat models, not base models. It is unknown whether the architectural advantages of Eagle/Finch (matrix-valued states, data-dependent decay) survive instruction tuning or whether instruction tuning disproportionately benefits Transformers, potentially erasing the performance gap or even reversing it. There are mechanistic reasons to suspect the transfer may not be straightforward:

First, instruction tuning typically involves learning to follow diverse formatting patterns and task specifications. Transformers' attention mechanism provides direct access to arbitrary tokens in the instruction — the model can "look back" at specific parts of the system prompt or few-shot examples at any point during generation. RWKV's recurrent state must compress the entire instruction into a fixed-size matrix representation. If the instruction is long and detailed (as is common in modern chat models with extensive system prompts), this compression may lose information that attention would preserve.

Second, the data-dependent decay in Finch is trained on next-token prediction from web text. The memory management policies learned during pretraining — when to retain information, when to forget — may not transfer to the instruction-following setting, where different types of information need to be retained (e.g., task specifications, formatting requirements, safety guidelines). Fine-tuning might need to substantially re-learn these policies, and the LoRA bottleneck (32–64 dimensions) may limit the model's ability to adapt its memory management to the instruction distribution.

Third, the only direct evidence — Eagle-7B-Hermes on Bamboo (Table 5) — shows a mixed picture. Eagle-7B-Hermes achieves 16.8% average vs. LLaMA2-Chat-7B's 24.1% vs. Mistral-Instruct-7B's 39.3%. This is a 7.3-point gap to LLaMA2-Chat and a 22.5-point gap to Mistral-Instruct — substantially larger than the 4.3-point English average gap between base Eagle-7B and base Mistral-7B (Table 4: 71.5% vs. 75.8%). While the Bamboo task distribution differs from the English benchmark suite, the larger gap under instruction tuning is suggestive that Transformers may benefit more from alignment than RWKV architectures do.

What evidence exists in the paper: Table 5 (Bamboo) shows Eagle-7B-Hermes at 16.8% vs. LLaMA2-Chat-7B at 24.1% vs. Mistral-Instruct-7B at 39.3%. Appendix G.1 discusses alignment benchmarks and G.2 discusses MT-Bench, but the specific results are not included in the provided excerpt. The main benchmark tables (3 and 4) evaluate only base models.

Mitigation status: The paper acknowledges instruction tuning as relevant (the Bamboo benchmark includes instruction-tuned models, Appendix G.1 covers alignment), but does not treat the lack of instruction-tuned Eagle/Finch models as a limitation. The future work section (Section 12) does not mention instruction tuning or alignment as a priority. This is a significant omission given that downstream adoption of LLMs is predominantly through instruction-tuned interfaces.

6.5 The Associative Recall to In-Context Learning Causal Chain Is Asserted, Not Demonstrated

The assumption or constraint: The paper makes a specific causal claim: that the MQAR improvements from data-dependent decay (Figure 4) explain why Finch performs better on language modeling benchmarks, and by extension that improved associative recall enables better in-context learning. Section 8.2 states that "prior research suggests that a model's ability to perform AR is indicative of its effectiveness in in-context learning" and the paper positions MQAR as the key diagnostic benchmark for architectural comparison. However, the paper never directly evaluates in-context learning capability — it does not measure few-shot accuracy as a function of the number of in-context examples, does not test whether Finch benefits more from additional examples than Eagle or Mamba, and does not isolate in-context learning from parametric knowledge on any benchmark.

The consequence: The paper's strongest architectural claim — that Finch's data-dependent decay closes the gap to Transformers on content-based memory management — rests on an unvalidated proxy. MQAR is a synthetic task where the model must store randomly generated key-value pairs and retrieve the correct value when the key reappears. Real-world in-context learning involves retrieving semantically related information, composing multiple retrieved facts, and integrating retrieved information with parametric knowledge. These are substantially more complex operations than key-value lookup.

The gap between MQAR performance and apparent in-context learning capability is visible in the English benchmark results. Finch-3B achieves near-perfect MQAR (Figure 4) but achieves 67.5% average English accuracy vs. Mamba-2.8B's 66.2% (Table 4) — a 1.3-point difference that does not obviously reflect the dramatic MQAR gap. If associative recall were the dominant factor in benchmark performance, Finch's MQAR advantage should translate into a larger English benchmark advantage. The fact that it does not suggests that either (a) MQAR performance is not the bottleneck for these benchmarks, (b) Mamba's data-dependent mechanisms achieve similar real-world in-context learning despite lower MQAR scores, or (c) Finch's MQAR advantage is offset by other architectural weaknesses.

More fundamentally, the paper cannot distinguish whether Finch's MQAR improvement comes from data-dependent decay specifically enabling in-context learning, or from some other consequence of the architectural change (e.g., better optimization dynamics, different inductive biases that happen to help on MQAR but not on real tasks). The paper states that Mamba "performed much worse on this task" despite sharing data-dependent recurrence (Section 8.2), but without explaining why — is Mamba's form of data-dependence fundamentally less suited to associative recall, or did the specific hyperparameters tested happen to be suboptimal? Without controlled ablations (e.g., Finch with static decay vs. Finch with data-dependent decay on few-shot benchmarks), the causal mechanism remains speculative.

What evidence exists in the paper: Figure 4 (MQAR accuracy). The connection to in-context learning is asserted via citation to prior work (Elhage et al., 2021; Olsson et al., 2022) but not experimentally validated within this paper. The English benchmark results (Table 4) provide implicit (and somewhat contradictory) evidence — Finch's small advantage over Mamba despite large MQAR advantage. The Bamboo benchmark (Table 5) involves some tasks that could benefit from associative recall (question answering over long documents requires retrieving specific facts mentioned earlier), but the paper does not analyze Bamboo performance through the lens of associative recall or provide evidence that Finch's Bamboo improvements are mediated by better key-value retrieval.

Mitigation status: The paper treats the MQAR-to-in-context-learning connection as established by prior work and does not attempt to validate it within this study. Future work could directly test this by evaluating few-shot scaling curves (accuracy vs. number of examples) for Eagle vs. Finch vs. Mamba — a model with better in-context learning should show steeper improvement as more examples are added. The paper does not propose this or acknowledge it as a gap.

6.6 LoRA Bottleneck Capacity for Data-Dependent Mechanisms Is Chosen Heuristically Without Scaling Analysis

The assumption or constraint: Finch's data-dependent operations — token shift blending weights ( $\text{lor}_{\square}$ in Equation 14) and decay rates ( $\text{lora}_d$ in Equation 17) — rely on LoRA-style low-rank augmentations with fixed bottleneck dimensions: 32 for most operations, 64 for the decay computation. The paper states (Section 4.2.1) that "future 7B and larger Finch models are expected to further increase the size of these weight matrices by double or more," indicating that a 64-dimensional bottleneck (or smaller) is not assumed to be optimal at larger scales. However, the paper provides no experiments varying the bottleneck dimension and analyzing the impact on model quality, MQAR performance, or computational cost.

The consequence: A critical design parameter in Finch's architecture — the capacity of the data-dependent mechanisms — is set heuristically without evidence that the chosen values are near-optimal or even in the right regime. This has several implications:

First, the performance comparison between Eagle and Finch is confounded by the fact that Finch has additional parameters (the LoRA weight matrices). The paper states that Finch's Channel Mixing hidden dimension is not reduced (unlike Eagle's reduction from $4D$ to $3.5D$ ), meaning Finch models have more parameters than comparably-configured Eagle models. Some of Finch's performance advantage over Eagle may simply reflect higher parameter count rather than the superiority of data-dependent mechanisms.

Second, if the bottleneck dimension is too small, Finch may be under-expressing the potential of data-dependent recurrence — the model literally cannot learn sophisticated enough memory management policies because the bottleneck constrains the complexity of the function mapping from context to decay rates. At model dimension $D = 4096$ , a 64-dimensional bottleneck can represent at most 64 independent learned patterns for how context should modulate decay. Whether this is sufficient for the diversity of linguistic contexts requiring different memory policies is an open empirical question that the paper does not address.

Third, if the bottleneck dimension is larger than necessary, Finch is wasting parameters and computation on capacity that the model cannot effectively use — a particularly costly error given that these parameters are in the critical path of every time step and every layer. The paper's statement that future models will increase bottleneck dimensions "by double or more" implies the authors believe current bottlenecks are undersized, but this belief is not supported by ablation evidence in the current paper.

The computational cost of the LoRA augmentations is not trivial. For each time-mixing block, computing $\tanh(x A) B$ involves a $D \times 32$ matrix multiply followed by a $32 \times D$ matrix multiply — $64D$ multiply-adds per LoRA application. With five LoRA applications per block ( $\text{lor}_\square$ for $r$ , $k$ , $v$ , $g$ , and the doubled $\text{lora}_d$ for $w$ ) plus the $\text{lor}_{\square}$ inside each ddlerp, the total LoRA cost per block is on the order of $500D$ operations. For $L = 32$ layers and $D = 4096$ , this is approximately 65 million operations per token just for the LoRA computations. Whether this cost is justified by the performance improvement is impossible to assess without bottleneck dimension ablations.

What evidence exists in the paper: None. Appendix K is titled "DDLerp Ablations" according to the table of contents but its content is not provided in the excerpt. Even if Appendix K contains some ablation, the main text's statement about future models doubling bottleneck sizes acknowledges that the current settings are not treated as final or validated as optimal.

Mitigation status: The paper explicitly defers bottleneck dimension optimization to future work (Section 4.2.1: "future 7B and larger Finch models are expected to further increase the size of these weight matrices by double or more"). There is no attempt to characterize the sensitivity of Finch's performance to this hyperparameter, no comparison of different bottleneck sizes at the 1.6B or 3B training scales, and no analysis of the computational cost vs. performance tradeoff. For a practitioner deciding whether to adopt the Finch architecture, this means one of the architecture's most novel components — the data-dependent mechanisms — has unknown sensitivity to a critical hyperparameter, making it difficult to estimate the cost of scaling Finch to larger sizes or to reason about whether the LoRA approach would transfer to different model dimensions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not claim to dethrone the Transformer. It does not demonstrate that RNNs are uniformly better, or that attention is obsolete, or that the field has been optimizing the wrong architecture. What it does — and this is arguably more valuable than a sweeping paradigm-shift claim — is establish a concrete, measurable performance envelope for modern RNN-based language models and provide a diagnostic toolkit for understanding why one recurrent architecture outperforms another.

The landscape shift is best understood along three axes: architectural design philosophy, evaluation methodology, and deployment pragmatics.

Architectural design philosophy: from "RNNs need to be more like Transformers" to "RNNs need the right algebraic structure." Prior to this work, the dominant narrative around RNNs for language modeling was that they needed to recapitulate the operations that make attention powerful — content-based retrieval, cross-dimensional interactions, dynamic memory management — but within a recurrent formulation. The RWKV lineage, culminating in Eagle and Finch, demonstrates that these capabilities can be achieved through specific, principled design choices (matrix-valued states, data-dependent decay, token shift as an architectural primitive) rather than through attempting to approximate attention directly. This is a reframing: the goal is not to build a cheaper Transformer, but to build a better RNN.

The paper's key conceptual moves enable this reframing:

Matrix-valued states (Eagle) establish that the algebraic type of the recurrent state is a first-class design dimension. This changes the conversation from "how big should the state be" to "what tensor shape should the state have, and what operations should it support?" Future RNN designs can now ask: is a matrix sufficient, or would higher-order tensors capture even richer interactions? What is the optimal rank? How does head size interact with state dimensionality? These questions were not on the table when all recurrent states were assumed to be vectors.
Data-dependent decay (Finch) establishes that content-based memory management is the specific capability gap between RNNs and Transformers, and that it can be closed through learned modulation of forgetting rates rather than through content-based retrieval. This is a more precise claim than "RNNs need to be more expressive." It identifies a mechanism — the model's ability to decide, at each time step, how much of each channel's accumulated information to retain based on what it is currently reading — and shows that this mechanism, when implemented through LoRA-augmented learned vectors, produces a discrete jump in associative recall capability (Figure 4) and a measurable improvement in long-context language modeling (Figure 5).
Token shift elevated from implementation trick to architectural primitive. The paper's explicit connection between token shift and induction head formation (Section 4.1.1) reframes a seemingly minor operation as a necessary condition for in-context learning in recurrent architectures. This suggests a broader design principle: recurrent architectures need explicit, cheap access to local temporal differences (handled by token shift), while the recurrent state handles long-range accumulation, and data-dependent mechanisms handle content-based modulation of memory. This layered approach to temporal modeling — different mechanisms for different timescales — is a conceptual contribution that could inform architectures beyond the RWKV family.

Evaluation methodology: MQAR as a diagnostic for architectural capability. The paper's use of Multi-Query Associative Recall as a targeted diagnostic tool (Section 8.2) represents a methodological contribution to how architectures should be compared. Aggregate benchmark scores (Tables 3 and 4) show that Finch improves over Eagle, but they do not reveal why. MQAR isolates a single capability — content-based key-value storage and retrieval — that mechanistic interpretability research has identified as the core operation underlying in-context learning. By showing that RWKV-4 fails on MQAR (static decay with vector states cannot disambiguate stored associations), that Eagle partially succeeds (matrix states provide capacity but static decay causes interference), and that Finch achieves near-perfect accuracy (data-dependent decay enables selective retention), the paper traces a causal chain from architectural mechanism to capability to benchmark performance.

This methodology — identify the computational primitive underlying a high-level capability, design a synthetic benchmark that isolates it, trace how architectural choices enable or prevent it — is more scientifically rigorous than the common practice of training models, running standard benchmarks, and speculating about why one architecture outperforms another. It enables architecture researchers to make falsifiable predictions: if a new architecture claims to improve in-context learning, it should demonstrate improved MQAR; if it does not, the claimed mechanism is likely incorrect. The paper does not fully execute this program — it does not, for example, ablate data-dependent decay from Finch and show MQAR reverting to Eagle levels — but the framework is clearly present and provides a template for future architectural research.

Deployment pragmatics: reframing the Transformer-RNN comparison around deployment constraints rather than benchmark scores. The paper's positioning of Eagle and Finch is notable for what it does not claim. It does not claim that Eagle-7B beats Mistral-7B on English benchmarks (it does not; Table 4 shows a 4.3-point gap). It does not claim that RNNs are strictly superior to Transformers. Instead, it argues that the relevant comparison is not "which architecture wins at a given parameter count on standard benchmarks" but rather "given a specific deployment constraint — memory budget, sequence length, per-token latency requirement — which architecture delivers the best quality?"

This reframing matters because it shifts the evaluation criteria from a single dimension (benchmark accuracy) to a multi-dimensional tradeoff space (accuracy, memory, latency, context length). Under memory constraints that make Transformers' KV caches infeasible — on-device deployment, high-throughput batch serving, unbounded context applications — the comparison is between Eagle/Finch and other $O(1)$ -memory architectures, not between Eagle and Mistral. The paper provides concrete evidence for this reframing: Finch uses 40% less memory than Flash Attention at 16K sequence length (Figure 6) and is 4.2× faster to train at that length (Figure 7). These numbers, not the benchmark tables, are the primary evidence for the paper's efficiency claims.

The paper also contributes to the open science landscape by releasing all models, training code, inference code, and the training dataset under Apache 2.0 license (Table 2). In a field where training data composition is increasingly treated as proprietary, this transparency enables the kind of controlled experimentation — training different architectures on identical data, ablating tokenizer choices, measuring data contamination — that the paper itself does not perform but that the community can now pursue.

What this work does not change: The paper does not resolve the question of whether RNNs will eventually match or exceed Transformers on all capabilities. The 4.3-point English average gap between Eagle-7B and Mistral-7B (Table 4), and the larger gaps on commonsense reasoning tasks (10.1 points on HellaSwag, 10.6 points on ARC Challenge), suggest that attention still provides advantages for certain types of knowledge integration and multi-step inference that recurrence has not fully replicated. The paper also does not establish whether the architectural advantages of Eagle/Finch survive instruction tuning — the limited evidence from Bamboo (Table 5, Eagle-7B-Hermes trailing LLaMA2-Chat-7B by 7.3 points) is suggestive but insufficient for strong conclusions. These are open questions, not resolved findings.

Reconciling prior contradictions: The paper helps reconcile a tension in the literature between works claiming that linear attention and RNNs can match Transformers (Katharopoulos et al., 2020a; Peng et al., 2023; Gu & Dao, 2023) and works showing persistent gaps on specific capabilities (Arora et al., 2023, who identified associative recall as a weakness of linear Transformer architectures). The resolution is not "one side is right" but rather: (a) the gap on associative recall is real but can be closed through specific architectural mechanisms (data-dependent decay), (b) closing this gap improves benchmark performance but does not fully close the Transformer gap because other capabilities (commonsense reasoning, multi-step inference) may depend on different mechanisms, and (c) the remaining gap may be as much about training data scale and quality as about architecture. This is a more nuanced, and more productive, position than either extreme.

Follow-Up Research This Work Enables

Training Eagle/Finch and Transformer baselines on identical data with identical tokenization to isolate architectural contribution. This is the most important immediate follow-up, directly motivated by the paper's largest inferential gap: the bundling of new architectures with a new tokenizer and new dataset. A clean experiment would train Eagle-7B, Finch-7B, and a strong Transformer baseline (e.g., Llama-2-7B architecture) on exactly the same 1.12T-token RWKV World v2 dataset using the same RWKV World Tokenizer (or, alternatively, train all three with a standard BPE tokenizer on the Pile). This would isolate the architectural contribution from the data and tokenization contributions, answering the question: how much of Eagle's multilingual advantage over Mistral-7B (Table 3: 58.2% vs. 55.5%) comes from architecture versus data? If the Transformer baseline trained on RWKV World v2 matches or exceeds Eagle on multilingual benchmarks, the paper's architectural claims would need significant revision — the tokenizer and dataset, not the recurrent design, would be the primary drivers of multilingual performance. Conversely, if Eagle still leads, the architectural claims are strengthened. This experiment is now feasible because the paper has released the dataset and training code.

Training and evaluating Finch at 7B parameters to determine whether data-dependent mechanisms scale. The paper demonstrates Finch's advantages at 1.6B and 3B but acknowledges no 7B Finch model exists (Section 12). Two specific questions need answers: (1) Does Finch-7B's performance advantage over Eagle-7B follow the same scaling trend observed at smaller sizes, or does the gap widen (suggesting data-dependent mechanisms become more important at scale) or narrow (suggesting larger models can compensate for static recurrence through sheer capacity)? (2) Does Finch-7B's MQAR advantage over Eagle-7B persist at model dimension 4096, or does the relationship between model dimension and MQAR accuracy saturate? The paper's MQAR experiments (Figure 4) maxed out at dimension 256 — two orders of magnitude below 7B-scale dimensions. A Finch-7B training run with MQAR evaluation at multiple sequence lengths would answer these questions and validate whether the paper's central architectural claim scales to production-relevant model sizes. Given that the paper reports a 7B Finch model as planned future work, this is the most natural immediate extension.

Testing whether data-dependent decay specifically enables few-shot in-context learning. The paper asserts a causal chain from MQAR performance to in-context learning capability (Section 8.2) but never measures in-context learning directly. A targeted experiment would evaluate Eagle and Finch models on standard few-shot benchmarks with varying numbers of in-context examples (0-shot, 1-shot, 4-shot, 8-shot) and measure the slope of improvement. If data-dependent decay specifically enables in-context learning, then: (a) Finch should show a steeper improvement slope than Eagle as more examples are added, (b) the gap between Finch and Eagle should widen with more examples, (c) on tasks where few-shot examples provide large benefits (e.g., translation, structured extraction), Finch should disproportionately outperform Eagle. If Finch does not show these patterns despite near-perfect MQAR, the paper's central causal claim is undermined and alternative explanations for Finch's benchmark improvements (better optimization dynamics, different inductive biases) would need investigation. The MMLU benchmark in few-shot mode would be a natural testbed since it measures diverse knowledge integration that should benefit from effective example retrieval.

Ablating individual components of Finch's data-dependent mechanisms to identify which matter most. The paper bundles multiple innovations in Finch: data-dependent token shift (ddlerp for $r$ , $k$ , $v$ , $g$ , and $d$ ), data-dependent decay (LoRA-augmented $w_t$ ), and the specific LoRA bottleneck dimensions (32 for most operations, 64 for decay). A structured ablation at the 1.6B scale could train variants with: (a) data-dependent token shift but static decay (ddlerp on $r$ , $k$ , $v$ , $g$ but static $w$ ), (b) static token shift but data-dependent decay (Eagle-style lerp on everything except LoRA-augmented $w_t$ ), (c) data-dependent decay with varying bottleneck dimensions (16, 32, 64, 128), (d) data-dependent decay with simpler parameterizations (direct linear projection instead of LoRA). The resulting MQAR and benchmark performance curves would reveal: which mechanism is primarily responsible for Finch's associative recall improvement? Is data-dependent token shift valuable independently, or is its contribution inseparable from data-dependent decay? What is the optimal bottleneck dimension for different model sizes? These ablations would transform Finch from a single-point design into a characterized design space that future work can navigate systematically. The paper's Appendix K ("DDLerp Ablations") may already contain some of this analysis, but the provided excerpt does not include it.

Evaluating the RWKV World Tokenizer's contribution to multilingual performance independently of architecture. The paper makes specific claims about the tokenizer's design philosophy — manual vocabulary curation to balance token coverage across languages, Trie-based greedy matching for speed — but provides no controlled comparison against standard BPE tokenizers. A clean experiment would take a fixed architecture (e.g., Eagle-1.5B) and train it on the same multilingual data using: (a) the RWKV World Tokenizer, (b) a standard BPE tokenizer with the same vocabulary size (65,536) trained on the same corpus, (c) a BPE tokenizer trained on a standard English-heavy corpus (mimicking what Llama-2 or Mistral use). Evaluating all three variants on the multilingual benchmarks from Table 3 (LAMBADA Multilingual, XCOPA, XNLI, XStoryCloze, xWinogrande) would isolate the tokenizer's contribution. If the RWKV World Tokenizer variant substantially outperforms the BPE variants, it validates the paper's claim that manual vocabulary curation mitigates undue burden on minor languages and provides a concrete tokenization methodology that other multilingual LLM projects could adopt. If the difference is small, the tokenizer contribution is separable from the architectural contribution, and the paper's multilingual leadership is primarily an architectural story.

Stress-testing the length extrapolation claims. The paper demonstrates (Figure 5) that Eagle and Finch trained on context length 4096 maintain decreasing loss curves beyond 4096 on PG19, but this is a single dataset and only tests language modeling loss, not downstream task performance. A more rigorous stress test would: (a) evaluate Eagle-7B and Finch-3B on the Bamboo benchmark's 16K-context version (the paper only evaluates 4K), measuring whether task performance degrades or is maintained at lengths beyond the training context, (b) test on retrieval tasks where a specific piece of information is placed at varying positions within a long document (similar to the "needle in a haystack" evaluation), measuring accuracy as a function of retrieval distance, (c) compare against Transformer models that use position extrapolation techniques (e.g., RoPE scaling, ALiBi) on the same long-context tasks. This would establish whether the paper's claim of "no fundamental context length limitation" (Table 2) holds for structured reasoning, not just for next-token prediction loss. If Finch maintains task accuracy at 16K+ contexts where Transformers with extrapolated position encodings degrade, it strengthens the practical case for RNN architectures in long-context applications. If Finch's task performance degrades despite stable language modeling loss, it reveals a gap between the architectural capability (the state can represent long contexts) and the model's learned ability to use that capability for structured tasks — a finding that would motivate research on long-context fine-tuning strategies for recurrent architectures.

Practical Applications and Downstream Use Cases

On-device multilingual assistants with persistent context. A smartphone manufacturer wants to deploy a language model that can serve as a real-time multilingual assistant — translating, composing messages, answering questions — while maintaining conversation context across arbitrarily long interactions. The model must run entirely on-device (no cloud offloading) within a strict memory budget (e.g., 2GB for model weights plus runtime state), must support at least 50 languages with balanced token efficiency, and must not slow down as the conversation lengthens. Eagle-0.4B or Eagle-1.5B, combined with the RWKV World Tokenizer, directly addresses these constraints: the recurrent state is fixed-size regardless of conversation length (no KV cache growth), the tokenizer provides balanced coverage for non-European languages (avoiding the 5–6× token blowup that BPE imposes on Korean or Japanese text), and the $O(1)$ per-token inference cost ensures that the 100th message in a conversation generates just as quickly as the first. The paper provides concrete supporting numbers: Eagle-1.5B achieves 54.3% average multilingual accuracy (Table 3), exceeding all comparably-sized Transformers, and Finch uses 40% less memory than Flash Attention at 16K context (Figure 6). The 0.46B Eagle model (the smallest released) could fit within a ~500MB memory budget for weights plus state, leaving room for the OS and other applications — a deployment scenario that is simply infeasible for even the smallest practical Transformer due to KV cache growth.

Cost-efficient batch inference for multilingual document processing at scale. A document processing service needs to classify, summarize, and extract entities from millions of documents across dozens of languages on a fixed GPU budget. The service runs batch inference — processing thousands of documents simultaneously on a single A100 — and the primary cost driver is GPU memory, which limits maximum batch size. Because Eagle/Finch models require no KV cache and maintain constant memory per sequence regardless of document length, they can support substantially larger batch sizes than equivalently-sized Transformer models. Concretely: at 16K sequence length, Finch uses 40% less memory than Flash Attention (Figure 6) for the attention-equivalent computation, meaning 1.67× more sequences can fit in the same GPU memory. For the multilingual document processing use case, Eagle-7B's 58.2% multilingual average (beating Mistral-7B's 55.5%) and the RWKV World Tokenizer's balanced token coverage mean the model is both more accurate on non-English text and more memory-efficient — a rare combination where accuracy and cost move in the same direction rather than trading off. The paper's demonstration that Eagle and Finch handle context lengths beyond training (Figure 5, PG19 evaluation at 4K+ on models trained at 4K) provides confidence that the models will not break on documents longer than the training context, a common failure mode for position-encoding-dependent Transformers.

Streaming speech-to-text and real-time transcription with low and predictable latency. A real-time transcription service processes a continuous audio stream (e.g., a live lecture, a court proceeding, a broadcast) and must output text with minimal and predictable latency. The service uses an audio encoder (e.g., AudioRWKV from Section 11) followed by an autoregressive language model for contextual biasing and rescoring. Latency predictablity — ensuring that processing the 10,000th audio frame takes exactly the same time as the 10th — is critical for maintaining real-time guarantees. The RWKV architecture's $O(1)$ per-step cost provides this guarantee intrinsically, without the growing computational burden that a Transformer's attention mechanism imposes as the audio stream lengthens. The paper's AudioRWKV results (Table 7) demonstrate that the architecture's parameter efficiency extends to audio: AudioRWKV-Tiny achieves 0.435 mAP at 8.7M parameters, matching HTS-AT's 0.437 mAP at 28.8M parameters (70% fewer parameters), and AudioRWKV-S achieves 0.452 mAP at 28.4M parameters, outperforming all baselines. For a combined audio-to-text pipeline where both the encoder and decoder use RWKV architectures, the entire system operates in $O(1)$ memory and $O(1)$ per-step time regardless of audio length — a property that Transformer-based pipelines with attention in either the encoder or decoder cannot match.

Open-source research infrastructure for controlled architectural comparisons. The paper's release of all models, training code, inference code, and the training dataset under Apache 2.0 license (Table 2) creates a complete, reproducible baseline for the research community to conduct the kind of controlled experiments that the paper itself does not perform. A research group with limited compute (e.g., an academic lab with access to 8×A100 GPUs) can now: (a) reproduce the Eagle-1.5B training run to verify the paper's claims, (b) train a Transformer baseline on the exact same data with the exact same tokenizer to isolate architectural contribution from data contribution, (c) ablate specific architectural components (e.g., data-dependent decay, token shift) and measure the impact on MQAR and benchmarks while holding all other variables constant. This infrastructure addresses the paper's largest methodological gap — the confound between architecture, tokenization, and data — by making it experimentally tractable for the community. No other 7B+ parameter model family (Table 2) provides this combination of open weights, open training code, open inference code, and open dataset under a permissive license. The practical application is not a specific product but rather an enabling platform: future papers on RNN architectures, linear attention, and state space models can use the RWKV World v2 dataset and RWKV World Tokenizer as a standardized testbed for controlled comparisons, increasing the scientific rigor of architectural research.

When to Prefer This Method

The paper itself does not articulate an explicit "use Eagle/Finch when X, use Transformers when Y" decision framework — the positioning throughout is that Eagle and Finch are competitive alternatives whose advantages materialize under specific deployment constraints. However, from the experimental results and the paper's stated motivations, the conditions under which Eagle or Finch would be preferred over Transformer alternatives can be inferred. These are not framed as absolute rankings but as contingent tradeoffs that depend on the deployment scenario.

Memory-constrained deployment with long or unbounded sequences strongly favors Eagle/Finch. The paper's Figure 6 shows Finch using 40% less memory than Flash Attention at 16K context length, and this gap grows with sequence length because RWKV memory is $O(1)$ while Transformer memory is $O(N)$ . For on-device deployment (phones, laptops, embedded systems) with limited total RAM, or for cloud serving with high batch sizes and long sequences, this memory difference is decisive — it determines whether the model can run at all, not just how fast it runs. The paper's long-context experiments (Figure 5, Table 5) provide evidence that the memory efficiency does not come at the cost of losing long-range capabilities: Eagle and Finch maintain decreasing loss and non-trivial task performance at context lengths beyond their training horizon.

Multilingual applications with balanced language coverage favor Eagle/Finch, but the role of architecture versus tokenizer requires caution. Eagle-7B's 58.2% multilingual average (Table 3) beats all comparably-sized competitors including Mistral-7B (55.5%) and Falcon-7B (54.7%). However, as discussed in Section 6.1, this advantage is confounded with the RWKV World Tokenizer and dataset — it is unknown how much of the multilingual lead is architectural versus data-driven. A practitioner deploying for multilingual applications should not assume Eagle/Finch will outperform Transformers on their specific language mix without testing, but the combination of the architecture and the tokenizer/dataset package released with the paper provides a strong starting point that no other open model family currently matches.

Training efficiency at long context lengths favors Finch over Flash Attention-based Transformers, with a crossover at approximately 4K sequence length. Figure 7 shows Finch training time scaling linearly while Flash Attention scales approximately quadratically, with Finch becoming 4.2× faster at 16K. For pretraining runs targeting 8K+ context lengths (increasingly common in 2024), Finch offers a training cost advantage that directly reduces the FLOP budget required to reach a given context length. The paper does not, however, compare Finch training efficiency against other linear-complexity training approaches (Mamba's associative scan, GLA's chunked parallelism), so this advantage is relative to standard attention, not relative to all sub-quadratic alternatives.

Tasks requiring predictable, constant per-step latency favor Eagle/Finch over Transformers, regardless of average speed. The $O(1)$ per-step inference property guarantees that processing token 10,000 takes exactly the same number of operations as processing token 10. Transformers' per-step cost grows with context length, creating a latency profile where response time degrades over the course of a long interaction. For real-time applications (streaming transcription, live translation, interactive dialogue), this predictability may be more important than average speed — an application that must guarantee sub-50ms per token for every token will prefer a model with constant per-step cost, even if that constant is slightly higher than a Transformer's average cost at typical sequence lengths. The paper does not provide inference latency numbers (Section 6.3 discusses this gap), so this recommendation is based on asymptotic properties rather than measured wall-clock times.

Tasks requiring the highest possible accuracy on English-language commonsense reasoning and knowledge integration currently favor Transformers. The 4.3-point English average gap between Eagle-7B and Mistral-7B-v0.1 (Table 4: 71.5% vs. 75.8%), and the larger gaps on HellaSwag (10.1 points) and ARC Challenge (10.6 points), indicate that state-of-the-art Transformers still hold an edge on complex English-language reasoning. For applications where every percentage point of accuracy matters — medical question answering, legal reasoning, high-stakes decision support — and where deployment constraints permit Transformer inference (sufficient GPU memory for KV caches, moderate context lengths), the paper's results do not support choosing Eagle/Finch over a stronger Transformer like Mistral-7B or Llama-2-7B on accuracy grounds alone. The tradeoff is between accuracy and efficiency — Eagle/Finch provide substantially better efficiency with a moderate accuracy cost on English tasks, and the paper provides the numbers to quantify that tradeoff but not to resolve which side a given application should choose.