RWKV-7 "Goose" with Expressive Dynamic State Evolution

ArXiv: 2503.14456

🎯 Pitch

RWKV-7 reimagines recurrent networks by giving each neuron its own dynamic, data-dependent learning rate and forgetting rule—a “relaxed” delta update—that allows the 2.9B model to match or beat Transformers on English and multilingual benchmarks while using far fewer training tokens, and it provably recognizes languages no fixed-attention Transformer can. This architecture blurs the line between training and inference, achieving practical parity with state-of-the-art 3B models at a fraction of the compute.

1. Executive Summary

This paper introduces RWKV-7 "Goose", a new recurrent neural network architecture for sequence modeling that generalizes the delta rule with vector-valued gating and in-context learning rates — replacing the scalar learning rate and decay of prior delta-rule models with per-channel, data-dependent vectors — alongside a relaxed value replacement rule that decouples removal and replacement keys. The 2.9 billion parameter model, trained on a newly released 3.1 trillion token multilingual corpus (RWKV World v3), achieves a new 3B state-of-the-art on multilingual benchmarks and matches the 3B English-language state-of-the-art despite being trained on dramatically fewer tokens than competitors like Qwen2.5 (5.6T vs. 18T tokens). The architecture's generalized delta rule transition matrix — formulated as St = St-1(diag(wt) - κ̂tᵀ(at ⊙ κ̂t)) + vᵀt k̃t — enables expressivity beyond the TC⁰ complexity class, and the paper proves that RWKV-7 can recognize all regular languages with a constant number of layers, establishing a theoretical capability advantage over Transformers and diagonal state-space models only when the state admits non-diagonal, input-dependent transitions (as in the rank-one update structure deployed here).

2. Context and Motivation

The Core Problem: Linear Attention Architectures Face a Fundamental Memory Tension

The paper addresses a specific, long-standing tension in the design of efficient sequence models: how to build a recurrent architecture that can edit its fixed-size memory state with sufficient precision to match or exceed the in-context learning capabilities of Transformers, while maintaining constant per-token inference cost and parallelizable training. This problem sits at the intersection of three competing desiderata—expressivity, efficiency, and trainability—that have historically forced architectural trade-offs.

To understand why this tension is so fundamental, consider what happens in a standard linear attention model (Katharopoulos et al., 2020b). The model maintains a fixed-size matrix-valued state that encodes key-value associations from all previously seen tokens. At each new token, the model adds a new outer product (vᵀk) to this state. The problem is that the state only grows — older values are never explicitly removed, only gradually diluted as a smaller fraction of the accumulating numerical sum. Eventually, for a state of finite size, this unbounded accumulation forces the model to mix distinct key-value pairs together, degrading the quality of retrieved outputs when queried with a specific key (Schlag et al., 2021; Yang et al., 2024b).

This is not merely a numerical nuisance. It represents a fundamental architectural bottleneck: linear attention models cannot selectively forget or replace specific stored associations, only globally decay them. A Transformer's key-value cache, by contrast, never mixes distinct values together because it stores each token's key and value separately and attends to them independently. The quadratic cost of this approach is the price paid for perfect memory fidelity.

Why This Problem Matters: Three Practical Motivations

The paper is motivated by three concrete, real-world concerns that extend well beyond academic interest:

1. The inference cost scaling crisis. Transformers incur O(N) time per token and O(N) memory with respect to sequence length N during autoregressive generation, because the entire key-value cache must be accessed at each step. For short sequences, modern GPU parallelism can amortize much of this cost, but as context lengths grow—a clear trend in the field, with production models now routinely handling hundreds of thousands of tokens—Transformer inference becomes increasingly expensive. A recurrent architecture with constant per-token cost and constant memory usage would scale gracefully to arbitrarily long sequences, making it attractive for deployment in latency-sensitive or memory-constrained settings (on-device inference, real-time streaming applications, very long document processing).

2. The state-tracking and expressivity gap. Merrill et al. (2024) proved an important theoretical result: Transformers and recurrent architectures with purely diagonal state transition matrices are limited to the complexity class TC⁰, meaning they cannot represent certain simple state-tracking operations—like tracking permutations of a fixed set of elements—that are in the higher complexity class NC¹. A practical consequence: a Transformer given a sequence of swap operations on five elements cannot track which element ends up in which position without external scratchpad computation (chain-of-thought). This is not just a theoretical curiosity; state tracking is central to many reasoning tasks (algorithmic reasoning, game playing, multi-step planning). An architecture that can perform such state tracking internally, without the costly token-by-token generation of intermediate reasoning steps, would have a meaningful capability advantage at inference time.

3. The pretraining compute bottleneck. The dominant paradigm for improving language model performance has been to scale up pretraining compute—train larger models on more data. However, the paper emphasizes that this approach faces diminishing returns per unit of compute and is increasingly inaccessible to all but the largest industrial labs. An alternative is to invest computation at inference time rather than at pretraining time. A recurrent architecture that can meaningfully use its internal state to process and reason over context—editing memories, swapping stored values, recognizing state transitions—effectively spends inference compute to improve output quality, offering a path to better performance without scaling model size or training data. The paper explicitly positions RWKV-7 as enabling this paradigm: the state is an "internal scratchpad" that can be modified through learned operations, not just appended to.

Where Prior Approaches Fall Short

The paper situates itself within a rapidly evolving lineage of RNN architectures for language modeling, each attempting to address the memory-fidelity problem with increasing sophistication. The key prior approaches and their specific limitations are:

Linear attention with per-timestep decay (RWKV-4 through RWKV-6, RetNet, GLA, Mamba, Mamba-2, HGRN-2). These models address the unbounded accumulation problem by applying a decay factor to the state at each timestep, reducing the contribution of older entries. The decay can be:

• A fixed scalar per head (RetNet, Mamba-2): computationally simple but lacks the flexibility to decay different channels at different rates.
• A data-dependent vector (RWKV-5/6, Mamba): more expressive, allowing each channel to independently decide how much old information to retain.
• A gating mechanism that interpolates between retaining and replacing (HGRN-2): uses the decay vector itself as an interpolation coefficient between old state and new value.

However, decay is fundamentally a blunt instrument. It can reduce the magnitude of old values, but it cannot selectively remove only the value stored at a specific key while preserving others. If the model needs to update a fact (e.g., "the capital is Paris" → "the capital is London"), decay must attenuate all stored values equally to reduce the influence of the old fact, potentially losing other useful information in the process. Decay can "forget" globally, but it cannot "edit" locally.

The delta rule (DeltaNet; Schlag et al., 2021). DeltaNet introduced a qualitatively different approach: instead of just decaying old values, actively remove the value stored at a specific key and replace it with a new value. It frames the state update as an online learning problem—train the state matrix S at test time to map keys to values using stochastic gradient descent. The update rule St = St-1(I - a kᵀk) + a vᵀk is equivalent to one step of SGD on the loss L = ½||S k - v||². The scalar learning rate a ∈ [0,1] controls how much of the old value at key k is removed and replaced by the new value v.

This elegantly solves the selective editing problem: when a = 1, the old value at key k is fully removed; when a = 0, nothing is changed. Parallelized DeltaNet (Yang et al., 2024c) showed that this update admits efficient parallel computation, making it practical for training.

However, DeltaNet's formulation has significant limitations that the paper identifies:

• Scalar learning rate a: all channels (dimensions of the key) share the same removal/replacement rate. In practice, the model might want to aggressively update some dimensions of a stored value while preserving others—a scalar cannot express this.
• Scalar decay in later variants: Gated DeltaNet (Yang et al., 2024a) added scalar data-dependent decay wt to DeltaNet, producing St = St-1(diag(wt) - kᵀk diag(at)) + vᵀk diag(at), but both wt and at remain scalars, so the expressive power relative to vector-valued operations is limited.
• Identical removal and replacement keys: DeltaNet uses the same key k both to locate the old value to remove and to specify where the new value should be stored. This is conceptually clean (SGD on the association for key k) but might be unnecessarily restrictive. The model might benefit from locating the old value using a different key representation than the one used for storage.

State Space Models (S4, Mamba) and their expressivity ceiling. Merrill et al. (2024) showed that models with purely diagonal state transition matrices—which includes S4, Mamba, and all linear attention variants with only scalar decay—are confined to the complexity class TC⁰. This means they provably cannot perform certain state-tracking operations that a simple RNN with a dense transition matrix could. The paper shows that this is not merely a theoretical concern: on the state-tracking benchmark using group multiplication (Section 7.6, Figure 8), Mamba and S4 require more layers than classical RNNs to achieve the same accuracy, and their performance degrades with sequence length.

TTT and Titans: batched gradient descent. Two concurrent works—TTT (Sun et al., 2024) and Titans (Behrouz et al., 2024)—apply the delta rule idea differently. Rather than updating the state at every timestep with a single SGD step, they accumulate gradients over multiple timesteps and apply them in a batched update. Titans also adds momentum to the state update. While this can produce more stable optimization, it complicates the recurrence and can make parallelization harder. More fundamentally, both use scalar learning rates and scalar decays (Table 1), inheriting the same limitations as Gated DeltaNet.

What was missing: a unified generalization. The paper observes that all of these approaches—linear attention with decay, DeltaNet with scalar learning rate, Gated DeltaNet with scalar decay—can be viewed as special cases of a more expressive update rule. In Table 1, the paper presents a taxonomy along four dimensions: Large State (matrix-valued states), Flexible Decay (vector-valued decay), Dynamic Dependence (data-dependent decay), and Generalized Eigenvalue (eigenvalues outside [0,1]). No prior architecture achieves all four simultaneously. The "Generalized Delta Rule" row—St = St-1(diag(wt) + zᵀt bt) + vᵀt kt—is a template that subsumes them all, and the paper's specific instantiation is RWKV-7, which sets zt = -κ̂t and bt = κ̂t ⊙ at to produce the stable but expressive form used in practice.

How This Paper Positions Itself

The paper positions RWKV-7 as the first architecture in this lineage to simultaneously achieve all four desiderata in Table 1: matrix-valued states, vector-valued data-dependent decay, vector-valued data-dependent in-context learning rate, and transition eigenvalues beyond [0,1] (including negative eigenvalues for certain configurations). This is not merely an incremental improvement—it is a qualitative expansion of the expressive toolkit available to recurrent architectures.

On the efficiency axis: RWKV-7 inherits the constant-time, constant-memory inference properties of its RNN predecessors, with training parallelism provided by the WKV kernel's chunked recurrent formulation (Section 8). The paper presents timing results showing that the RWKV-7 kernel is approximately 3× faster than RWKV-6 and scales linearly with sequence length, while Flash Attention v3 scales quadratically (Figure 9). This positions RWKV-7 as practical for deployment, not just a theoretical advance.

On the expressivity axis: The central theoretical contribution (Appendix D) proves that RWKV-7 can recognize all regular languages with a constant number of layers (Theorem 3) and can solve an NC¹-complete state-tracking problem (Theorem 2). This places it strictly above Transformers and diagonal SSMs in the Chomsky hierarchy, at least for fixed-computation inference (without chain-of-thought). The proof relies on the non-diagonal, input-dependent transition matrix—specifically, the ability to represent swap and copy matrices (Lemma 1 and Lemma 3) that edit the state in ways impossible for append-only or purely diagonal dynamics. The paper also notes concurrent work by Grazzi et al. (2024) showing the benefits of negative eigenvalues in linear RNNs; RWKV-7's transition matrix diag(wt) - κ̂ᵀ(at ⊙ κ̂t) can produce eigenvalues in [-1, 1] (Theorem 1, Appendix C), going beyond the [0,1] range of decay-only models.

On the practical scaling axis: The paper is transparent that RWKV-7's pre-trained models are not trained from scratch at maximum scale—computational constraints forced a "model upgrade" approach where pre-existing RWKV-5 and RWKV-6 checkpoints were converted to the RWKV-7 format and then continued training on additional data (Table 2). The 2.9B model has seen 5.6 trillion tokens total across its architectural lineage, but only 3.1 trillion of those were in the RWKV-7 format on the new data. Despite this, it matches or exceeds models like Qwen2.5 that were trained on 18 trillion tokens from scratch. The paper positions this as evidence that the architecture is substantially more sample-efficient than Transformers, with the implication that a from-scratch RWKV-7 trained on equivalent token budgets would show even larger gains.

On the open-science axis: All models, datasets, and training/inference code are released under Apache 2.0. The paper explicitly frames this as enabling reproduction, comparative study, and adoption—a positioning that acknowledges the field's current concentration of large-scale training capability in a few industrial labs. The release of Pile-trained models using the GPT-NeoX tokenizer specifically enables apples-to-apples architectural comparisons with other models trained on the same data.

In summary, RWKV-7 positions itself as a synthesis of theoretical expressivity advances (the generalized delta rule, negative eigenvalues, non-diagonal transitions) with practical engineering (efficient CUDA kernels, stable training recipes, model upgrading from predecessors), aimed at closing the remaining gap between RNN architectures and Transformers while maintaining the inference efficiency advantages that make RNNs attractive for long-context deployment.

3. Technical Approach

3.1 Reader Orientation

RWKV-7 "Goose" is a recurrent neural network architecture for sequence modeling — specifically, an autoregressive language model — that maintains a fixed-size matrix-valued "memory" state which it actively edits at each token rather than simply appending to or uniformly decaying.

What problem it solves: The fundamental challenge is that prior efficient architectures (linear attention, state space models, earlier RWKV versions) could either only add new information to their memory or could only globally decay old information — they could not selectively remove or replace specific stored associations. RWKV-7 solves this through a generalized delta rule: an update formula that learns at test time to map keys to values, using vector-valued gating to independently control how much old information is removed from each channel of the state and how much new information is written in. The "shape" of the solution is a diagonal-plus-rank-one transition matrix applied to a multi-headed matrix-valued state at every token, implementing what is effectively one step of stochastic gradient descent per token on a per-channel basis with decoupled removal and replacement keys.

3.2 Big-Picture Architecture (Diagram in Words)

RWKV-7's forward pass through one layer consists of five major stages. I name each component and its responsibility; Section 3.4 expands each into full detail.

1. Token Shift (temporal mixing of adjacent inputs): The current token x_t and the previous token x_{t-1} are linearly interpolated on a per-channel basis using learned mixing coefficients μ. This produces separate "shifted" representations for the receptance, key, value, decay, in-context learning rate, and gate pathways. The purpose is to give the model access to change information (the difference between consecutive tokens) within a single layer, enabling the formation of induction heads.

2. Weight Preparation (computing the control signals): The shifted representations pass through low-rank MLPs and linear projections to produce six per-token vectors: the decay w_t (how much old state to retain per channel, bounded to [0.545, 1]), the in-context learning rate a_t (how aggressively to replace stored values per channel, in (0,1)), the removal key κ̂_t (a unit-norm vector specifying which stored entries to target for removal), the replacement key k̃_t (specifying where to write the new value), the value v_t (the new content to store), and the receptance r_t (a query vector used to read from the state later). These are all data-dependent: the model decides at each token how much to forget, how aggressively to update, and what to target for editing, based on the input.

3. WKV State Evolution (the core memory update): The matrix-valued state wkv_t (one per head, dimension 64 × 64 in the released models) is updated via the generalized delta rule: wkv_t = wkv_{t-1} (diag(w_t) - κ̂_tᵀ(a_t ⊙ κ̂_t)) + v_tᵀ k̃_t. This means: first, apply per-channel decay to all stored values; second, subtract a rank-one matrix that removes the value associated with the removal key κ̂_t (weighted by the in-context learning rate a_t); third, add the new outer product of value v_t and replacement key k̃_t. The result is a state that encodes key-value associations and can edit them on a per-key, per-channel basis. This is computed in parallel across time using a WKV kernel (Section 8) that exploits the diagonal-plus-rank-one structure.

4. Bonus and Output (reading from the state): The receptance r_t (query) is applied to the WKV state to retrieve stored values: u_t = (r_t · (ρ ⊙ k̃_t)ᵀ) v_t is a "bonus" term that allows the model to attend to the current token's value without needing to first store it in the state. The main attention output is p_t = LayerNorm(r_t · wkv_tᵀ) + u_t. The result is gated by g_t and projected back to the model dimension.

5. Channel Mixing (feedforward with squared ReLU): A separate pathway applies a 1D convolution (token shift) to the input, projects to a 4×-larger hidden dimension, applies ReLU(x)², and projects back. This replaces the gated MLP (with separate "receptance" gate) used in prior RWKV versions, trading a slight reduction in expressivity for training and inference speed.

Information flows sequentially: embedding → LayerNorm → (Time Mix block → Channel Mix block) repeated for L layers → LayerNorm → output head. Each layer's Time Mix and Channel Mix blocks operate on the residual stream, with LayerNorm applied at their inputs. The WKV state is per-layer, per-head, and is the only component that carries information across timesteps; all other operations are per-token.

3.3 Roadmap for the Deep Dive

I structure the detailed technical breakdown as follows, motivated by the logical flow of information:

• First, the generalized delta rule and its specialization to RWKV-7's transition matrix (the mathematical core — what operation does the state perform at each token, and why is this form maximally expressive while remaining stable?). This establishes the recurrence that everything else serves.

• Second, the weight preparation pipeline (how the model computes w_t, a_t, κ̂_t, k̃_t, v_t, r_t, g_t from the input). This explains how data-dependent control signals are generated efficiently using low-rank MLPs and linear projections, and covers key design choices like value residual learning and the decoupling of removal and replacement keys.

• Third, the WKV state evolution in detail (the exact recurrence relation, the parallel formulation, and the properties of the transition matrix including eigenvalue analysis and stability guarantees). This is the engine of the architecture.

• Fourth, the bonus, readout, and output gating (how the model extracts information from the state and produces the layer output). This includes the purpose of the bonus term and the rationale for head-wise normalization.

• Fifth, the channel mixing module (the simplified feedforward network). I explain how it differs from prior RWKV versions and why the changes were made.

• Sixth, training configuration and model upgrading (the phased training schedule, dynamic batch sizing, the checkpoint conversion process from RWKV-5/6 to RWKV-7, and hyperparameter choices). This is essential for understanding the released models and for reproduction.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural design paper whose core idea is that the delta rule can be generalized with vector-valued, data-dependent gating and decoupled keys to produce a recurrent state update that is both more expressive than prior RNNs (capable of recognizing all regular languages) and more stable in practice (state entries remain bounded of order O(1) over long sequences). The released language models demonstrate that this architecture is highly sample-efficient, matching Transformer-based models trained on 3–5× more tokens.

The Generalized Delta Rule and RWKV-7's Transition Matrix

The paper frames the state update problem through a unifying template called the Generalized Delta Rule (Table 1):

$S_t = S_{t-1}(\text{diag}(w_t) + z_t^T b_t) + v_t^T k_t$

where S_t is the matrix-valued state after processing token t, w_t is a vector-valued decay (one decay coefficient per state column), z_t and b_t are vectors whose outer product z_t^T b_t produces a rank-one modification to the transition matrix, v_t is the value vector to be stored, and k_t is the key vector specifying where the value should be retrieved from.

What it computes: the new state S_t is the old state S_{t-1} transformed by a diagonal-plus-rank-one matrix and then added to the outer product of the current value and key. The diagonal part applies per-channel decay: each column j of the state is multiplied by w_{t,j}, independently reducing the magnitude of stored values on that channel. The rank-one part z_t^T b_t allows the transition to perform a targeted edit — it can add or remove content at specific "locations" in the state space determined by the interaction of z_t and b_t. The addition of v_t^T k_t stores the new key-value association.

Why this form: prior architectures (Table 1) are all special cases:

• Setting z_t or b_t to zero recovers pure decay-based linear attention (RWKV-5/6, RetNet, Mamba).
• Setting w_t = 1 (no decay), z_t = -√a · k_t, b_t = √a · k_t recovers the original DeltaNet: S_t = S_{t-1}(I - a k_t^T k_t) + a v_t^T k_t with scalar a.
• Setting w_t to a scalar and z_t, b_t proportional to k_t recovers Gated DeltaNet.

The generalized form decouples three things that are tied together in simpler formulations: (a) how much old information to globally decay (diag(w_t)), (b) what locations in the state to target for removal (z_t), and (c) what information to add to the state (v_t^T k_t). This decoupling gives the model the flexibility to, for example, aggressively decay some channels while precisely editing specific stored keys on other channels.

RWKV-7 specializes this template with a specific choice for z_t and b_t:

$z_t = -\hat{\kappa}_t, \quad b_t = \hat{\kappa}_t \odot a_t$

where \hat{\kappa}_t is a unit-norm removal key vector (see Weight Preparation below), a_t ∈ (0,1)^D is a vector-valued in-context learning rate, and ⊙ denotes element-wise multiplication.

Substituting these into the generalized delta rule produces the RWKV-7 state recurrence:

$wkv_t = wkv_{t-1}\left(\text{diag}(w_t) - \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t)\right) + v_t^T \tilde{k}_t$

where I now use wkv_t for the state (the paper's notation), \tilde{k}_t is the replacement key (potentially different from the removal key), and all vectors are per-head with dimension D/h = 64 in the released models.

What it computes: the new WKV state is obtained by (1) multiplying the old state element-wise by the decay vector w_t (reducing all stored values, with potentially different attenuation per channel), (2) subtracting the rank-one matrix \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t) from the decayed state (which removes, in a per-channel weighted manner, the value associated with the key \hat{\kappa}_t), and (3) adding the outer product of the new value v_t and the replacement key \tilde{k}_t (which stores the new association).

Why this form specifically: the transition matrix G_t = \text{diag}(w_t) - \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t) has several crucial properties established in Appendix C (Theorem 1):

1. Stability: When c = 1 (the default multiplier for the rank-one term), all eigenvalues of G_t lie in (-1, 1), and the matrix G_t is a contraction. This means the state does not explode over long sequences — a critical property for training stability and numerical precision.

2. Expressivity beyond decay: Unlike pure decay models whose eigenvalues are in [0, 1], G_t can have a negative eigenvalue (at most one). This allows the state update to oscillate — to represent alternating patterns — which is impossible with purely positive eigenvalues. The negative eigenvalue corresponds to the removal direction: if a_t is large in some channels, the rank-one subtraction can flip the sign of the state along the \hat{\kappa}_t direction.

3. Similarity to a symmetric matrix: G_t is similar to the symmetric matrix \text{diag}(w_t) - c(\hat{\kappa}_t \text{diag}(a_t)^{1/2})^T (\hat{\kappa}_t \text{diag}(a_t)^{1/2}), which guarantees real eigenvalues and simplifies stability analysis. This is not just a theoretical nicety — it means the transition does not introduce complex rotations that could destabilize training.

4. Approximation to a Householder reflection: The paper notes that G_t = (I - \hat{\kappa}_t^T (a_t / w_t \odot \hat{\kappa}_t)) \text{diag}(w_t). The parenthesized term is a scaled approximate Householder matrix (a proper Householder would have a_t / w_t = 2 everywhere). Householder matrices implement reflections — they can "flip" state components — and this connection explains why RWKV-7 can implement swap and copy operations (Appendix D) that pure decay models cannot.

The paper also provides a parallel formulation of the recurrence (Equation 18):

$wkv_t = \sum_{i=1}^{t} \left( v_i^T \tilde{k}_i \prod_{j=i+1}^{t} \left( \text{diag}(w_j) - \hat{\kappa}_j^T (a_j \odot \hat{\kappa}_j) \right) \right)$

where the product \prod denotes cumulative matrix multiplication from right to left (later timesteps applied on the right).

What this computes: the state at time t is the sum over all previous positions i of the value-key outer product at time i, transformed by the product of all transition matrices from time i+1 to t. This is analogous to the parallel prefix sum formulation of linear attention, but with non-diagonal transition matrices. The parallel form is what enables efficient training: the WKV kernel computes this using chunked associative scans that exploit the diagonal-plus-rank-one structure (Yang et al., 2024c).

Weight Preparation: Computing Control Signals from Input

The weight preparation stage transforms the input token x_t (after LayerNorm and token shift, dimension D) into the vectors that control the WKV state evolution and readout. The paper uses D for model dimension, h = D/64 for the number of heads, and V for vocabulary size. All operations in this subsection are at full model dimension D; head splitting occurs later in the WKV kernel.

Token Shift (temporal mixing of adjacent inputs):

For each of six pathways □ ∈ {r, k, v, d, a, g} (receptance, key, value, decay, in-context learning rate, gate), the model computes a shifted representation by linearly interpolating between the current token x_t and the previous token x_{t-1}:

$x_t^\square = \text{lerp}(x_t, x_{t-1}, \mu_\square) = x_{t-1} + (x_t - x_{t-1}) \odot \mu_\square$

where \mu_\square \in \mathbb{R}^D is a learned per-channel mixing coefficient (not data-dependent — this is a simplification from RWKV-6, which used data-dependent token shift via low-rank MLPs).

What it computes: each shifted representation is a weighted average of the current and previous embedding, with a separate learned weight per channel per pathway. Channels with \mu_\square ≈ 1 strongly favor the current token; channels with \mu_\square ≈ 0 strongly favor the previous token.

Why this form (design choice: removing data-dependent token shift): RWKV-6 introduced data-dependent token shift where \mu was predicted from the input via a low-rank MLP. The paper states that while this was "beneficial in terms of loss decrease per step," the authors made "the judgement call that the improvement in training and inference efficiency was not worthwhile." By reverting to static learned \mu parameters (as in RWKV-4 and RWKV-5), the model sacrifices a small amount of per-step expressivity for faster training and simpler CUDA kernels. The token shift still provides the essential capability: within a single layer, the model can compare adjacent tokens to detect transitions, which is necessary for forming induction heads (Elhage et al., 2021; Olsson et al., 2022).

The paper notes (Appendix F) that token shift "is a variety of 1D short convolution" and is analogous to the short convolutions used in Mamba (Dao AI Lab, 2023).

In-Context Learning Rate a_t:

The model computes a vector-valued in-context learning rate from the shifted input x_t^a:

$a_t = \text{sigmoid}\left(\text{loramlp}_a(\text{Identity}, x_t^a, \text{bias=True})\right)$

where \text{loramlp}_a is a 2-layer MLP with a small hidden dimension d_a (Table 16: 64 for 768-dim model, 96 for 2048/2560-dim models, 128 for 4096-dim). The \text{Identity} argument indicates no activation function in the first projection; the second projection is linear. The bias=True flag means a bias term is added to the output before the sigmoid.

The low-rank MLP template (Equation 2) is:

$\text{loramlp}_\square(f, x, \text{bias}) = f(x A_\square) B_\square + (\lambda_\square \text{ if bias else } 0)$

where A_\square \in \mathbb{R}^{D \times d_\square} reduces the input to a small hidden dimension, B_\square \in \mathbb{R}^{d_\square \times D} expands back to D, f is an activation (Identity, tanh, or sigmoid depending on the pathway), and \lambda_\square \in \mathbb{R}^D is an optional learned bias. The total parameters in such a module are 2D d_\square + D (with bias) or 2D d_\square (without). This is much fewer than a full D \times D linear layer (D^2), which is important because RWKV-7 uses four such low-rank MLPs per layer.

What it computes: a_t \in (0, 1)^D determines, independently per channel, what fraction of the old value stored at the removal key \hat{\kappa}_t should be removed from the state. A value near 1 means "aggressively remove"; a value near 0 means "barely remove." Crucially, this is vector-valued: different channels can have different removal rates. For example, if a stored association has some dimensions that are still relevant and others that are outdated, the model can set a_t high on the outdated channels and low on the relevant ones.

Why this form (design choice: vector-valued vs. scalar ICLR): All prior delta-rule models (DeltaNet, Gated DeltaNet, TTT, Titans) use a scalar in-context learning rate — the same removal amount for all state dimensions. The paper argues this is unnecessarily restrictive: "each key channel in the state to vary independently." The sigmoid activation ensures a_t ∈ (0, 1), keeping the transition matrix stable (Theorem 1 requires this for the eigenvalue bound). The low-rank MLP provides data-dependence: the model can decide how aggressively to edit based on the current token's content.

Key Precursor k_t:

$k_t = x_t^k W_k$

where W_k \in \mathbb{R}^{D \times D} is a learned weight matrix and x_t^k is the key-shifted input. This produces the base key representation from which both the removal key and replacement key are derived.

Removal Key κ̂_t (normalized, targeted removal):

$\kappa_t = k_t \odot \xi, \quad \hat{\kappa}_t = \kappa_t / \|\kappa_t\|_2$

where \xi \in \mathbb{R}^D is a learned removal key multiplier vector (Appendix L reports its range in the trained 2.9B model as approximately [-5.3, 9.4]). The element-wise product k_t \odot \xi selectively amplifies or attenuates each channel of the key for removal purposes. The resulting vector is L2-normalized per head (\|\hat{\kappa}_t\|_2 = 1).

What it computes: \hat{\kappa}_t is a unit-length direction in the head's key space (\mathbb{R}^{64}) that specifies which stored value to target for removal. Conceptually, if the state wkv_{t-1} contains values mapped from various keys, multiplying by \hat{\kappa}_t^T extracts the value stored at (or near) the key \hat{\kappa}_t. The rank-one subtraction \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t) wkv_{t-1} removes that extracted value from the state, weighted per-channel by a_t.

Why this form: the normalization is critical for the delta rule mechanism. The outer product \hat{\kappa}_t^T \hat{\kappa}_t has eigenvalues of either 1 (in the \hat{\kappa}_t direction) or 0 (orthogonal directions). Without normalization, the removal amount would depend on \|k_t\|^2, which varies widely across tokens, potentially causing catastrophic removal when key norms are large. The multiplier \xi is learned during training and determines which dimensions of the key are most relevant for locating stored values; the paper provides statistics (Figures 17-20) showing that \xi varies substantially across layers and models, suggesting it captures non-trivial learned structure.

Replacement Key k̃_t (decoupled from removal key):

$\tilde{k}_t = k_t \odot \text{lerp}(1, a_t, \alpha)$

where \alpha \in \mathbb{R}^D is a learned replacement rate booster (range in trained models: roughly [0, 1], see Appendix L, Figures 17-18), and \text{lerp}(1, a_t, \alpha) = 1 + (a_t - 1) \odot \alpha = 1 - \alpha + \alpha \odot a_t. The element-wise product k_t \odot \text{lerp}(1, a_t, \alpha) scales each channel of the key by a value between a_t (when \alpha = 1) and 1 (when \alpha = 0), modulated by the learned booster.

What it computes: \tilde{k}_t is the key that specifies where to write the new value v_t into the state, via the outer product v_t^T \tilde{k}_t. The amount of key written to each channel is a learned interpolation between the "full" key k_t and the ICLR-scaled key a_t \odot k_t. When \alpha = 0 everywhere, \tilde{k}_t = k_t (full key written). When \alpha = 1 everywhere, \tilde{k}_t = a_t \odot k_t (only the ICLR fraction is written, meaning the amount written matches the amount removed).

Why this form (design choice: decoupling removal and replacement): In the original delta rule, the same key k_t is used both to remove old values (via k_t^T k_t) and to write new ones (via v_t^T k_t). RWKV-7 decouples these two uses: \hat{\kappa}_t (unit-norm, multiplied by \xi) is the removal key; \tilde{k}_t (unnormalized, boosted by \alpha) is the replacement key. The paper argues this gives the model more flexibility: "the ability to use a different removal key than replacement key" (Section 3). A concrete scenario: the model might recognize that the current token relates to a stored fact at key location A, but wants to update a related but different key location B. With decoupled keys, \hat{\kappa}_t can target location A for removal while \tilde{k}_t writes to location B.

The paper further explains (Section 4.1) the motivation for the specific form of \tilde{k}_t: earlier experimental versions (RWKV-6c, a sub-version with no trained models) set \tilde{k}_t = k_t \odot (1 - w_t) to ensure that the amount added never exceeded the amount removed by decay. RWKV-7 instead gives the model "enough mathematical leeway to make these decisions on its own" by introducing the learned booster \alpha. The model can learn to balance removal and replacement amounts end-to-end during training rather than having it enforced by architectural constraints.

Decay w_t (per-channel, data-dependent, bounded):

The decay computation involves two steps:

$d_t = \text{loramlp}_d(\text{tanh}, x_t^d, \text{bias=True})$

$w_t = \exp\left(-e^{-0.5} \cdot \text{sigmoid}(d_t)\right)$

where d_t \in \mathbb{R}^D is an unconstrained "decay precursor" produced by a low-rank MLP with tanh activation (hidden dimension d_w, Table 16), and w_t \in (\exp(-e^{-0.5}), 1) = (0.545..., 1) is the per-channel decay factor.

What it computes: w_t determines, per channel, what fraction of the old state is retained before applying the delta-rule edit. A value of 1 means "retain everything" (no decay); a value of 0.545... means "retain at most 54.5%" (maximum decay, removing up to 45.5% of the stored value). The decay is applied as multiplication by \text{diag}(w_t) on the right: wkv_{t-1} \text{diag}(w_t) scales each column j of the WKV state by w_{t,j}.

Why this form: the double-exponential \exp(-e^{-0.5} \cdot \text{sigmoid}(d_t)) is described as "a rearrangement of an original source formula \exp(-\exp(-0.5 - \text{softplus}(d_t)))" (Appendix F). The outer \exp(-\exp(x)) is "nearly a flipped version of sigmoid, but with a better gradient." The -0.5 in the exponent clamps the inner exponential to be ≤ e^{-0.5}, which after the outer exponential gives w_t ≥ \exp(-e^{-0.5}) ≈ 0.545. The paper states that this lower bound is necessary "to maintain training stability and to assist the creation of fast but numerically stable kernel implementations." Without this clamp, w_t could approach 0, effectively erasing all stored information in one step and causing gradient vanishing. The bound of 0.545 means decay can remove at most 45.5% of pre-existing values per timestep; more aggressive removal is handled by the delta rule mechanism (\hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t) subtraction). The paper notes the desire to "reduce the decay limit further in future revisions" if numerical stability permits.

Value v_t (with residual learning from layer 0):

The value computation is more involved because it incorporates value residual learning (Zhou et al., 2024) across layers:

First, a shared value precursor is computed:

$\nu_t = \text{sigmoid}\left(\text{loramlp}_\nu(\text{Identity}, x_t^v, \text{bias=True})\right)$

$v'_{t,l} = x_t^v W_v$

where v'_{t,l} \in \mathbb{R}^D is the layer-specific value precursor (linear projection of the shifted input), and \nu_t \in (0, 1)^D is a per-channel interpolation gate.

The actual value v_t depends on the layer index l:

$v_t = \begin{cases} v'_{t,0} & \text{if } l = 0 \\ \text{lerp}(v'_{t,0}, v'_{t,l}, \nu_t) = v'_{t,0} + (v'_{t,l} - v'_{t,0}) \odot \nu_t & \text{if } l \geq 1 \end{cases}$

What it computes: the value at layer 0 (l = 0) is directly the projected input v'_{t,0}. For deeper layers, v_t is a per-channel interpolation between the layer-0 value precursor v'_{t,0} and the current layer's value precursor v'_{t,l}, controlled by the gate \nu_t. When \nu_{t,j} ≈ 1, channel j uses the current layer's computed value; when \nu_{t,j} ≈ 0, it uses the first layer's value.

Why this form: value residual learning addresses "attention concentration" (Zhou et al., 2024) — the tendency of deeper layers in Transformers to attend to the same positions as earlier layers, reducing diversity. By allowing deeper layers to directly access the first layer's value representation (via setting \nu_t ≈ 0), the model can choose to bypass intermediate layer transformations when the original value is already suitable. The paper states this "has shown to improve the final language modeling loss." The first layer is treated specially (no residual, v_t = v'_{t,0}) because there is no earlier layer to reference.

Receptance r_t (the query vector):

$r_t = x_t^r W_r$

where W_r \in \mathbb{R}^{D \times D} is a learned weight matrix. Unlike in RWKV-6, there is no low-rank MLP or sigmoid gating on the receptance; it is a straightforward linear projection of the shifted input.

Gate g_t (output gating):

$g_t = \text{loramlp}_g(\text{sigmoid}, x_t^g, \text{bias=False})$

where the low-rank MLP uses sigmoid activation (so output is in (0, 1)^D because of the sigmoid) and no bias term. This gate controls how much of the attention output is passed to the output projection.

Summary of design philosophy for weight preparation: four of the six control signals (a_t, w_t, \nu_t, g_t) are produced by low-rank MLPs, meaning they use only O(D · d_\square) parameters instead of O(D^2). The paper explicitly states this is to "implement data dependency using minimal parameters" (Section 4.1). The two signals that carry the main information content — keys and values — use full-rank linear projections (W_k, W_v, W_r). This creates a division of labor: high-capacity linear projections move information, while parameter-efficient low-rank networks control the flow (how much, where, and when).

WKV State Evolution in Detail

After weight preparation, all vectors (r, w, \tilde{k}, v, \hat{\kappa}, a)_t are reshaped to split them into h heads, each of dimension D/h = 64 (a constant in all released models). The WKV state wkv_t is per-head, shape (64, 64), initialized to the zero matrix at the start of each sequence.

State normalization before the transition: within each head, the removal key is L2-normalized:

$\hat{\kappa}_t = \kappa_t / \|\kappa_t\|_2$

where the norm is computed over the 64-dimensional head slice. This ensures \|\hat{\kappa}_t\| = 1 per head, which is necessary for the outer product \hat{\kappa}_t^T \hat{\kappa}_t to represent a projection operator.

The recurrence (Equation 17):

$wkv_0 = 0$

$wkv_t = wkv_{t-1} \left( \text{diag}(w_t) - \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t) \right) + v_t^T \cdot \tilde{k}_t$

where all vectors are per-head (dimension 64), \text{diag}(w_t) is a 64 × 64 diagonal matrix with w_t on the diagonal, \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t) is a 64 × 64 rank-one matrix (outer product of \hat{\kappa}_t^T with a_t \odot \hat{\kappa}_t), and v_t^T \tilde{k}_t is a 64 × 64 rank-one matrix (outer product of v_t^T with \tilde{k}_t).

What it computes (step by step):

1. wkv_{t-1} \text{diag}(w_t): Each column j of the previous state is multiplied by w_{t,j}, the per-channel decay for that column. This globally attenuates old stored values.

2. wkv_{t-1} \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t): The vector \hat{\kappa}_t^T (shape 64 × 1) selects a linear combination of the columns of wkv_{t-1} — this is the "value stored at key \hat{\kappa}_t." The outer product with (a_t \odot \hat{\kappa}_t) (shape 1 × 64) distributes this extracted value across all columns, weighted per-channel by a_t. The result is subtracted from the decayed state. This implements targeted removal: the amount removed is proportional to (a) how strongly the state encodes the key \hat{\kappa}_t (via the projection wkv_{t-1} \hat{\kappa}_t^T) and (b) the per-channel removal willingness a_t.

3. + v_t^T \tilde{k}_t: The new value v_t is associated with the replacement key \tilde{k}_t by adding their outer product. When later queried with a receptance r that matches \tilde{k}_t, the state will retrieve (a scaled version of) v_t.

How this compares to the original delta rule: In DeltaNet's S_t = S_{t-1}(I - a k_t^T k_t) + a v_t^T k_t, the removal key and replacement key are identical (k_t), and the learning rate a is a scalar. The removal amount is a · S_{t-1} k_t^T k_t — the state projected onto k_t, then subtracted from all columns uniformly scaled by a. RWKV-7's version allows: different removal and replacement keys (\hat{\kappa}_t vs. \tilde{k}_t), per-channel removal amounts (a_t is a vector, not a scalar), and per-channel decay (w_t applied before the delta rule edit).

The transition matrix and its eigenvalues (Appendix C, Theorem 1):

The transition matrix is:

$G_t = \text{diag}(w_t) - \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t)$

Theorem 1 (proved in Appendix C) establishes:

1. Similarity to a symmetric matrix: G_t is similar to B = \text{diag}(w_t) - c(\hat{\kappa}_t \text{diag}(a_t)^{1/2})^T (\hat{\kappa}_t \text{diag}(a_t)^{1/2}) via conjugation by \text{diag}(a_t)^{1/2}. Since B is symmetric (a diagonal minus a rank-one symmetric update), G_t has real eigenvalues.

2. Eigenvalue range: All eigenvalues of G_t lie in (-1, 1) when c = 1, w_t ∈ (u, 1) with u = \exp(-e^{-1/2}) ≈ 0.545, and a_t ∈ (0, 1). This is proved by bounding the Rayleigh quotient of B: for any unit vector \hat{s}, \hat{s} B \hat{s}^T ≥ u - c \|\hat{\kappa}_t \text{diag}(a_t)^{1/2} \hat{s}^T\|^2 ≥ u - c > -1 and \hat{s} B \hat{s}^T < 1 - c \|\hat{\kappa}_t \text{diag}(a_t)^{1/2} \hat{s}^T\|^2 ≤ 1.

3. At most one negative eigenvalue: Via Sylvester's law of inertia, B is congruent to I - u^T u (where u = \sqrt{c} \hat{\kappa}_t \text{diag}(a_t)^{1/2} \text{diag}(w_t)^{-1/2}), which has exactly one eigenvalue potentially less than 1 (specifically 1 - \|u\|^2) and all others equal to 1. Hence G_t has at most one eigenvalue below 1, which can be negative if \|u\|^2 > 1.

4. Stability (contraction): For time-independent a_t = a, the product of transition matrices is bounded: \|\prod_{t=1}^T G_t\|_2 ≤ \min(a)^{-1/2}. This means the state cannot explode over time.

The note on c = 1 vs. c = 2: The paper uses c = 1 for language modeling (the default multiplier on the rank-one term). Some constructions in Appendix D use c = 2 to enable specific operations (swap and copy matrices) that require eigenvalues of -1. The paper explains (Appendix D.3) that the model with c = 1 can simulate c = 2 by halving both c and w_t, which scales the transition matrix by ½. Since the wkv heads are immediately followed by group normalization, the magnitude scaling does not affect subsequent computations (only directions matter). The paper notes that "floating point numbers store a separate exponent, this rescaling only requires log-precision."

State dimension and information capacity: In all released models, the head dimension is fixed at D_h = 64, so the state per head is 64 × 64 = 4096 entries. The number of heads is D/64. For the 2.9B model (D = 2560), there are 40 heads, giving a total state size of 40 × 64 × 64 = 163,840 scalar entries per layer, or 32 × 163,840 = 5,242,880 entries across all 32 layers (Table 15). The paper reports (Section 7.3) that on the MQAR associative recall task with WKV state dimension 8192 (likely a smaller model configuration), the model achieves an information density of 0.547 bits per dimension, storing approximately 4480.8 bits in the state.

Bonus, Readout, and Output Gating

After updating the WKV state, the model reads from it to produce the attention output. All operations are per-head.

Bonus term (current token attention without state storage):

$u_t = \left( r_t \cdot (\rho \odot \tilde{k}_t)^T \right) v_t$

where \rho \in \mathbb{R}^{D/h} is a learned per-head bonus multiplier and r_t \cdot (\rho \odot \tilde{k}_t)^T is the dot product between the receptance (query) r_t and the boosted replacement key \rho \odot \tilde{k}_t. The result is a scalar per head, which is multiplied element-wise by the value v_t.

What it computes: u_t is a contribution that allows the current token's value v_t to influence the output directly, without first being stored in and retrieved from the WKV state. The receptance r_t is compared to the replacement key \tilde{k}_t (boosted by \rho), and if they match (high dot product), the current value is emphasized in the output.

Why this form: the paper states this "resembles the design of 'time-first' u term existing from RWKV-4 to RWKV-6, under the belief that the information of the current token deserves special treatment." In previous RWKV versions, this "time-first" term was fused inside the WKV kernel; extracting it simplifies the kernel implementation. The \rho multiplier functions similarly to the u parameter in RWKV-4/5/6 — a learned scaling that determines how much the current token's value contributes relative to the state-retrieved value. The paper states (Equation 20) that this is applied per head.

Main attention readout:

$p_t = \text{LayerNorm}(r_t \cdot wkv_t^T) + u_t$

where r_t \cdot wkv_t^T is the matrix-vector product of the receptance r_t (shape 64) with the transposed WKV state wkv_t^T (shape 64 × 64), producing a vector of dimension 64 per head. The LayerNorm is applied per head (implemented as GroupNorm with num_groups = h groups, eps = h × 1e-5).

What it computes: r_t \cdot wkv_t^T retrieves the value stored in the state that best matches the query r_t. Since the state stores values via outer products v_i^T \tilde{k}_i, retrieving with r_t computes (approximately) a weighted sum of stored values v_i, weighted by the similarity r_t · \tilde{k}_i (after accounting for decay and delta-rule edits). The LayerNorm normalizes this retrieved vector to unit variance across the 64 dimensions of the head, removing the effect of the state's numerical scale (which can grow or shrink over time due to decay and delta-rule edits). The bonus u_t is then added.

Why LayerNorm per head: the paper explains that this "is a way of ensuring that change in the numerical size of the state over time does not impact the model's ability to use the state" (Appendix F). Earlier RWKV versions (RWKV-4) used a denominator term inside the attention computation to normalize; the switch to LayerNorm/GroupNorm is "less costly, easier to code, and uses less memory."

Head recombination and output gating:

After computing p_t for all heads, the per-head vectors are concatenated back to dimension D. The output is gated by g_t and projected:

$o_t = (g_t \odot p_t) W_o$

where W_o \in \mathbb{R}^{D \times D} is the output projection matrix, and \odot is element-wise multiplication. The result o_t \in \mathbb{R}^D is added to the residual stream.

Why gating (design choice: g_t as a data-dependent gate): the gate g_t ∈ (0, 1)^D (computed via sigmoid-activated low-rank MLP) controls how much of the attention output is passed through to the residual stream. This is analogous to the output gate in an LSTM or the gating in Gated Linear Units. The paper notes that unlike RWKV-6, the receptance does not gate the WKV readout directly; instead, a separate gate g_t is learned specifically for output modulation. This decouples the query mechanism (receptance for retrieval) from the output modulation (gate for flow control).

Channel Mixing Module (Simplified Feedforward Network)

The channel mixing module is a per-token feedforward network that replaces the Channel Mixing module of previous RWKV architectures. The key change is the removal of a separate "receptance" gate, making it a simpler two-layer MLP.

Token shift for channel mixing:

$k'_t = \text{lerp}(x'_t, x'_{t-1}, \mu'_k) W_{k'}$

where x'_t is the input to the channel mixing block (after the residual addition from Time Mix and its own LayerNorm), x'_{t-1} is the previous token's channel-mix input, \mu'_k \in \mathbb{R}^D is a learned mixing coefficient, and W_{k'} \in \mathbb{R}^{D \times 4D} is a projection matrix that expands the dimension by 4×.

What it computes: a temporally-mixed representation (combining current and previous token) is projected to a 4× larger dimension. This is analogous to the W_k projection in RWKV-6's Channel Mix but without a separate W_r (receptance) projection.

Squared ReLU activation and output projection:

$o'_t = \text{ReLU}(k'_t)^2 W_{v'}$

where \text{ReLU}(x) = \max(0, x) is applied element-wise, the result is squared, and W_{v'} \in \mathbb{R}^{4D \times D} projects back to the model dimension.

What it computes: the expanded representation is thresholded at zero (all negative values become zero), then squared (which amplifies larger positive values relative to smaller ones), and projected back. The squared ReLU (\text{ReLU}^2) is used throughout the RWKV lineage; it is a smooth alternative to GELU that is faster to compute and has shown good empirical performance.

Why this change from RWKV-6: previous RWKV versions used a gated MLP with a "receptance" gate: o' = \sigma(r) \odot \text{ReLU}(k)^2 W_v, where r was a separate linear projection and \sigma is sigmoid. RWKV-7 removes this gate entirely, simplifying to a single linear projection W_{k'} before the activation. The paper states (Section 4.2): "We remove the gating matrix W_r, making it a two-layer MLP. In compensation for the removed gating parameters to satisfy the equi-parameter condition, we set the hidden dimension to be 4 times the size of model dimension." In RWKV-6, the hidden dimension was 3.5×; the expansion to 4× recovers the parameter count lost by removing the gate. This trades the dynamic gating capability for faster training and inference (one less matrix multiply, one less sigmoid), while keeping the total parameter count similar.

The paper also notes (Appendix F) that the model modification process for upgrading from RWKV-6 checkpoints involved "widen[ing] the FFN MLP from 3.5x (in RWKV-6) to 4x and add[ing] new small uniform initializations in the new regions, removing the RWKV-6 FFN receptance weights."

Training Configuration and Model Upgrading

Data loading with pseudo-random sampling:

The dataset of 3.119 trillion tokens is memory-mapped using mmap. To ensure diverse, pseudo-random access without replacement within one epoch, the paper uses a modular arithmetic trick. The k-th sample's starting address is:

$\text{start\_address} = 4096 \cdot (a k^3 \bmod p)$

where p = 761521949 is the largest prime of the form 3n + 2 smaller than ⌊\text{dataset\_size}/4096⌋, and a is an integer close to 0.618p chosen to ensure good mixing. The sequence length for pretraining is 4096 tokens, and the sample spans [start_address, start_address + 4097) (the extra token is for the target in next-token prediction).

Why this form: the cubic function f(k) = a k^3 over the finite field \mathbb{Z}/p\mathbb{Z} is a bijection when p is a prime of the form 3n + 2 (since 3 is then invertible modulo p-1), ensuring every index in [0, p) is visited exactly once per epoch. The cubic mapping provides better pseudo-random mixing than a linear congruential generator, and the arithmetic is simple to compute. The paper notes this "guarantees both simple calculation and uniform access to the dataset while maintaining pseudo-randomness."

Model upgrading from RWKV-5/6 checkpoints:

The paper describes this process in Section 6 and Appendix E. Because of compute budget constraints, the 0.1B and 0.4B World models started from RWKV-5 checkpoints, while the 1.5B and 2.9B World models started from RWKV-6 checkpoints. The conversion process involves:

• Removing token-shift low-rank MLPs: RWKV-6 used data-dependent token shift via low-rank MLPs; these are removed since RWKV-7 uses static token shift.
• Rescaling by half: embeddings, wkv receptance, wkv output matrix weights, and LayerNorm/GroupNorm bias values are halved.
• LayerNorm/GroupNorm adjustment: weights are clamped above zero and square-rooted.
• FFN widening: the channel mixing MLP is expanded from 3.5× to 4× hidden dimension, with new parameters initialized uniformly with small values (1 × 10^{-3}).
• Time decay low-rank MLP widening: the low-rank MLP for decay receives additional parameters, initialized uniformly with 1 × 10^{-4}.
• Gate weight replacement: the gate weights are replaced with a LoRA obtained through singular value decomposition and rescaled by half.

This procedure allows reusing previously trained knowledge while adapting to the new architecture, reducing total training cost. The paper acknowledges that training from scratch on the full dataset would likely yield better results: "We theorize that if we were less constrained by compute and were able to train these models from scratch with the same amount of total tokens instead of from pre-trained checkpoints of earlier RWKV versions, the difference would be even more dramatic" (Section 7.1).

Training hyperparameters and schedule (Table 17, Appendix E):

All models use:

• Precision: bfloat16.
• Hardware: nodes of 8 × Nvidia H800 GPUs, with total GPU count ranging from 8 (0.1B phase 1) to 96 (2.9B phase 4).
• Optimizer: AdamW with β₁ = 0.9, β₂ = 0.99, ϵ = 1 × 10^{-18}, weight decay 0.1 applied only to linear layers and embedding weights.
• The extremely small ϵ: chosen based on Molybog et al. (2023), which argues that smaller ϵ stabilizes training in large-scale models "by ensuring that intermediate layers remain in a regime of active updates, thus mitigating sudden loss spikes and promoting smoother convergence."
• Weight decay exception: the base decay rate w_0 parameters (learned initial values for the decay mechanism) are placed in a special 2× learning rate multiplier group.
• Context length: 4096 tokens for pretraining.
• Vocabulary size: 65536 for World models (RWKV World tokenizer), 50304 for Pile models (GPT-NeoX 20B tokenizer).
• Head size: D_h = 64 for all models, giving h = D/64 heads.

Phased training with dynamic batch sizing:

Rather than a single cosine decay over the entire training run, the paper uses a multi-phase schedule where both batch size and learning rate change across phases (Table 17). For the 2.9B model:

• Phase 1: 4 nodes, batch size 640 × 4096, initial LR 4 × 10^{-4}
• Phase 2: 6 nodes, batch size 1008 × 4096, initial LR 5 × 10^{-4}
• Phase 3: 7 nodes, batch size 1120 × 4096, initial LR 5.4 × 10^{-4}
• Phase 4: 12 nodes, batch size 2016 × 4096, initial LR 8 × 10^{-4}

Within each phase, the learning rate follows a cosine decay from the phase's initial LR to the final LR of 1 × 10^{-5} (which is the target for the end of the entire run). The "implied initial rate varies across phases" because each phase starts from the previous phase's ending LR.

Why phased training: the paper cites McCandlish et al. (2018) on the critical batch size concept and Smith et al. (2018) on increasing batch size during training. The intuition is that larger batch sizes reduce gradient noise, allowing higher learning rates to be used stably. By progressively increasing batch size (and commensurately increasing LR), the model can train faster in later stages when gradients are smaller. The paper also notes a practical benefit: "after smaller models complete their training, additional GPU resources become available for the later stages of training larger models. This cascading resource allocation ensures that computational power is dynamically reallocated, maximizing hardware utilization and reducing idle time."

Training stability:

The paper reports "extremely stable training without any loss spikes in all four runs" (Figure 12 shows smooth loss curves). However, the authors "did sometimes observe NaN loss across a single training step," which they attribute to the extremely low AdamW ϵ. The mitigation strategy: "rewind the training to the prior checkpoint, clear optimizer states, and continue from that point."

Parameter initialization:

The paper emphasizes that "proper parameter initialization is crucial for ensuring training stability and achieving optimal performance" and that "using the recommended initialization is essential for replicating the results in this paper." The detailed initialization scheme is deferred to the official code repository, but the paper mentions specific choices: uniform initialization with scale 1 × 10^{-3} for new FFN parameters and 1 × 10^{-4} for new decay low-rank MLP parameters in the model upgrade process.

Low-rank MLP dimensions (Table 16):

The intermediate dimensions for the four low-rank MLPs scale sub-linearly with model dimension:

The paper states these values are "based on our mere speculation of how much information can be passed through," acknowledging the heuristic nature of these choices. The gate low-rank MLP (d_g) uses larger hidden dimensions than the others, presumably because output gating requires more capacity. The value residual low-rank MLP (d_v) uses the smallest hidden dimensions (32–96), consistent with its role as an interpolation weight rather than a content generator.

Total parameter count formula (Equation 26):

The paper provides an exact formula for the parameter count of an RWKV-7 model:

$\#(\text{Params}) = 2DV + 4D + LD\left(12D + 2(d_w + d_a + d_v + d_g) + 19\right) - (2D d_v + D)$

where the terms are:

• Embedding and head: 2DV + 4D (embedding matrix V × D, output head D × V, plus 4 LayerNorm parameters 2 × 2D)
• Per layer (L layers): D(12D + 2(d_w + d_a + d_v + d_g) + 19) — this accounts for the linear projections (W_r, W_k, W_v, W_o: 4 × D²), the low-rank MLPs (A and B matrices: 2 × D × d each), token shift parameters (μ vectors: 6 × D), and various biases and scalars
• First layer adjustment: -(2D d_v + D) — the value residual low-rank MLP is absent in the first layer (since there is no layer-0 value to reference), and one fewer LayerNorm bias term is needed

For the 2.9B model (D = 2560, V = 65536, L = 32, d_w = 96, d_a = 96, d_v = 64, d_g = 320), this yields approximately 2.95 billion parameters, consistent with the reported 2.9B (Table 15).

4. Key Insights and Innovations

Innovation 1: The Generalized Delta Rule as a Unifying Framework — and Why Vector-Valued, Decoupled Gating Crosses a Qualitative Threshold

The paper's most distinctive conceptual move is not simply adding vector-valued parameters where prior work used scalars. It is the systematic decomposition of the state transition into four independently controllable axes — per-channel decay (w_t), per-channel removal rate (a_t), targeted removal location (κ̂_t), and targeted replacement location (k̃_t) — and the demonstration that crossing from scalar to vector-valued control on all axes simultaneously crosses a qualitative threshold in expressivity, not merely an incremental capacity improvement.

What the field assumed before this work: The dominant framing, inherited from DeltaNet (Schlag et al., 2021) and carried through Gated DeltaNet (Yang et al., 2024a), TTT (Sun et al., 2024), and Titans (Behrouz et al., 2024), treated the in-context learning rate a and the decay w as scalar hyperparameters of an SGD-like update — conceptually analogous to the global learning rate and weight decay in neural network training, where a single number governs all parameters uniformly. Under this framing, making a data-dependent (scalar but input-conditioned) was a natural extension, and Gated DeltaNet achieved this. But the dimensionality of the control — scalar vs. vector — was treated as an implementation detail, not a fundamental property.

RWKV-7's architecture argues, implicitly through its design and explicitly through its theoretical results, that this framing is incorrect. A scalar learning rate, even if data-dependent, cannot edit different dimensions of a stored value independently. If the state associates a key k with a 64-dimensional value v, and some dimensions of v remain accurate while others are outdated, a scalar removal rate must compromise — either over-remove (damaging accurate dimensions) or under-remove (leaving stale information). The vector-valued a_t resolves this by making the tradeoff per-dimension. Similarly, a scalar decay cannot attend differentially to different aspects of stored information; vector-valued decay allows the model to maintain high-fidelity storage on some channels while aggressively forgetting on others.

Why this is a qualitative threshold, not an incremental improvement: The paper's theoretical results (Appendix D) depend critically on the interaction between vector-valued decay, vector-valued ICLR, and the non-diagonal transition matrix. Lemma 1 (representing arbitrary permutation matrices) and Lemma 3 (factoring DFA transitions into elementary matrices) require the ability to target specific state dimensions independently — what the paper calls "swap" and "copy" operations on the state. A scalar learning rate can express the magnitude of an edit but not its shape across dimensions. The proof that RWKV-7 recognizes all regular languages (Theorem 3) collapses if a_t is forced to be scalar, because the construction requires per-column copy operations that modify specific state indices while leaving others untouched.

The paper's ablation (Appendix K, Table 19) provides empirical support: on a small-scale 6-layer, 768-dim model trained on 1.6B tokens, replacing vector-valued decay with scalar decay increases validation loss from 2.541 to 2.609 — a meaningful degradation, but perhaps survivable. Replacing vector-valued ICLR with scalar ICLR increases validation loss from 2.541 to 2.591. Removing the decoupling of removal and replacement keys increases loss to 2.560. Each individual ablation shows a modest effect. The architectural argument is that these components are synergistic: the ability to independently control decay, removal, and replacement per-channel and with different key representations is what enables the state update to be a general-purpose memory editor rather than merely a parameterized forget-and-add operation.

The Table 1 taxonomy as a diagnostic tool: The paper's organization of prior architectures along the four axes in Table 1 — Large State, Flexible Decay, Dynamic Dependence, Generalized Eigenvalue — is more than a literature summary. It functions as a diagnostic framework: it reveals that each prior architecture made a specific tradeoff, leaving one or more axes constrained. RWKV-7 is the first to simultaneously occupy all four. This framing makes the contribution legible as filling a conceptual gap rather than as a grab-bag of improvements.

Significance beyond raw performance: The vector-valued, decoupled gating design is significant less for any single benchmark number and more for establishing a design principle for future RNN architectures: control signals that govern state editing should match the dimensionality of the state they control. A scalar decay is to a 64×64 matrix-valued state what a single global learning rate is to a billion-parameter neural network — it can work, but it needlessly constrains the optimization. This principle, if adopted, would steer future work away from the scalar-control regime entirely.

Innovation 2: The NC¹ Expressivity Proof as a Complexity-Theoretic Justification for Non-Diagonal Transitions

The paper's second major contribution is a theoretical result that provides formal grounding for a design choice that might otherwise appear ad hoc: the rank-one (non-diagonal) component of the transition matrix. Theorem 2 proves that RWKV-7 can solve a problem (tracking permutations of five elements) that is NC¹-complete under AC⁰ reductions, placing it strictly above the TC⁰ complexity class that contains Transformers and diagonal state-space models (Merrill et al., 2024). Theorem 3 proves that RWKV-7 can recognize any regular language with a constant number of layers.

What the field assumed before this work: Merrill et al. (2024) established that Transformers and RNNs with purely diagonal transition matrices — which includes S4, Mamba, and all linear attention variants that use only scalar or vector decay without a rank-one edit — are confined to TC⁰, a relatively weak complexity class that cannot represent basic state-tracking operations like following a permutation. This result was interpreted by some as a fundamental limitation of efficient recurrent architectures: you could have constant-time inference, or you could have expressive state tracking, but not both. Classical RNNs (with dense transition matrices) are in NC¹ and can recognize all regular languages, but are not parallelizable and suffer from vanishing gradients.

What RWKV-7 changes about this picture: The diagonal-plus-rank-one structure of the RWKV-7 transition matrix is a minimal perturbation to diagonal dynamics that escapes the TC⁰ expressivity ceiling. The rank-one term κ̂_tᵀ(a_t ⊙ κ̂_t) adds exactly enough off-diagonal interaction to implement the swap and copy operations that diagonal models cannot, while preserving the efficient parallelization properties (the WKV kernel exploits the diagonal-plus-rank-one structure via the techniques from Yang et al., 2024c). This is a principled answer to the question: "How much non-diagonality is enough?" The answer is: rank-one suffices to cross from TC⁰ to NC¹.

Why the proof matters beyond the theorem statement: The constructive proof of Theorem 3 is not merely an existence result — it demonstrates a specific mechanism by which RWKV-7 layers can compose to simulate a deterministic finite automaton. The construction uses the first three layers to convert blocks of DFA transitions into products of elementary matrices (swap, copy, identity), and the fourth layer to multiply these elementary matrices, tracking the DFA state in the WKV state. This reveals that the architecture's expressivity comes from its ability to factor complex state transitions into sequences of simple, learnable primitives — exactly the kind of composition that deep learning excels at.

The proof also clarifies the role of the MLP layers: the construction requires exponentially wide MLP layers (in the number of DFA states) to implement lookup tables for the elementary matrix selection. This is consistent with the practical architecture, where MLP hidden dimensions are 4× the model dimension — large enough to implement substantial lookup-table-like computation in a single layer.

Connection to the empirical state-tracking results (Section 7.6, Figure 8): The group multiplication experiment provides empirical validation of the theoretical hierarchy. RWKV-7 requires fewer layers than Transformers, Mamba, and S4 to achieve >95% validation accuracy on the S₅, A₄ × Z₅, and Z₆₀ group multiplication tasks — exactly as the theory predicts. Classical RNNs (not shown in the figure with the same directness, but described as requiring even fewer layers) sit at the top of the expressivity hierarchy, but the paper notes they "cannot be parallelized efficiently, unlike RWKV-7." So RWKV-7 occupies a sweet spot: strictly more expressive than Transformers and diagonal SSMs on state tracking, while retaining parallelizability that classical RNNs lack.

The significance of the NC¹ result for practical reasoning: The paper argues (Appendix F) that state-tracking capability is not merely a theoretical curiosity. It gives the concrete example: "if you give a Transformer an ordered set of items and a list of which ones swap places, it will not be able to tell you which item ends up in which position at the end" — without chain-of-thought. An architecture that can perform such tracking internally, within its fixed-size state, avoids the costly token-by-token generation of intermediate reasoning steps. This positions RWKV-7's expressivity advantage as directly relevant to inference-time efficiency for tasks that involve maintaining and updating structured representations.

Innovation 3: The State as an Editable Scratchpad — A Reframing of RNN Memory from "Decaying Trace" to "Learned Data Structure"

The paper's third conceptual contribution is a reframing of what the recurrent state represents. Prior RNN architectures for language modeling — from linear attention through RWKV-6 — conceptualized the state as a trace: a compressed, decaying record of past inputs, where older information fades and newer information dominates. The model's job was to manage this decay intelligently (data-dependent decay, gating) but the fundamental metaphor was of a fading memory.

RWKV-7's generalized delta rule enables a qualitatively different metaphor: the state as an editable key-value data structure. The model can:

• Insert new key-value pairs (the v_tᵀ k̃_t term).
• Delete specific keys (the κ̂_tᵀ(a_t ⊙ κ̂_t) removal term, which subtracts the value associated with a particular key).
• Update values associated with existing keys (combining removal and insertion with different keys).
• Swap entries (the permutation matrices enabled by Lemma 1, using c = 2).
• Copy entries from one key location to another (Lemma 3).

This is not merely a more expressive version of "forget and add." It is a fundamentally different computational primitive. A decaying trace can only attenuate old information; an editable data structure can reorganize it.

Why this reframing is significant: It changes how one thinks about the architecture's capabilities and limitations. Under the "decaying trace" metaphor, long-range dependencies are about resisting decay — keeping the decay factor near 1 so important information isn't lost. Under the "editable data structure" metaphor, long-range dependencies are about explicit state management — moving, copying, and deleting information as needed, independent of time elapsed. This suggests that RWKV-7's success on long-context tasks (Section 7.5) may come not from better decay management but from its ability to actively restructure stored information as context evolves.

The state inspection results (Appendix J) provide indirect evidence for this reframing. RWKV-7's WKV matrices show significantly lower root-mean-square values (O(1) vs. O(1000s) for RWKV-5/6) and lower stable rank for contexts longer than 32 tokens. Under the "decaying trace" metaphor, lower stable rank would suggest less information stored. But RWKV-7 outperforms predecessors on long-context tasks. The paper's interpretation — that RWKV-7 achieves "stronger information compression and utilization capabilities" — aligns with the editable-data-structure view: the state is not a lossy trace but a compact, organized representation where irrelevant information is actively removed rather than passively decayed.

The significance of the copy operation: Lemma 3's proof that RWKV-7's transition matrix can implement copy matrices (x → y: replacing column y with a copy of column x) is particularly revealing. A decaying trace fundamentally cannot copy — information can only move from input to state, not from one state location to another. The copy operation enables the state to restructure itself based on internal computation, not just external input. This is the architectural basis for the paper's claim (Appendix F) that "you might think of the RWKV-7 state as being like an internal scratchpad."

Innovation 4: Architecture Upgrade as a Practical and Scientific Strategy — The Demonstration That Architectural Improvements Can Compound Without Restarting Training

The paper's fourth contribution is methodological rather than architectural: the demonstration that a model trained under one architecture (RWKV-5 or RWKV-6) can be converted to a substantially different architecture (RWKV-7) and continue training with meaningful gains, amortizing the cost of the earlier training. This is not a matter of fine-tuning — the architectures differ in their transition matrices (diagonal vs. diagonal-plus-rank-one), their gating mechanisms (receptance-gated MLP vs. squared-ReLU MLP), and their token-shift implementations (data-dependent vs. static). The conversion process (described in Section 6 and Appendix E) is a non-trivial mapping between parameter spaces.

What this enables scientifically: In a field where architectural innovation is increasingly gated by the cost of large-scale training, the ability to "upgrade" a model mid-training lowers the barrier to testing architectural ideas at meaningful scale. The paper's released models — one 1.5B and one 2.9B — were trained for only 3.1 trillion tokens as RWKV-7, but benefit from 2.5 trillion tokens of prior RWKV-6 training. If each architectural iteration required training from scratch, the compute budget would be prohibitive for most research groups. The upgrade methodology means that a research team can train a model under the current best architecture, then upgrade it when a better architecture is developed, accumulating improvements without restarting.

Why this is not merely an engineering trick: The upgrade process reveals something about the relationship between the RWKV architectural family. The fact that a diagonal-state model (RWKV-6) can be meaningfully converted to a diagonal-plus-rank-one model (RWKV-7) suggests that the architectures share a common representational substrate — that the rank-one edit in RWKV-7 is building on and refining the representations learned under the diagonal regime, not learning entirely new representations from scratch. The rescaling operations (halving weights, adjusting norms, adding small random initializations for new parameters) are a form of architecture-aware transfer learning that preserves the "knowledge" encoded in the earlier model while giving the new mechanisms room to learn.

Limitations that make this a starting point, not a solved problem: The paper is transparent that training from scratch would likely be better — the models re-used some data (some documents were seen two or three times across architecture versions), and the initial training was not optimized for the new architecture. The upgrade approach is a pragmatic response to compute constraints, not a claim of optimality. The paper's suggestion that "the difference would be even more dramatic" if trained from scratch is speculative but plausible, given that the model had to spend some fraction of its RWKV-7 training learning to use the new mechanisms effectively rather than continuing to improve its representations.

Nevertheless, the upgrade methodology is a genuine innovation in research practice. It demonstrates that architectural progress can be cumulative even under fixed total compute budgets — a principle that, if widely adopted, could accelerate progress in efficient architecture design by allowing researchers to iterate on model architectures without paying the full pretraining cost each time.

Innovation 5: The Bonus Term as a Mechanism for Single-Layer Induction — Architectural Support for In-Context Pattern Detection

The paper's architectural innovation in the readout mechanism — the bonus term u_t = (r_t · (ρ ⊙ k̃_t)ᵀ) v_t — represents a deliberate design for enabling single-layer induction heads, a circuit motif identified by Elhage et al. (2021) and Olsson et al. (2022) as central to in-context learning in Transformers.

The problem it solves: In a pure recurrent architecture, there is a one-timestep lag between storing a value in the state and being able to retrieve it. At time t, the state wkv_t has just been updated with the current token's value v_t, but the retrieval r_t · wkv_tᵀ queries the state after the update. This means the current token's own value can influence the output, but only by being stored and then immediately retrieved — a conceptually roundabout path. More importantly, detecting that the previous token matched some pattern and using that to amplify the current token's value requires the model to exploit the fact that wkv_t contains v_{t-1} stored at key k̃_{t-1}. This works but requires the model to route information through the state.

The bonus term shortcuts this. r_t · (ρ ⊙ k̃_t)ᵀ computes the dot product between the current query r_t and the current (boosted) replacement key k̃_t. If they match — which the model can learn to arrange when the current token should attend to itself — the bonus adds the current value v_t directly to the output, without the state roundtrip.

Why this matters for induction: An induction head needs to detect that token B follows token A in the current context, and then predict that B will follow A again. In a Transformer, this is implemented by a previous-token head attending to the position of A and copying B from the subsequent position. In a recurrent architecture, the analog would be: (1) at time t-1 (when seeing A), store something that will be retrieved at time t (when seeing B), and (2) at time t, retrieve that stored information and use it to amplify B's prediction. The bonus term directly supports step (2): the model can arrange for r_t to match k̃_t when the induction condition is met, giving the current token's value an amplified influence on the output.

Connection to token shift: The token shift mechanism (x_t^□ = lerp(x_t, x_{t-1}, μ_□)) provides access to the change between consecutive tokens. Combined with the bonus term, a single RWKV-7 layer has all the pieces needed to form an induction head: token shift detects token transitions, the WKV state stores associations, and the bonus term allows self-attention to the current token when a previously-stored pattern is recognized. The paper explicitly frames this as the purpose: "Token shift is a variety of 1D short convolution that is intended to allow the model to create induction heads within a single layer" (Appendix F).

Comparison to prior RWKV versions: Earlier RWKV versions fused a "time-first" term into the WKV kernel — essentially the same idea, but implemented inside the recurrence rather than as a separate additive term. The paper's extraction of this term as an explicit bonus (Equation 20) is described as simplifying the kernel while preserving functionality. The innovation is not the concept of a self-attention shortcut (which existed implicitly), but the architectural articulation of it as a named, separable mechanism with its own learned parameter ρ. This makes the design principle explicit: a recurrent architecture needs an explicit mechanism for the current token to influence the output without a state roundtrip, specifically to enable efficient induction.

Empirical role: The ablation in Appendix K (Table 19) shows that removing the bonus term increases validation loss from 2.541 to 2.588 on the 6-layer, 768-dim model. This is a meaningful but not catastrophic degradation — the model can compensate through the WKV state retrieval path, but less efficiently. For full-scale models, the bonus term likely plays a proportionally larger role in enabling the efficient in-context learning that drives performance on tasks like LAMBADA and associative recall.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three categories of data: (a) standard English-focused and multilingual benchmarks via LM Evaluation Harness (Gao et al., 2023) — LAMBADA, HellaSwag, PIQA, ARC (Easy and Challenge), GLUE, Winogrande, SciQ, MMLU for English (Table 3), and LAMBADA Multilingual, PAWS-X, XCOPA, XNLI, XStoryCloze, XWinogrande for multilingual (Table 4); (b) the Pile dataset (Gao et al., 2020) with 332B tokens for architectural ablation comparisons and for Pile-trained model releases; (c) synthetic/specialized benchmarks — MQAR (associative recall with multi-query key-value pairs), MAD (Mechanistic Architecture Design; Poli et al., 2024), PG19 (Rae et al., 2019) for long-context loss measurement, and group multiplication state-tracking tasks (Merrill et al., 2024). For the Pile-trained models, the GPT-NeoX-20B tokenizer is used (vocabulary size 50,304). For World dataset models, the RWKV World tokenizer is used (vocabulary size 65,536). The RWKV World v3 corpus used for pretraining contains 3.119 trillion tokens across web, books, code, science/Wikipedia, fiction, chat/QA/instruction, math, law/government, and poetry/lyrics domains (Table 14).

• Base model. Four RWKV-7 "Goose" models are evaluated: 0.1B, 0.4B, 1.5B, and 2.9B parameters, all trained on the RWKV World v3 dataset (Table 15). Additionally, three Pile-trained models (0.17B, 0.42B, 1.47B) are released for comparative architectural study. The 0.1B and 0.4B World models were upgraded from RWKV-5 checkpoints; the 1.5B and 2.9B World models were upgraded from RWKV-6 checkpoints (Table 2). All models use head dimension D_h = 64 and were trained with context length 4096 tokens. The models are chosen to span the sub-3B parameter range where RNN architectures have historically struggled to match Transformers.

• Metrics. For language benchmarks, the primary metrics are accuracy (acc) or accuracy normalized (acc_n) as reported by LM Evaluation Harness v0.4.8, along with perplexity (ppl) for LAMBADA. For the recent internet data evaluation (Section 7.2), compression rate (lower is better, as percentage) is used, following Delétang et al. (2024); Li et al. (2024b). For associative recall (MQAR, Section 7.3), accuracy is reported. For the MAD benchmark, per-task accuracy scores are reported. For long-context experiments (Section 7.5), cross-entropy loss on PG19 is plotted versus sequence position. For state tracking (Section 7.6), the minimum number of layers required to achieve >95% validation accuracy is reported. All LM Evaluation Harness benchmarks are evaluated under fp32 precision with 0-shot prompting, except MMLU which uses 5-shot.

• Baselines. The paper compares against multiple categories of baselines. For English benchmarks (Table 3): RWKV-5 and RWKV-6 predecessors (Peng et al., 2023, 2024b), SmolLM2 at 135M/360M/1.7B scales (Allal et al., 2025), Qwen2.5 at 0.5B/1.5B/3B scales (Qwen et al., 2025), and Llama-3.2 at 1B/3B (pruned and distilled from Llama-3.1-8B; Grattafiori et al., 2024). For multilingual benchmarks (Table 4): the same set plus additional comparisons. For Pile ablations (Table 18): Pythia (Biderman et al., not cited — actually the paper cites no specific Pythia reference in Table 18, but refers to Pythia-160M, Pythia-410M, Pythia-1.4B), Mamba at 130M/370M/1.4B (Gu & Dao, 2023), Mamba-2 at 130M/370M/1.3B (Dao & Gu, 2024), and prior RWKV versions. For recent internet data (Table 5): an extensive set of 1B-3B and 3B-scale models including Qwen2.5, Llama-3.2, SmolLM2, stablelm, mamba variants, Zamba2, MobileLLM, gemma, and pythia. For the MAD benchmark (Table 7): results for Transformer, Multihead Hyena, DeltaNet, Mamba, Hyena, and GLA are taken from Yang et al. (2024c). For state tracking (Figure 8): Transformer, Mamba, S4, and classical RNN baselines.

• Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of search-based test-time compute papers; rather, all evaluations use standard autoregressive generation with greedy decoding (for benchmark evaluations) or perplexity computation. For the FLOPs comparison (Section 7.1, Figures 3a and 4a), total training FLOPs are plotted against benchmark accuracy. The paper notes that Llama-3.2 models are not included in FLOPs plots because "they have no corresponding FLOPs amounts due to having been created via pruning and distillation from larger models." For the speed and memory measurements (Section 8, Figure 9), forward + backward pass timing is measured for batch size 8, head dimension 64, model dimension 4096 on an H100 SXM GPU at varying sequence lengths.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for benchmark evaluations — standard LM Evaluation Harness protocols are used. For the initial token sensitivity analysis (Appendix M), statistical significance testing is performed comparing performance with and without the <|endoftext|> token, using *p < 0.05, **p < 0.01, ***p < 0.001 notation (Table 20). No confidence intervals or error bars are reported for benchmark results. For the MQAR experiments, results are reported as "maxed over 3 different learning rate settings" (Table 6).

Main Quantitative Results

English and Multilingual Benchmark Performance (Section 7.1)

The headline result is that RWKV-7-2.9B achieves an average English benchmark accuracy of 71.5% (Table 3), matching Qwen2.5-3B (71.4%) while being trained on 5.6T total tokens versus Qwen2.5's 18T tokens — a more than 3× token efficiency advantage. On multilingual benchmarks (Table 4), RWKV-7-2.9B achieves an average accuracy of 61.1%, substantially outperforming Qwen2.5-3B at 55.6% and Llama-3.2-3B at 58.1%, establishing a new 3B state-of-the-art for multilingual performance.

Breaking down English performance by model scale (Table 3):

• At 0.1B scale: RWKV-7-0.1B achieves 50.5% average, compared to SmolLM2-135M at 51.0% (but trained on 2.0T tokens vs. RWKV-7's 1.6T) and RWKV-5-0.1B at 43.7%. The largest gains over RWKV-5 are on LAMBADA (+9.7 points) and HellaSwag (+10.2 points).

• At 0.4B scale: RWKV-7-0.4B achieves 57.1%, trailing Qwen2.5-0.5B (57.9%, 18T tokens) and SmolLM2-360M (57.4%, 4T tokens), but trained on only 3.1T tokens. The multilingual story is stronger: 52.0% average vs. Qwen2.5-0.5B at 50.0% and SmolLM2-360M at 47.3%.

• At 1.5B scale: RWKV-7-1.5B achieves 67.6%, slightly ahead of Qwen2.5-1.5B at 67.4% and SmolLM2-1.7B at 66.6%, despite being trained on 5.6T vs. 18T and 11T tokens respectively. On MMLU, however, RWKV-7-1.5B scores 43.3% vs. Qwen2.5-1.5B's 61.0% — a substantial gap suggesting that MMLU's knowledge-intensive nature benefits more from total training tokens than from architectural efficiency. On multilingual benchmarks, RWKV-7-1.5B achieves 58.0% vs. Qwen2.5-1.5B's 54.5% — a 3.5 point advantage.

• At 2.9B scale: RWKV-7-2.9B achieves 71.5% English average, matching Qwen2.5-3B's 71.4%. Individual benchmark comparisons show RWKV-7 ahead on LAMBADA (73.4 vs. 67.1), HellaSwag (76.4 vs. 73.5), ARC Easy (81.0 vs. 77.4), and ARC Challenge (48.7 vs. 45.0), but behind on MMLU (55.0 vs. 65.7) and GLUE (61.8 vs. 70.2). Multilingual: 61.1% vs. 55.6% for Qwen2.5-3B, with consistent advantages across all six multilingual benchmarks (Table 4). Llama-3.2-3B achieves 58.1% multilingual average — 3 points behind RWKV-7.

The FLOPs vs. accuracy plots (Figures 3a and 4a) show that RWKV-7 models occupy a distinctly favorable Pareto frontier: for multilingual benchmarks, RWKV-7 models achieve higher accuracy at dramatically lower training FLOPs compared to Transformer baselines (Qwen2.5, SmolLM2). For English benchmarks, the advantage is present but less dramatic — RWKV-7 models roughly match the accuracy-FLOPs tradeoff of the best Transformer models. The paper notes that RWKV-7 models' FLOPs figures account for total FLOPs across all training phases (RWKV-5/6 pre-training + RWKV-7 continued training), which disadvantages them relative to models trained from scratch on all tokens.

Recent Internet Data Evaluation (Section 7.2)

RWKV-7-1.5B achieves an average compression rate of 8.16% across seven temporally novel data sources (Table 5), placing it second behind Qwen2.5-1.5B at 8.06% and ahead of Llama-3.2-1B (8.23%), SmolLM2-1.7B (8.23%), and various other 1B-2B scale models. At the 3B scale, RWKV-7-2.9B achieves 7.74%, third behind Llama-3.2-3B (7.57%) and Qwen2.5-3B (7.66%), but ahead of all other models in the 2B-4B range including stablelm-3b (7.86%), gemma-2-2b (8.12%), and mamba variants (8.18-8.41%).

Notable per-domain patterns: On GitHub Python, RWKV-7-1.5B achieves 5.57% vs. Qwen2.5-1.5B's 4.42% (worse by 1.15 points) and on GitHub C++ 5.29% vs. 4.40% — suggesting code compression is a relative weakness. On AO3 fiction (10.93%), BBC news (9.34%), and Wikipedia English (8.97%), RWKV-7-1.5B is competitive or slightly better than Qwen2.5. The paper frames this as evidence that the benchmark improvements are not artifacts of data leakage, since the evaluation data postdates the training data.

However, the paper does not discuss whether the compression rate metric itself might favor certain architectural properties (e.g., lower perplexity on certain token distributions) in ways that don't directly translate to generation quality, and no generation-based evaluation (e.g., perplexity on standard benchmarks) is reported for these temporally novel datasets.

Associative Recall (MQAR) (Section 7.3)

RWKV-7 achieves perfect (>99%) accuracy on MQAR at sequence lengths up to 2048 tokens with 256 key-value pairs (Table 6). More specifically, with a WKV state dimension of 8192 (corresponding to a 128-head model with head dimension 64), RWKV-7 achieves 98.43% accuracy at (512, 64) — meaning sequence length 512 with 64 key-value pairs — 95.01% at (1024, 128), and 72.93% at (2048, 256). With a larger WKV state dimension of 32768 (256 heads × 64), the model achieves >99% up to (512, 64) and 98.97% at (256, 16). With dimension 65536 (512 heads), performance reaches >99% across all tested configurations.

The paper computes an information density metric: at (2048, 256) with 72.93% accuracy and a state of 8192 dimensions, the model stores 256 × 0.7293 × log₂(4096) × 2 = 4480.8 bits, yielding 0.547 bits per dimension. The paper presents this as evidence of efficient information compression in the state, though no comparable metric is reported for baseline architectures (Mamba, DeltaNet), making it difficult to assess whether this density is unusually high.

The training setup uses two-layer RWKV-7 models with the RWKV-7-specific initialization, AdamW with ϵ = 1 × 10^{-18}, and weight decay of 0.1 applied only to weight matrices (not to normalization parameters). The paper notes that the small ϵ is necessary to "stabilize learning in later stages" of MQAR training.

Mechanistic Architecture Design (MAD) Benchmark (Section 7.4)

RWKV-7 achieves the highest average score (79.3) across all six MAD tasks, compared to Transformer (74.5), Multihead Hyena (73.2), DeltaNet (71.8), Mamba (69.3), Hyena (66.0), and GLA (60.0) (Table 7). Individual task scores show:

• Fuzzy Recall: 43.2%, the highest reported (DeltaNet: 35.7, Transformer: 29.8). This task requires retrieving values from keys that are noisy/approximate versions of the stored keys — a direct test of the model's ability to perform associative memory with partial key matching.
• In-Context Recall: 100%, matching DeltaNet and exceeding Transformer (94.1%) and Mamba (90.4%).
• Noisy Recall: 100%, again matching DeltaNet, vs. Transformer (86.8%) and Mamba (90.1%).
• Selective Copy: 98.8%, slightly below Transformer (99.6%) and DeltaNet (100%), suggesting a minor weakness in precise token-level copying.
• Memorize: 89.1%, competitive with Mamba (89.5%), Multihead Hyena (89.4%), and Hyena (89.5%), but above DeltaNet (52.8%) by a wide margin — DeltaNet's poor memorization score is attributed to its aggressive state editing, which may overwrite information that should be retained.
• Compress Recall: 44.5%, second behind Mamba (52.7%) but ahead of Transformer (51.6%).

The results for comparison models are reported "from Yang et al. (2024c)," meaning the experimental setup may not be identical, and the paper does not detail whether all models were trained with comparable hyperparameter budgets. Nevertheless, the MAD benchmark provides direct evidence that RWKV-7's architectural mechanisms translate to improved performance on synthetic tasks designed to probe specific capabilities.

Long Context Experiments (Section 7.5)

The PG19 loss-vs-position analysis reveals a surprising dataset-dependent behavior that the paper does not fully explain. For Pile-trained models (Figure 5), RWKV-7-1.5B shows monotonically decreasing loss with context position up to the maximum tested length of ~32k tokens, substantially outperforming both RWKV-6 and Mamba at long ranges. The loss curve for RWKV-7 continues to decline where its predecessors flatten or increase — a direct demonstration that the editable state mechanism improves long-context retention.

However, for World dataset-trained models (Figure 6), the pattern is different: RWKV-7-1.5B and 2.9B both show loss increasing after approximately 10,000 tokens, whereas their RWKV-6 predecessors show decreasing or flat loss. The paper speculates that "the larger dataset and model size created inductive biases that caused overfitting to specific context lengths" (4,096 tokens during pretraining). This is a significant observation: it suggests that RWKV-7's enhanced state-editing capability may make it more prone to learning context-length-specific strategies during training, which then fail to extrapolate, rather than less.

Further evidence comes from the pass-key retrieval evaluation (Figure 7): without extended-context fine-tuning, RWKV-7-1.5B achieves perfect retrieval accuracy up to 19,600 tokens but degrades beyond 20,600; RWKV-7-2.9B extends this to 35,000 tokens. After fine-tuning on packed 128k-token sequences (from a specially constructed dataset mixing public and custom sources, Table 8), RWKV-7-1.5B reliably retrieves up to 29,000 tokens (degrading at ~40k), and RWKV-7-2.9B reliably retrieves up to 30,000 tokens (degrading at ~50k). This demonstrates that the architecture can handle extended contexts, but requires explicit long-context training to do so — the inductive biases from 4k-context pretraining do not automatically generalize.

The fine-tuning dataset (Table 8) uses a document length-based weighting scheme: documents shorter than 32,768 characters receive weight 1.0; longer documents receive linearly increasing weights between 2.0 and 3.0, capped at 3.0 beyond 512,000 characters (which approximates 128k tokens at typical character-per-token ratios). This weighting "increases the inclusion of longer documents to bolster the model's handling of extended contexts while retaining shorter documents for diversity."

State Tracking Using Group Multiplication (Section 7.6)

RWKV-7 requires fewer layers than Transformer, Mamba, and S4 to achieve >95% validation accuracy across all three group multiplication tasks (S₅, A₄ × Z₅, Z₆₀) at all tested sequence lengths (5, 10, 15, 20) (Figure 8). The minimum number of layers for RWKV-7 is consistently 1–2 across all configurations, compared to Transformers requiring 2–3 layers, Mamba requiring 2–4 layers (and failing entirely on S₅ at sequence length 20 with 4 layers), and S4 requiring 3–4+ layers.

The paper notes that RWKV-7 "exhibits stronger state-tracking capabilities than Transformers, Mamba, and S4, though slightly weaker than classical RNNs." Classical RNNs (with dense transition matrices) would theoretically require only 1 layer for any regular language — RWKV-7 requires 1–2 layers, suggesting its diagonal-plus-rank-one structure captures most but not all of the expressivity of a fully dense recurrence, which is consistent with the theoretical result that constant layers (not necessarily 1) suffice.

The alignment with theory (Appendix D.2) is specifically highlighted: "Figure 8 also aligns with our theory from Appendix D.2, which predicts that RWKV-7 can perform state tracking and recognize any regular language with a constant number of layers." The key practical advantage over classical RNNs is noted: "classical RNNs, while being theoretically expressive, typically suffer from gradient vanishing and memorization problems and cannot be parallelized efficiently, unlike RWKV-7."

Speed and Memory Usage (Section 8)

On an H100 SXM GPU with batch size 8, model dimension 4096 (64 heads, head dimension 64), the RWKV-7 bfloat16 kernel achieves forward + backward pass times that scale linearly with sequence length, while Flash Attention v3 scales quadratically (Figure 9). At sequence length 1024, RWKV-7 takes approximately 12 ms total (forward + backward), compared to approximately 15 ms for RWKV-6 and approximately 18 ms for Flash Attention v3. At sequence length 16384, RWKV-7 takes approximately 42 ms (forward: ~11 ms, backward: ~22 ms, with state storage adding overhead), compared to approximately 110 ms for RWKV-6 and approximately 55 ms for Flash Attention v3 (forward only). The paper notes that "Flash Attention v3 is heavily optimized for the H100 GPU," making its sub-quadratic scaling at this sequence length impressive; however, the quadratic asymptote means RWKV-7 will eventually overtake it at very long sequences.

The forward pass of RWKV-7 "is about twice as fast as the backward pass." For inference, "the forward pass does not need to store the wkv state, making it faster": at sequence length 16k, forward without state storage takes 7.9 ms, with state storage takes 11.2 ms, and backward takes 22.5 ms. The Flash Attention v3 forward pass (without backward) takes 33.9 ms at the same sequence length.

Memory usage formulas are provided: peak training memory is batch_size × model_dimension × sequence_length × 2 bytes × num_variables, where num_variables is 10 for Flash Attention v3, 10 for RWKV-6, 18 for RWKV-7 (bfloat16), and 24 for RWKV-7 (fp32). For sequence length 1024, this gives approximately 640 MB, 640 MB, 1152 MB, and 1536 MB respectively — a 2.4× increase over Flash Attention for the fp32 kernel. The paper notes that "memory usage is constant for single token inference" and "pre-fill can easily be accomplished in a chunked manner," with memory usage growing linearly with chunk size, allowing a user-selectable tradeoff between speed and memory.

Multimodal Results (Section 9)

The paper includes multimodal experiments as a secondary evaluation, using VisualRWKV-7 (an adaptation of RWKV-7 for vision-language tasks) and AudioRWKV-7 (for audio classification).

VisualRWKV-7 (Table 9): With the 2.9B RWKV-7 backbone, VisualRWKV-7 achieves 80.5% on VQAv2, 63.4% on ScienceQA, 58.0% on TextQA, and 63.7% on GQA. Compared to VisualRWKV-6 with a 3.1B backbone: improvements of +1.4 on VQAv2 (80.5 vs. 79.1), +0.5 on ScienceQA (63.4 vs. 62.9), +5.3 on TextQA (58.0 vs. 52.7), and +2.7 on GQA (63.7 vs. 61.0). The TextQA improvement is highlighted as evidence of "superior associative recall capabilities."

At smaller scales: VisualRWKV-7-0.4B achieves 77.9% on VQAv2 and 62.3% on GQA, surpassing VisualRWKV-6-1.6B (73.6% and 58.2% respectively) with only 1/4 of the parameters. The paper frames this as "demonstrating the powerful modeling capabilities of RWKV-7."

The vision encoder configuration uses SigLIP, DINOv2, and SAM encoders (concatenated features projected through an MLP with context gating), processing images at up to 1024×1024 resolution. Training uses 558k alignment data (stage 1) and 665k SFT data (stage 2), aligned with LLaVA-1.5 training data. The paper notes that both VisualRWKV-7 and VisualRWKV-6 use identical vision encoders and training data, so improvements are attributable to the language model backbone.

AudioRWKV-7 (Table 10): On the AudioSet dataset (Gemmeke et al., 2017), AudioRWKV-7-19.8M achieves a mean Average Precision (mAP) of 0.431, compared to AudioRWKV-6-19.8M at 0.426, MambaOut-101.3M at 0.397, and a 26M-parameter CNN baseline at 0.392. At 8.9M parameters, AudioRWKV-7 achieves 0.392, compared to AudioRWKV-6 at 0.381. The larger Transformer-based HST-AT (88.5M) achieves 0.433 — AudioRWKV-7 approaches this with roughly 1/4.5 of the parameters. The paper notes that to "ensure a fair comparison, we retrained AudioRWKV-6 without ensembling models with different patch settings."

Ablation Studies and Robustness Checks

Pile-trained architecture comparison (Table 18): Three model sizes (168M, 421M, 1.47B parameters) are trained from scratch on the full 332B-token Pile dataset, comparing RWKV-7 against RWKV-4, RWKV-6, Pythia, Mamba, and Mamba-2. At 168M, RWKV-7 achieves 49.8% average accuracy vs. RWKV-6 at 49.2% and Mamba-2 at 49.0%. At 421M, 56.0% vs. Mamba-2 at 55.2% and RWKV-6 at 49.2% (note: the RWKV-6 421M model is not listed in the table — the 421M comparison is to Mamba-2 370M at 55.2% and RWKV-7-421M at 56.0%). At 1.47B, 62.6% vs. Mamba-2-1.3B at 61.2% and Mamba-1.4B at 61.1%. The gap over RWKV-6 grows with model size: +0.6 points at 168M, not directly comparable at 421M (different model sizes), and the 1.47B comparison is against RWKV-4-1.5B (57.2%) rather than RWKV-6 (no 1.5B RWKV-6 Pile model is listed). The paper states that "the performance gap sustains as the model size increases, suggesting that RWKV-7 may scale more effectively than its predecessors." However, the Pile-trained comparisons are somewhat confounded by different model sizes (e.g., RWKV-7-421M vs. Mamba2-370M vs. Pythia-410M — all slightly different parameter counts).

Architecture component ablation (Table 19): A small 6-layer, 768-dimension model trained on the 1.6B-token MiniPile (Kaddour, 2023) dataset at context length 512 is used to ablate four design choices:

• Scalar decay replacing vector-valued decay: validation loss increases from 2.541 to 2.609. This is the largest single-component degradation.
• Scalar in-context learning rate replacing vector-valued ICLR: validation loss increases from 2.541 to 2.591.
• Same removal and replacement keys (no decoupling): validation loss increases from 2.541 to 2.560.
• No bonus term: validation loss increases from 2.541 to 2.588.

These results are internally consistent in showing that each component contributes to performance, with vector-valued decay being the most important single factor. However, the paper does not report ablations that remove combinations of components (e.g., scalar decay + scalar ICLR together), which would test the synergy hypothesis advanced in the architectural motivation. The ablation is also conducted at a very small scale (6 layers, 768 dimensions, 1.6B tokens), and it is unknown whether the relative importance of these components changes at larger scales (2.9B parameters, 3T+ tokens).

State stability comparison (Appendix J, Figures 15–16): Comparing RWKV-5-1.5B, RWKV-6-1.6B, and RWKV-7-1.5B on PG19 validation samples of length 8,192 tokens: RWKV-7's WKV state entries have root-mean-square (RMS) values consistently of order O(1) (visualized as 0.00–0.10 in representative heads), while RWKV-5 and RWKV-6 show RMS values reaching into the hundreds (RWKV-5: 13–97; RWKV-6: 6–445). The stable rank of RWKV-7's WKV matrices is lower than predecessors for contexts longer than 32 tokens (Figure 16), which the paper interprets as "stronger information compression and utilization capabilities."

Removal key multiplier and replacement rate booster statistics (Appendix L, Figures 17–23): The learned parameter ξ (removal key multiplier) ranges from approximately -5.3 to 9.4 across layers and models. The learned parameter α (replacement rate booster) ranges from roughly 0 to 1. Both show systematic variation across layers, suggesting non-trivial learned structure rather than convergence to a uniform value. The biases of d_t (decay precursor) also vary across layers, with some layers showing positive mean bias (favoring stronger decay) and others negative.

Initial token sensitivity (Appendix M, Table 20): On a subset of 142 LAMBADA examples where the answer appears as the first word of the paragraph and is not repeated, omitting the <|endoftext|> token at the start of input causes dramatic performance degradation: RWKV-7-0.1B drops from 36.6% to 9.2% accuracy (p < 0.001); RWKV-7-0.4B drops from 48.6% to 28.9% (p < 0.001). In contrast, Qwen2.5-0.5B shows no significant difference (54.9% vs. 47.9%, NS). The paper also notes that "two consecutive <|endoftext|> tokens at the beginning can further improve performance," despite this pattern never appearing in training. This reveals a practical sensitivity: RWKV-7's state initialization, and specifically its ability to retain the first token, is fragile in a way that Transformers are not.

Critical Assessment

The experimental results support several central claims but leave others insufficiently tested or conditional on specific evaluation choices.

Claim: RWKV-7 achieves new 3B SoTA on multilingual tasks and matches English 3B SoTA despite being trained on dramatically fewer tokens.

The multilingual claim is strongly supported. RWKV-7-2.9B's 61.1% average on multilingual benchmarks (Table 4) exceeds Qwen2.5-3B (55.6%) and Llama-3.2-3B (58.1%) by clear margins (5.5 and 3.0 points respectively), and the advantage is consistent across all six multilingual benchmarks. The English claim is more nuanced. RWKV-7-2.9B's 71.5% average matches Qwen2.5-3B's 71.4% (Table 3) — a tie, not a clear win — and the aggregate masks substantial variation: RWKV-7 leads on LAMBADA, HellaSwag, and ARC, but trails significantly on MMLU (55.0 vs. 65.7) and GLUE (61.8 vs. 70.2). The "dramatically fewer tokens" claim (5.6T vs. 18T total, 3.1T vs. 18T in the final architecture) is accurate but must be contextualized: the 5.6T figure includes training under earlier architectures (RWKV-5/6) on earlier datasets (World v1, v2, v2.1), so the effective amount of RWKV-7-specific training is 3.1T tokens on World v3. The paper acknowledges that training from scratch would likely be better, making this a lower bound on RWKV-7's efficiency — a point in its favor, but also a limitation of the direct comparison, since Qwen2.5 was trained from scratch with a consistent architecture.

A critical missing comparison: the paper does not compare against Transformer models trained on the same RWKV World v3 dataset with the same token budget, which would isolate the architectural contribution from the dataset composition. The sample efficiency claim conflates architecture and data quality effects. If the RWKV World v3 dataset is higher-quality per token than Qwen2.5's training data (a plausible but unverified possibility), some fraction of the efficiency gain could be data-driven rather than architectural.

Claim: RWKV-7's generalized delta rule enables expressivity beyond TC⁰, proven by recognizing all regular languages.

The theoretical proof (Appendix D) is rigorous and self-contained, establishing that a constant number of RWKV-7 layers can recognize any regular language. However, the empirical validation is limited. The state-tracking experiment (Figure 8) demonstrates that RWKV-7 requires fewer layers than Transformer/Mamba/S4 on three group multiplication tasks — consistent with the theory, but insufficient to demonstrate that the mechanism used by the trained model actually exploits the non-diagonal transition matrix in the way the proof constructs. The model could be solving these tasks through a different mechanism that does not rely on rank-one editing. The proof is a constructive existence result, not a demonstration that gradient-based training discovers the constructed solution. The paper does not mechanistically analyze whether trained RWKV-7 models on state-tracking tasks actually learn the swap/copy operations described in Lemmas 1 and 3. Without such analysis (e.g., probing the WKV state transition matrices for learned permutation-like structures), the connection between the theoretical capability and practical behavior remains speculative.

Additionally, the proof requires MLP layers that are "exponentially wide in the number of states of the original DFA." The practical MLP hidden dimension is 4× model dimension — far from exponential. This means that while the architecture is theoretically capable of recognizing all regular languages given unlimited width, the practical models are capacity-constrained in ways the proof does not address. The paper does not discuss this gap.

Claim: The model upgrade methodology enables compounding architectural improvements without restarting training.

The paper demonstrates feasibility (models were successfully upgraded and improved) but does not isolate the contribution of the upgrade process from the contribution of additional training on new data. The upgraded models continued training on the new World v3 dataset (3.1T tokens), which is substantially larger than the datasets used for earlier versions. The observed improvements could come from (a) the architectural upgrade, (b) the additional data, or (c) the interaction. An ablation where an RWKV-6 model is further trained on World v3 for the same number of additional tokens (without upgrading the architecture) would disentangle these factors, but is not reported.

The paper also does not report the performance of the pre-upgrade checkpoints on the evaluation benchmarks, making it impossible to compute the marginal contribution of the upgrade itself versus the continued training. Table 2 describes the total training tokens per stage, but Table 3 reports only final performance.

Claim: RWKV-7 maintains constant memory and linear time complexity, offering a compelling alternative to Transformers.

The speed and memory measurements (Figure 9) are for a specific configuration (batch size 8, model dimension 4096, head dimension 64, H100 SXM). The memory comparison shows RWKV-7 using 1.8× the memory of Flash Attention v3 at training time (18 vs. 10 variable equivalents). The inference advantage — constant memory per token — is not empirically demonstrated at scale; the pass-key retrieval experiments show retrieval accuracy but do not measure inference memory or latency at long context lengths. The paper reports training times but not end-to-end inference throughput at deployment-relevant batch sizes (batch size 1, which is typical for autoregressive generation). For the inference advantage to be practically meaningful, constant memory must translate to lower cost or higher throughput relative to KV-cache-based Transformers at long sequence lengths where the KV cache becomes the bottleneck. This comparison is not provided.

Experiments that would strengthen the paper:

• Same-data, same-token-budget comparison: Train a small Transformer (e.g., 1.5B parameters) on the exact RWKV World v3 dataset for the same number of tokens as RWKV-7-1.5B (5.6T if including earlier datasets, or 3.1T if comparative only for the final architecture) to isolate architecture from data effects.
• Pre-upgrade baseline evaluation: Report benchmark performance of the RWKV-5 and RWKV-6 checkpoints immediately before upgrade, to quantify the marginal benefit of the architectural change versus continued training.
• Mechanistic analysis of state-tracking solution: Probe the WKV state transition matrices of trained state-tracking models to determine whether they learn structures resembling the swap/copy matrices constructed in the proof.
• Inference latency at scale: Measure tokens-per-second and peak memory for autoregressive generation at sequence lengths of 100k+ tokens, comparing RWKV-7 to Flash Attention and other efficient Transformer variants.
• Ablation at scale: The architecture component ablation (Table 19) is at 768 dimensions, 6 layers, and 1.6B tokens. Repeating this at 1.5B scale (even on a smaller dataset) would test whether the component contributions are consistent across scales.
• Confidence intervals: All benchmark numbers are reported as point estimates without error bars, making it impossible to assess whether differences of 1–2 percentage points (common in the 1.5B and 2.9B comparisons) are statistically meaningful.

Genuine weaknesses that the paper partially acknowledges:

• Single benchmark family for some analyses: The MAD and MQAR experiments use synthetic tasks; the state-tracking experiment uses three groups. It is unknown whether the demonstrated advantages generalize to other synthetic tasks or to natural language tasks that require state tracking.
• Small-scale ablation: The component ablation uses a 6-layer, 768-dim model on 1.6B tokens. The paper's own results show that the architecture's advantages over predecessors grow with scale — a small-scale ablation may underestimate the contribution of components that primarily benefit larger models.
• Long-context behavior is dataset-dependent and requires explicit training: The PG19 loss curves show that RWKV-7 on World data degrades beyond 10k tokens without extended-context fine-tuning. This means the architecture's theoretical long-context advantage does not emerge automatically from pretraining — it requires explicit intervention, which complicates the claim of inherent long-context capability.
• The sample efficiency narrative conflates multiple factors: The comparison against Qwen2.5 (18T tokens from scratch) involves different datasets, tokenizers, and training strategies. The paper's claim of "dramatically fewer tokens" is accurate in raw count, but the effective information per token may differ substantially between the RWKV World v3 curated multilingual corpus and Qwen2.5's (unspecified but likely web-heavy) training data.
• No analysis of failure modes on MMLU/GLUE: The 10-point MMLU gap versus Qwen2.5-3B is substantial and is not investigated. Is it because MMLU requires factual knowledge that benefits from more training tokens? Is it because RWKV-7's state-editing mechanism is less suited to knowledge-intensive tasks than to reasoning tasks? The paper does not offer hypotheses or analysis.

6. Limitations and Trade-offs

Limitation 1: The Architecture Upgrade Methodology Confounds Architectural Gains with Additional Data and Training

The assumption or constraint. The paper's headline performance results rely on a "model upgrade" process: the 1.5B and 2.9B RWKV-7 models were not trained from scratch but were converted from pre-existing RWKV-6 checkpoints and then trained on an additional 3.1 trillion tokens of the new RWKV World v3 corpus (Table 2). This means the models have seen a total of 5.6 trillion tokens across their architectural lineage, but only 3.1 trillion tokens in the RWKV-7 format. The paper acknowledges this explicitly in Section 6:

"Due to compute budget constraints, the Goose World 3 0.1B and 0.4B models were trained from pre-existing RWKV-5 World v1 and v2 checkpoints, and the Goose World 3 1.5B and 2.9B models were trained from pre-existing RWKV-6 World v2.1 checkpoints."

And further:

"Under this methodology, some documents were seen two or even three times."

The consequence. The paper's central empirical claim — that RWKV-7 achieves state-of-the-art performance despite being trained on "dramatically fewer tokens" than competitors — conflates at least three factors: (a) the architectural improvement from RWKV-6 to RWKV-7, (b) the additional 3.1 trillion tokens of new data in World v3 over World v2.1, and (c) the continued training itself (more optimization steps, even on partially repeated data). The paper cannot distinguish how much of the performance gain comes from each factor. This matters because a practitioner deciding whether to adopt RWKV-7 must know whether the gains are primarily architectural (and thus reproducible when training from scratch) or primarily driven by continued training on a larger dataset (in which case an RWKV-6 model given the same additional data might perform similarly). The paper's own speculation — "we theorize that if we were less constrained by compute and were able to train these models from scratch with the same amount of total tokens instead of from pre-trained checkpoints of earlier RWKV versions, the difference would be even more dramatic" (Section 7.1) — is plausible but unverified.

What evidence exists in the paper. The paper provides no ablation that disentangles these factors. There is no RWKV-6 model trained on the full World v3 dataset for comparison, no evaluation of the pre-upgrade RWKV-6 checkpoints on the benchmark suite, and no from-scratch RWKV-7 training run at comparable scale. The Pile-trained models (Table 18) provide the cleanest architectural comparison (all models trained from scratch on the same 332B-token dataset), but these are at smaller scale (168M-1.47B parameters) and use a different data distribution than the World models. The Pile results show RWKV-7 outperforming RWKV-6 and Mamba-2 at matched sizes, which supports the architectural claim, but the scale gap between the Pile experiments and the flagship 2.9B World model is substantial (1.47B vs. 2.9B parameters; 332B vs. 5.6T tokens).

Mitigation status. The paper partially mitigates this by being transparent about the methodology and by releasing Pile-trained models that enable cleaner comparison. But the core efficiency claim — "matches Qwen2.5 despite 3× fewer tokens" — cannot be verified as an architectural property without a from-scratch training run or a same-data continued-training baseline for the predecessor architecture. The paper does not claim to have solved this; it describes from-scratch training as future work constrained by compute resources (Section 10.1).

Limitation 2: Long-Context Performance Is Dataset-Dependent and Requires Explicit Intervention — The Architecture Does Not Automatically Extrapolate

The assumption or constraint. RWKV-7 is motivated in part by the promise of constant-time, constant-memory inference that should scale gracefully to arbitrarily long sequences. The paper states (Section 2) that a recurrent architecture with these properties "would scale gracefully to arbitrarily long sequences." All models are pretrained with a context length of 4,096 tokens (Appendix E).

The consequence. The PG19 long-context experiments reveal a striking and unexpected pattern: the architecture's ability to extrapolate to longer contexts is highly dependent on the training data distribution. For Pile-trained models (Figure 5), RWKV-7 shows monotonically decreasing loss with context position, outperforming predecessors at long range. But for World dataset-trained models (Figure 6), RWKV-7 actually performs worse than RWKV-6 at long context: loss increases after approximately 10,000 tokens, while RWKV-6 loss decreases. The paper speculates about the cause:

"We speculate this is because the larger dataset and model size created inductive biases that caused overfitting to specific context lengths."

The consequence is that RWKV-7's enhanced state-editing capability — which gives it greater expressivity — may also make it more prone to learning context-length-specific strategies during pretraining, which then fail to extrapolate to longer sequences. A practitioner cannot assume that RWKV-7 will automatically handle long contexts better than a Transformer or a diagonal-state RNN; long-context capability must be explicitly trained.

Furthermore, even after explicit long-context fine-tuning on packed 128k-token sequences (Section 7.5, Figure 7), performance degrades well before the maximum trained context length: the 1.5B model degrades around 40k tokens and the 2.9B model around 50k tokens. The pass-key retrieval task is a relatively simple test (retrieving a single sentence from a long context); degradation here suggests the architecture may struggle with more complex long-context reasoning tasks.

What evidence exists in the paper. Figures 5 and 6 show the dataset-dependent long-context behavior. Figure 7 shows pass-key retrieval accuracy degrading at 20k-50k tokens depending on model size and fine-tuning. The PG19 loss curves (Figures 5-6) are shown for sequence positions up to approximately 32k tokens — beyond the pretraining context length but well short of the lengths that Transformers with advanced KV-cache compression or ring attention can handle (100k+).

Mitigation status. The paper demonstrates that explicit long-context fine-tuning substantially improves extrapolation (Figure 7, comparing pre- and post-fine-tuning retrieval accuracy). The fine-tuning recipe is described (Table 8, document-length-based weighting), making it reproducible. However, this represents an additional training step that complicates deployment — the long-context capability is not "free" with the architecture. The paper does not explore why World-trained models overfit to context length while Pile-trained models do not, which limits the ability to predict whether a given training dataset will produce long-context extrapolation.

Limitation 3: The Expressivity Proofs Require Exponential Width — Trained Models Are Capacity-Constrained in Ways the Theory Does Not Address

The assumption or constraint. The paper's flagship theoretical results (Theorem 2: solving NC¹-complete state tracking; Theorem 3: recognizing all regular languages) prove that RWKV-7's architecture is capable of these computations given appropriate parameter settings. However, the constructive proof for Theorem 3 (Appendix D.2) requires MLP layers to implement lookup tables whose size scales exponentially with the number of states in the DFA:

"Our construction uses MLPs to implement lookup tables with sizes on the order of |Σ|^{2n}, which may require MLP layers that are exponentially wide in the number of states of the original DFA."

In the released models, the MLP hidden dimension is 4× the model dimension (Section 4.2, Table 16). For the 2.9B model, this is 4 × 2560 = 10,240 — far from exponential in any non-trivial number of states. Furthermore, the proof for Theorem 2 uses c = 2 (the rank-one multiplier), while the language models use c = 1, requiring a rescaling argument (Appendix D.3) that depends on floating-point exponent representation, which may degrade with finite numerical precision.

The consequence. There is a substantial gap between the theoretical expressivity (what the architecture could represent with unlimited width and perfect precision) and the practical capacity (what a trained model with finite width and bfloat16 precision can actually learn through gradient descent). The paper's proofs establish that the architecture is not fundamentally limited to TC⁰ in the way Transformers and diagonal SSMs are — an important result — but they do not establish that a practically-sized RWKV-7 model trained with standard optimization can learn to implement the constructed solutions. A practitioner cannot conclude from these proofs that RWKV-7 will reliably perform state tracking on complex real-world problems; the proofs are existence results, not learning guarantees.

The group multiplication experiment (Section 7.6, Figure 8) provides some empirical validation: RWKV-7 requires fewer layers than Transformers and diagonal SSMs on three specific state-tracking tasks. However, these tasks involve small groups (the largest is S₅ with 60 elements), and the paper does not analyze how the trained models solve them — whether through mechanisms resembling the proof's construction (swap and copy matrices) or through some other learned strategy that happens to work on these specific groups. Without mechanistic analysis, the connection between the theoretical results and the empirical behavior remains correlational rather than causal.

What evidence exists in the paper. Figure 8 shows RWKV-7 outperforming Transformer, Mamba, and S4 on state tracking, requiring 1–2 layers versus 2–4+. The paper explicitly notes the gap between theory and practice for classical RNNs: "classical RNNs, while being theoretically expressive, typically suffer from gradient vanishing and memorization problems" (Section 7.6). But it does not provide analogous analysis for whether RWKV-7 suffers from similar optimization difficulties on more complex state-tracking problems.

Mitigation status. The paper acknowledges the exponential width requirement in the proof (quoted above) but does not discuss its practical implications. The gap between theoretical capacity and practical learnability is a known challenge across machine learning theory, and the paper's proofs are valuable as upper bounds on what the architecture cannot be limited to. However, they do not provide a lower bound on what a practically-sized model can learn, which is what a practitioner needs.

Limitation 4: The Difficulty Estimation and Adaptive Allocation Framework from Prior Work Is Absent — The Architecture Has No Built-In Mechanism for Allocating Its Own Compute

The assumption or constraint. RWKV-7 is a purely architectural contribution: it defines a state update rule and demonstrates that models using this rule achieve strong benchmark performance when trained with a fixed compute budget and evaluated with standard greedy decoding. Unlike work on test-time compute scaling (e.g., the "compute-optimal scaling" framework studied in other papers), RWKV-7 provides no mechanism for adaptively allocating computation based on input difficulty. Every token receives the same amount of computation (one forward pass through all layers), regardless of whether the token is trivial or demands complex state editing.

The consequence. This limitation is not a flaw in the architecture per se — most language models share it — but it represents a missed opportunity given RWKV-7's specific strengths. The architecture's central innovation is an editable state that can implement complex operations (swaps, copies, deletions) on stored information. The paper argues (Appendix F) that this state functions as an "internal scratchpad" for computation. But the model has no mechanism for deciding when to use this scratchpad versus when a simpler operation (pure decay, no delta-rule edit) would suffice. Every token triggers the full weight preparation pipeline (six low-rank MLPs, multiple linear projections, a rank-one state update), even when the optimal action might be to do nothing.

A natural extension would be to allow the model to dynamically allocate its "state editing budget" — for instance, by making the in-context learning rate a_t learn to be near-zero for tokens that don't require state editing, or by introducing a halting mechanism that skips the delta-rule update entirely. But the current architecture provides no such control: a_t is always computed and applied, and while it can be near-zero, the computation to produce it is still incurred.

What evidence exists in the paper. There is no experiment that probes this limitation directly. The state inspection results (Appendix J) show that RWKV-7's state entries are of order O(1) and that the stable rank is lower than predecessors — which is interpreted positively (better compression). But an alternative interpretation is that the model is performing substantial computation (the delta-rule edit) at every token, yet the resulting state is low-rank — meaning much of that computation may be wasted. The paper does not measure the "effective utilization" of the delta-rule mechanism: what fraction of tokens trigger meaningful state edits versus near-identity updates.

Mitigation status. Not addressed. The paper's future work section (Section 10.2) focuses on scaling, speed optimization, and chain-of-thought reasoning, but does not discuss adaptive computation or dynamic resource allocation. This is a fundamental architectural choice — uniform computation per token — that the paper does not problematize or explore alternatives to.

Limitation 5: Practical Deployment Requires Careful Initialization and Token Handling — The Architecture Has Fragile Edge Cases Not Present in Transformers

The assumption or constraint. The paper's evaluation protocol for LM Evaluation Harness benchmarks assumes standard prompting with the <|endoftext|> token prepended to inputs. However, Appendix M reveals a significant sensitivity: omitting this token causes dramatic performance degradation on a specific class of examples (where the answer appears as the first word and is not repeated).

The consequence. On a subset of 142 LAMBADA examples where the answer is the first word and does not reappear, RWKV-7-0.1B accuracy drops from 36.6% to 9.2% (p < 0.001) when the <|endoftext|> token is omitted. RWKV-7-0.4B drops from 48.6% to 28.9% (p < 0.001). The paper identifies the root cause:

"This suggests that the model may struggle to retain the first token in memory."

This is a specific, diagnosable failure mode: RWKV-7's recurrent state at the start of a sequence is initialized to zero (Equation 16: wkv_0 = 0). The first token's value is stored in the state, but retrieval appears to depend on proper state initialization. Without the <|endoftext|> token, the state may begin in a regime where the first token's contribution is difficult to access. The paper also notes that "two consecutive <|endoftext|> tokens at the beginning can further improve performance," despite this pattern never appearing in the training corpus — suggesting the sensitivity is not merely about domain shift but about how the state dynamics function in the initial steps.

A practitioner deploying RWKV-7 as a base model for downstream applications must be aware of this sensitivity. The recommended prompt format (Appendix M):

<|endoftext|>User: <Your Question>
Assistant: <Assistant Answer>

is a workaround, but it constitutes a hidden dependency on a specific token that may break if users omit it or if the model is used in a setting where prepending <|endoftext|> is unnatural (e.g., zero-shot classification, sentence embedding extraction). In contrast, the paper shows that Qwen2.5-0.5B exhibits no significant sensitivity to the <|endoftext|> token on the same examples (Table 20: 47.9% vs. 54.9%, NS).

What evidence exists in the paper. Table 20 documents the effect across multiple models. The paper identifies 142 problematic examples out of 5,153 LAMBADA test cases — roughly 2.8% of the test set, which is small but systematic. The significance testing (p-values reported) confirms the effect is not noise. However, the paper does not investigate whether similar sensitivity exists on other benchmarks (HellaSwag, PIQA, ARC) or for longer prompts where the first few tokens are critical for context.

Mitigation status. The paper recommends a mitigation (always include <|endoftext|>) and provides a recommended chat template. This is practical advice but does not address the underlying architectural issue. A more robust solution — such as a learned initial state, a modified state initialization that does not zero out the state, or a mechanism that explicitly protects the first few tokens from being edited away — is not explored. The paper does not analyze why the <|endoftext|> token stabilizes performance; it merely documents the effect and recommends the workaround.

Limitation 6: Training and Inference Kernel Complexity Creates Numerical Precision Sensitivities That Complicate Deployment and Reproduction

The assumption or constraint. The RWKV-7 WKV kernel operates on 64 × 64 matrix-valued states per head, with a recurrence involving element-wise decay, a normalized key, and a rank-one update. The paper reports:

"We observed that some operators, particularly the WKV7 kernel, are sensitive to the numerical precision of the implementation. This highlights the need for careful handling of numerical precision during model deployment. We also observed differences in training dynamics when using different kernels, which implies that the correct handling of precision while calculating and applying state updates is of utmost importance in this architecture." (Section 10.1)

Furthermore, the paper uses an unusually small AdamW epsilon of 1 × 10^{-18} (Appendix E) and reports that "we did sometimes observe NaN loss across a single training step, which we theorize may be due to our use of such an extremely low AdamW ϵ" (Appendix E).

The consequence. The architecture is demonstrated to work under a specific, carefully tuned training setup: a custom bfloat16 kernel tuned for head dimension 64 on H100 GPUs, with an extremely small optimizer epsilon and a specific mitigation strategy for NaN losses (rewind to checkpoint, clear optimizer states). A practitioner attempting to reproduce the results — or train RWKV-7 models in a different environment (different GPU, different precision, different optimizer settings) — may encounter training instabilities or performance degradation that are not present in the paper's controlled setting. The paper acknowledges that the RWKV-7 fp32 kernel (used to train the World models "to maximize precision") is slower than the bfloat16 kernel, and that kernel efficiency "drops off at larger head dimensions" (Section 8).

This practical brittleness is a barrier to adoption. Transformer training is relatively robust across a wide range of hyperparameters and hardware configurations; RWKV-7 appears to require more careful tuning. The paper does not provide a systematic study of how performance varies with precision, kernel choice, or optimizer settings, making it difficult for a practitioner to anticipate and debug issues.

What evidence exists in the paper. The paper explicitly documents the precision sensitivity (Section 10.1) and the NaN loss workaround (Appendix E). Figure 9 shows the speed gap between the bfloat16 and fp32 kernels. Table 17 specifies exact learning rates, batch sizes, and phase boundaries for each model size — the level of detail suggests that deviations from these settings may be problematic. The paper notes that the noise parameter ϵ = 1 × 10^{-18} was chosen based on theoretical analysis by Molybog et al. (2023), but does not test whether more conventional values (1 × 10^{-8}) would work.

Mitigation status. The paper is transparent about the precision sensitivity and NaN issues, which is better than not disclosing them. The recommended initialization scheme is referenced but deferred to the code repository — a practitioner must consult external sources to reproduce training. The paper suggests future work on kernel optimization (Section 10.2) but does not propose architectural modifications that would reduce precision sensitivity (e.g., constraining the state update to use only well-conditioned operations). The mitigation for NaN losses (rewind and restart) is a training-time workaround, not a solution to the underlying precision fragility.

7. Implications and Future Directions

How This Work Changes the Landscape

RWKV-7 changes the landscape by proving that recurrent architectures for language modeling can cross the TC⁰ expressivity barrier without sacrificing training parallelizability — and by demonstrating that this expressivity translates to practical gains in sample efficiency and multilingual performance at scale. This is a conceptual advance, not merely an incremental refinement, because it resolves a tension that the field had implicitly accepted: the assumption that efficient (constant-time, constant-memory) sequence models must be less expressive than Transformers on state-tracking tasks.

The TC⁰ ceiling was treated as a fundamental limitation of efficient architectures. Merrill et al. (2024) established that Transformers and diagonal-state models are confined to TC⁰, and the natural interpretation was that escaping this limitation would require either (a) abandoning constant-time inference (by using a dense recurrent state, which is not parallelizable and suffers from vanishing gradients) or (b) paying the quadratic cost of attention. RWKV-7 demonstrates a third path: a minimal non-diagonal perturbation — a rank-one update added to a diagonal transition matrix — suffices to cross from TC⁰ to NC¹. The fact that this perturbation is exactly the generalized delta rule, which already had independent motivation from the memory-editing perspective (Section 2), makes the result more than a theoretical curiosity: the mechanism that enables state tracking is the same mechanism that enables selective memory editing, and both emerge from the same rank-one structure.

This reframes the design space for recurrent architectures. Before this work, the primary axis of innovation was how to decay the state — scalar vs. vector, data-dependent vs. fixed, per-head vs. per-channel. RWKV-7 reframes the question as how to edit the state — what operations can the transition matrix perform, and what expressivity do those operations enable? This shifts attention from the diagonal of the transition matrix (decay) to the off-diagonal structure (the rank-one term), and from the magnitude of state changes to their shape across dimensions. The Table 1 taxonomy makes this reframing explicit: prior work occupied subsets of the four-axis space (Large State, Flexible Decay, Dynamic Dependence, Generalized Eigenvalue); RWKV-7 occupies all four simultaneously, and the proof that this combination escapes TC⁰ gives the taxonomy theoretical teeth.

The paper reconciles conflicting findings about RNN state tracking. Prior work had shown that classical RNNs can recognize all regular languages but cannot be parallelized (Zucchet & Orvieto, 2024), while parallelizable architectures (Mamba, S4, linear attention) are provably limited to TC⁰. This created a narrative that parallelizability and expressivity are in fundamental tension. RWKV-7 resolves this: the diagonal-plus-rank-one structure is parallelizable (via the WKV kernel's chunked associative scan, exploiting techniques from Yang et al., 2024c) while retaining NC¹ expressivity. The conflict was not fundamental — it was an artifact of restricting the transition matrix to be purely diagonal. The paper thus converts an apparent impossibility result into a constructive existence proof with a practical implementation.

The practical consequence is a reallocation of scaling effort. The paper's results suggest that architecture matters more than token count in a specific, quantifiable way: RWKV-7-2.9B, trained on 5.6T total tokens (and only 3.1T in the final architecture), matches Qwen2.5-3B, trained on 18T tokens. This is a gap of roughly 3–5× in token efficiency, depending on how one accounts for the multi-architecture training history. If this efficiency gap holds at larger scales (a big "if" that the paper does not test), it implies that investing in better recurrent architectures may yield larger returns than investing in larger pretraining budgets — at least for models in the sub-10B parameter range where recurrent architectures have historically lagged. This does not make pretraining scaling obsolete, but it suggests that the field's overwhelming focus on scaling transformers may have left architectural gains on the table.

The paper also changes the landscape for open-source language model development. The release of Apache 2.0-licensed models, datasets, and training/inference code at competitive performance levels — matching proprietary-scale models (Qwen2.5, Llama-3.2) despite a fraction of the training budget — lowers the barrier to entry for groups without industrial compute resources. The model upgrade methodology (converting RWKV-5/6 checkpoints to RWKV-7 and continuing training) is particularly significant for resource-constrained research: it demonstrates that architectural innovation can be cumulative, not requiring a full pretraining restart. If this methodology generalizes to other architectural transitions, it could accelerate the pace of architecture research by reducing the cost of testing new ideas at scale.

However, the landscape change is not a paradigm shift. RWKV-7 does not obsolete Transformers or diagonal SSMs. It demonstrates that recurrent architectures can be more expressive than previously proven, but it does not demonstrate that they are as expressive as Transformers on all tasks of interest. The MMLU gap (55.0 vs. 65.7 for Qwen2.5-3B) suggests that knowledge-intensive tasks may still favor Transformer architectures or larger pretraining budgets. The long-context extrapolation results (Section 7.5) show dataset-dependent behavior that complicates the "constant-memory inference scales to arbitrary length" narrative. And the initial-token sensitivity (Appendix M) reveals a fragility that Transformers do not share. The paper's contribution is to open a new region of the Pareto frontier — efficient, expressive, sample-efficient recurrent models — not to claim that this region dominates all others.

Follow-Up Research This Work Enables

1. From-scratch training of RWKV-7 at matched token budgets to isolate the architectural contribution. The paper's headline efficiency claim conflates architecture, additional data (World v3 over v2.1), and continued training. A clean experiment would train RWKV-7 from scratch on exactly the same data as an RWKV-6 baseline (e.g., the full World v3 corpus, or the 332B-token Pile), with matched hyperparameter tuning, and compare final performance. The Pile-trained models in Table 18 provide partial evidence at 168M-1.47B scale, but the crucial scale is 1.5B-3B where the flagship results live. A from-scratch 1.5B RWKV-7 trained on 5.6T tokens of World v3 data — compared against an RWKV-6 model given the same budget — would either confirm the paper's speculation that from-scratch training widens the gap, or reveal that the upgrade methodology is more important than the architecture per se. The paper's release of training code and dataset composition makes this experiment reproducible for groups with sufficient compute.

2. Mechanistic analysis of whether trained RWKV-7 models learn the swap/copy operations constructed in the NC¹ proofs. The theoretical results (Appendix D) prove that RWKV-7 can implement swap and copy matrices in its transition matrix (Lemmas 1 and 3), and that these operations enable state tracking and regular language recognition. But the paper does not demonstrate that gradient-based training actually discovers these solutions. A follow-up study would train RWKV-7 models on the group multiplication task (Section 7.6) and then probe the learned WKV transition matrices: do they exhibit the structure of swap matrices (I - eᵀ_x e_x - eᵀ_y e_y + eᵀ_x e_y + eᵀ_y e_x) when processing swap tokens? Do they implement the copy operation (column y replaced by column x) when processing tokens that require state duplication? This could be done by analyzing the rank-one update matrices κ̂_tᵀ(a_t ⊙ κ̂_t) at each timestep and comparing their action on the state to the theoretically constructed matrices. A negative result — RWKV-7 solves state tracking without using the mechanisms from the proof — would suggest that the theoretical expressivity result does not explain practical behavior, redirecting attention to other aspects of the architecture (depth, nonlinearities, LayerNorm). A positive result would validate the paper's central claim that the rank-one edit is the mechanism enabling expressivity beyond TC⁰.

3. Characterizing the limits of long-context extrapolation through controlled pretraining experiments. The paper's finding that long-context behavior is dataset-dependent (World-trained models degrade beyond 10k, Pile-trained models do not) is surprising and unexplained. A controlled experiment would vary pretraining dataset properties — document length distribution, repetition rate, topic diversity — and measure the resulting PG19 extrapolation curves for RWKV-7 versus RWKV-6 and Mamba-2. The hypothesis to test: does RWKV-7's editable state make it more prone to learning context-length-specific strategies (because it can perform complex state edits that only make sense at the training context length), or is the World vs. Pile difference simply due to data repetition causing the model to memorize rather than generalize? This experiment would use a fixed small-scale architecture (e.g., RWKV-7-168M, comparable to the Pile ablation model) trained on multiple synthetic datasets with controlled document length distributions, measuring both perplexity extrapolation and pass-key retrieval accuracy. The outcome would either identify a fundamental limitation (recurrent models with editable states overfit to training context length) or a fixable training issue (data repetition, not architecture, causes the observed degradation).

4. Investigating whether the benefits of vector-valued over scalar gating are scale-dependent. The component ablation (Table 19) shows that scalar decay increases validation loss from 2.541 to 2.609 at 768 dimensions and 1.6B tokens — a meaningful but not dramatic difference. The paper argues these components are "synergistic" (Section 4), but the ablation tests components individually, not in combination. A rigorous scale-dependence study would train RWKV-7 variants at three scales (e.g., 168M, 421M, 1.5B parameters on the Pile dataset) with: (a) full vector-valued decay and ICLR, (b) scalar decay + vector ICLR, (c) vector decay + scalar ICLR, (d) scalar decay + scalar ICLR, and (e) no delta rule at all (pure decay-based RWKV, equivalent to an RWKV-6-style model). The prediction from the paper's narrative is that the gap between (a) and (d) should increase with model scale, because larger models can exploit the per-channel control more effectively. A flat or decreasing gap would undermine the argument that vector-valued gating is a qualitative threshold. This experiment also tests whether scalar-decay models (akin to Mamba-2 or Gated DeltaNet) catch up to RWKV-7 at large scale, which would constrain the practical importance of the architectural innovation.

5. Testing the "editable scratchpad" hypothesis through diagnostic synthetic tasks that require explicit state editing. The paper argues (Appendix F) that the RWKV-7 state functions as "an internal scratchpad" capable of operations that a Transformer's immutable KV cache cannot perform — swapping, copying, and deleting stored entries. A diagnostic benchmark could test this directly: design synthetic sequence tasks that require state editing and cannot be solved by append-only or decay-only strategies. For example: present a sequence of key-value pairs, then a sequence of "delete key X" and "copy key X to key Y" operations, and finally a query that requires retrieving a value that was moved or whose original location was overwritten. A pure decay model would fail on this task because it cannot selectively delete or copy; a Transformer could solve it with attention to the operation tokens, but would require O(N²) compute to re-attend to the full history. RWKV-7, if it learns to implement the swap/copy operations in its transition matrix, should solve it with O(1) memory and O(N) time. Evaluating RWKV-7, RWKV-6, Mamba-2, and a Transformer on this benchmark — with accuracy and state-inspection analysis — would either validate the editable-scratchpad hypothesis or reveal that trained RWKV-7 models rely on simpler strategies (e.g., learning to ignore the edit operations and attend to the original key-value pairs via decay-based retrieval).

6. Stress-testing the initial-token sensitivity and developing architectural mitigations. Appendix M documents a specific fragility: omitting the <|endoftext|> token causes catastrophic performance degradation on examples where the answer is the first word. This is an architectural vulnerability that Transformers do not share, and it limits RWKV-7's robustness in deployment scenarios where prompt formatting is not fully controlled. A systematic study would: (a) measure initial-token retention accuracy as a function of sequence position and state initialization (zero vs. learned initial state vs. special initialization token), (b) test whether the sensitivity extends to other "early context" information (the first sentence, the first paragraph), and (c) evaluate architectural mitigations — a learned non-zero initial state, a dedicated "first token protection" mechanism that prevents the delta rule from editing away the first token's contribution, or an explicit initial-token readout pathway analogous to the bonus term. The goal is not merely to document the fragility but to design an architectural fix that makes RWKV-7 as robust to prompt formatting as Transformers, which is a prerequisite for production deployment as a general-purpose language model.

Practical Applications and Downstream Use Cases

1. Multilingual deployment with sub-3B models on resource-constrained hardware. RWKV-7-2.9B achieves 61.1% average accuracy on multilingual benchmarks, substantially outperforming Qwen2.5-3B (55.6%) and Llama-3.2-3B (58.1%) while requiring only constant memory per token during inference (Section 8). For applications such as on-device translation, multilingual document understanding on edge devices, or cross-lingual assistants in low-resource languages, this combination — strong multilingual performance in a small memory footprint — is directly actionable. A deployment on a mobile device with 4GB of RAM could run RWKV-7-2.9B for arbitrarily long multilingual inputs without the memory growth that a Transformer's KV cache would incur. The 4× training token efficiency (5.6T vs. 18T) also means that domain-specific multilingual models could be trained for individual languages or language families at proportionally lower cost than Transformer-based alternatives.

2. Long-context document processing with explicit extended-context fine-tuning. The pass-key retrieval results (Figure 7) show that after fine-tuning on packed 128k-token sequences, RWKV-7-2.9B reliably retrieves information up to 50,000 tokens. For applications like legal document review, scientific literature synthesis, or long-form report generation — where the model must reference information distributed throughout a 30k-50k token context — RWKV-7 offers a practical advantage: the constant memory usage means inference cost does not grow with context length once the state is established. A deployment processing 50,000-token legal filings would see roughly the same per-token inference cost as processing a 500-token query, whereas a Transformer's KV cache would consume memory proportional to 50,000 × model dimension × 2 bytes per float16 ≈ hundreds of megabytes per layer per batch element. The paper's provided fine-tuning recipe (Table 8, document-length-based weighting) and dataset composition give practitioners a concrete starting point for adapting RWKV-7 to their specific long-context domain.

3. Training data generation for self-improvement loops where sample efficiency matters. The paper's finding that RWKV-7 achieves competitive performance with 3–5× fewer training tokens suggests it is a strong candidate for the generator role in self-improvement pipelines (e.g., STaR, rejection sampling fine-tuning). In such pipelines, a model generates candidate solutions, which are filtered by correctness and used to fine-tune the model. If the generator is sample-efficient (needs fewer tokens to reach a given performance level), each iteration of the loop requires less computation, accelerating the overall improvement cycle. More speculatively, RWKV-7's editable state mechanism might enable the model to learn from rejected generations during inference — a revision-like capability where an incorrect first attempt is edited into a correct one within the context, using the state update mechanism to refine stored associations. The paper provides no direct evidence for this capability, but the architectural foundation (selective removal and replacement of stored key-value pairs) is exactly what such a mechanism would require.

4. Architectural comparison and benchmarking for the RNN research community. The release of Pile-trained RWKV-7 models (168M, 421M, 1.47B) using the GPT-NeoX tokenizer — the same tokenizer and dataset used by Pythia, Mamba, and other open-source models — enables apples-to-apples architectural comparisons at matched data and tokenizer. A researcher can now run controlled experiments comparing RWKV-7 against Mamba-2, Pythia, or any other Pile-trained model on a new diagnostic task, knowing that performance differences are attributable to architecture rather than data or tokenizer artifacts. The paper's detailed architecture description (Section 4, Appendices E-H), including the exact low-rank MLP dimensions (Table 16) and initialization scheme (Appendix E code repository reference), lowers the barrier to implementing and modifying RWKV-7 for research purposes. The release of training code and CUDA kernels under Apache 2.0 means that architectural variants (e.g., testing different rank-one update structures, different gating mechanisms) can be trained and compared without reimplementing the core WKV recurrence from scratch.

When to Prefer This Method

The paper articulates an explicit tradeoff between RWKV-7 (constant-time inference, editable state, lower training cost) and Transformers (quadratic attention, immutable KV cache, higher training cost but established ecosystem). The comparison against Qwen2.5 and Llama-3.2 throughout Section 7 frames this as a decision between architectural efficiency and total training compute. The decision rule is:

• Prefer RWKV-7 when (a) deployment involves long-context inference (10k+ tokens) where Transformer KV-cache memory would exceed hardware limits, and (b) the model can be explicitly fine-tuned for the target context length, since long-context extrapolation does not emerge automatically from pretraining (Section 7.5, Figure 6 vs. Figure 5); or (c) the task involves state tracking, associative recall, or structured memory operations where the paper demonstrates advantages — multilingual benchmarks (RWKV-7-2.9B: 61.1% vs. Qwen2.5-3B: 55.6%, Table 4), associative recall (MQAR: 72.93% at 2048 sequence length with 256 KV pairs, Section 7.3), and fuzzy recall (MAD benchmark: 43.2% vs. Transformer: 29.8%, Table 7); or (d) training compute is the primary constraint and sample efficiency matters — RWKV-7 matched Qwen2.5's English performance with ~3× fewer total training tokens (5.6T vs. 18T, Table 3).

• Prefer Transformers when (a) the task requires broad factual knowledge measured by MMLU, where RWKV-7-2.9B trails Qwen2.5-3B by 10.7 points (55.0 vs. 65.7, Table 3), or (b) prompt robustness is critical and input formatting cannot be guaranteed to include the <|endoftext|> token — the 27.4-point accuracy drop on initial-token LAMBADA examples without this token (Appendix M, Table 20) signals a fragility that Transformers do not share (Qwen2.5-0.5B: no significant effect), or (c) the deployment environment is a standard short-context setting (<4k tokens) with established Transformer infrastructure (Flash Attention, vLLM, TensorRT-LLM) where the ecosystem maturity outweighs architectural efficiency.

The paper does not articulate a tradeoff against other recurrent architectures (Mamba-2, Gated DeltaNet) in terms of a clear decision rule — the comparisons are empirical but not distilled into a "prefer X when Y" format. The Pile-trained results (Table 18) show RWKV-7 outperforming Mamba-2 at 1.47B scale (62.6% vs. 61.2%), but the gap is small and the context-length, task-dependence, and robustness dimensions are not systematically compared.