ArXiv: 2505.03005

🎯 Pitch

A full softmax-attention transformer can be converted into a pure linear-attention RNN with near-zero quality loss using under 700 million tokens—less than 0.005% of the original training budget—slashing the conversion cost for a 72B model below $2,000. The resulting QRWKV models hit 100.4% relative accuracy on LAMBADA, setting a new state of the art for efficient recurrent language models.

1. Executive Summary

This paper introduces RADLADS (Rapid Attention Distillation to Linear Attention Decoders at Scale), a three-step protocol for converting pretrained softmax-attention transformer models into purely recurrent linear-attention decoder models while preserving most of the original model’s quality. Using Qwen2.5 open-source models at 7B, 32B, and 72B scales, the process applies attention hidden-state alignment (training student time-mixing layers to match the teacher’s attention outputs via L2 loss), knowledge distillation (minimizing KL divergence between student and teacher logits), and context-length extension (finetuning on longer sequences) — requiring only 350–700M tokens total, less than 0.005% of the teacher’s pretraining tokens and costing under $2,000 for the 72B model. The converted QRWKV models achieve state-of-the-art downstream performance among pure RNN language models, with the 72B variant reaching 100.4% relative accuracy on LAMBADA and 89.9% on MMLU, establishing that near-parity with the original transformer is achievable even when softmax attention is replaced entirely, provided the target RNN architecture — here, the authors’ custom RAD-RWKV6 (“RADFinch”) and RAD-RWKV7 (“RADGoose”) designs — is well-matched to the conversion process.

2. Context and Motivation

The Core Problem: Training Large Linear-Attention Models Is Prohibitively Expensive

The fundamental tension this paper addresses is economic and practical: linear-attention transformer variants promise dramatic efficiency gains at inference time, but their adoption is bottlenecked by the enormous cost of training them from scratch. This gap matters because, as the authors observe in Section 1, the most capable linear-attention models have begun to match or even exceed the quality of traditional softmax-attention transformers (Peng et al., 2025), yet "the cost of training usable large models at scale is cost prohibitive for all but the largest organizations" — state-of-the-art pretraining runs for large language models routinely require processing over 10 trillion tokens.

This tension has sharpened as the field has progressed. On one side, linear attention offers clear, well-understood advantages: O(1) time per token at inference instead of O(N) for softmax attention, and elimination of the memory-bandwidth-intensive key-value cache (Arora et al., 2023). These properties make linear-attention models particularly attractive for long-context applications, edge deployment, and any setting where inference latency or memory footprint is a binding constraint. On the other side, the open-source ecosystem and research community have already invested enormous collective resources into pretraining high-quality softmax-attention models — models like the Qwen2.5 family, Llama 3, and Mistral — whose weights are publicly available and whose capabilities are well-characterized. The economic logic is straightforward: if there exists a way to convert these existing transformers into functionally equivalent linear-attention models at a fraction of the original training cost, then the community gets the best of both worlds — the inference efficiency of linear attention and the quality of multi-trillion-token pretrained transformers, without requiring every research group to fund their own massive training run.

Why Conversion, Not Pretraining from Scratch?

The paper's approach is motivated partly by a resource asymmetry in the current landscape. A handful of well-funded organizations (Meta, Alibaba/Qwen, Mistral, Google) have the capital to pretrain models at the 70B+ scale, releasing the resulting weights openly. A much larger community of researchers, startups, and applied practitioners wants to use those models — and increasingly, wants to deploy them efficiently. If the only path to a high-quality 72B linear-attention model is to pretrain one from scratch on 10T+ tokens, then that path is effectively closed to all but the largest industrial labs. The paper's central bet is that conversion can break this dependency, enabling a small team with modest compute (the 72B conversion cost under $2,000) to produce a linear-attention model competitive with the original transformer.

There is also a research velocity argument embedded in the motivation. The authors explicitly note (Section 1) that RADLADS "opens up new avenues for researchers who work on the next generation of compressive state attention variants to test, train, and release models containing their new designs at scale." If every new RNN architecture proposal needed a full pretraining run to demonstrate viability at the 7B+ scale, architectural innovation would slow to a crawl. Conversion provides a rapid feedback loop: design a new time-mixing block, convert an existing transformer, evaluate, iterate — all in days rather than months.

Where Prior Conversion Approaches Fall Short

The paper situates itself within a lineage of attempts to convert pretrained transformers to recurrent architectures, all of which exhibit a consistent pattern: they either require too much training data, fail to preserve model quality, or resort to hybrid designs that retain some softmax attention (and therefore sacrifice the O(1) inference guarantee). The authors characterize these shortfalls systematically in Sections 1 and 2.

Early work: full logit distillation from scratch. Gerstenberger et al. (2020) performed logit distillation from a pretrained transformer teacher to a freshly initialized recurrent student. This approach preserves no weights from the teacher — the student learns everything through the distillation objective alone — and consequently requires "a very long training cycle." The fundamental inefficiency is that knowledge embedded in the teacher's MLP layers and embeddings, which represent the bulk of the model's parameters and factual knowledge, is discarded and must be relearned from the distillation signal. This is a poor use of available information.

T2R: weight reuse with traditional linear attention. Kasai et al. (2021) improved on this by keeping most of the original model intact — MLPs, embeddings, and layer structure are preserved — and only swapping out the softmax attention mechanism. The replacement is a traditional linear attention module using a learnable MLP feature map, followed by finetuning on approximately 3.5 billion tokens. While this is substantially more efficient than training from scratch, two limitations persist: (a) 3.5B tokens is still a significant training budget, roughly 5–10× more than what RADLADS uses; and (b) traditional linear attention is now understood to be less expressive than modern RNN architectures like RWKV-6/7 or Mamba (Sun et al., 2023; Yang et al., 2024; Peng et al., 2025), so the student model's representational capacity is inherently limited.

SUPRA and DiJiang: combining weight transfer with long training. Mercat et al. (2024) (SUPRA) and Chen et al. (2024) (DiJiang) combine the weight-transfer strategy of T2R with full model distillation, but both require around 100B tokens of training — two orders of magnitude more than RADLADS. Moreover, as Table 1 shows, their downstream performance exhibits notable deficiencies: SUPRA achieves only 0.216 relative score on MMLU, and DiJiang produces a negative relative score on MMLU (meaning its accuracy is worse than random guessing). These results demonstrate that large training budgets alone do not guarantee quality preservation; the details of the architecture, training protocol, and dataset choice matter enormously.

GSA: architecture designed for conversion. Zhang et al. (2024b) introduced Gated Slot Attention (GSA), an architecture explicitly designed to be compatible with a SUPRA-like conversion process. This represents progress in recognizing that the target architecture matters for conversion quality — a theme RADLADS amplifies — but GSA still requires long training cycles, and its benchmark performance is not competitive with RADLADS models at comparable scale.

Mamba in the Llama: progressive distillation with hybrid compromise. Wang et al. (2024) developed a pipeline of progressive distillation, supervised fine-tuning, and direct preference optimization to convert Llama models into hybrid Mamba-Transformer architectures. Their approach uses 20B tokens — less than SUPRA/DiJiang but still 40–60× more than RADLADS — and critically, it produces hybrid models that retain some proportion of softmax attention layers. The paper notes (Section 2) that "it focuses on hybrid models rather than pure RNNs, and performs poorly when the softmax attention is removed completely." This is a crucial observation: hybrid models do not achieve O(1) inference complexity, because the retained softmax attention layers still require O(N) computation and a KV cache. They represent a partial efficiency gain, not the full promise of linear attention. The poor performance in the pure-RNN regime suggests that Mamba in the Llama's distillation protocol is not sufficiently effective at transferring the teacher's capabilities when all attention is replaced — precisely the regime RADLADS targets.

MOHAWK: two-phase training with remaining data hunger. Bick et al. (2024a) introduced a key conceptual advance: separating the conversion process into an attention-alignment phase (matching attention outputs between student and teacher) followed by a knowledge-distillation phase (matching logits). This two-phase decomposition is a direct precursor to RADLADS's three-step protocol. However, MOHAWK still requires 3–5B tokens of training — roughly 10× what RADLADS needs — and its reported downstream performance (Table 1, relative MMLU score of −0.047) indicates severe quality degradation on some benchmarks, with MMLU accuracy actually below the random-guessing baseline. The paper does not elaborate on why MOHAWK underperforms despite the structural similarity of its approach, but the implication is that the specific architectural choices (MOHAWK uses Mamba-based SSMs rather than RWKV variants), hyperparameter schedules, and dataset selection are all critical variables that MOHAWK did not fully optimize.

LOLCats/Hedgehog: extremely low data, compromised by LoRA and weak architecture. Zhang et al. (2024a) achieved the most data-efficient conversion prior to RADLADS, requiring only 40M tokens — an order of magnitude less than RADLADS. This appears to challenge RADLADS's central claim of efficiency. However, the paper directly addresses this comparison (Section 2), and the explanation reveals important trade-offs. LOLCats's downstream performance is poor: its relative MMLU score is −0.288 (again, worse than random guessing), and its scores across other benchmarks (arc_c, arc_e, PIQA, Winogrande) are substantially lower than RADLADS models. The authors attribute this to two factors:

  1. Use of LoRAs for training. Low-rank adaptation restricts the model's capacity to adapt to the new sequence mixer. Rank reduction "was generally quite detrimental to performance" (Section 8, negative results), suggesting that full-weight training — or at least training without rank constraints — is important for achieving high-quality conversion.

  2. Vanilla linear attention rather than a more advanced recurrent architecture. Traditional linear attention, even with the denominator-removal modifications used in T2R and SUPRA, is less expressive than the gated, data-dependent recurrence of RWKV-6 or RWKV-7. The authors note that "other modern RNN architectures have been shown to be more expressive than linear attention" (Section 3), and the RADLADS results bear this out: a better-matched architecture enables better quality at a given training budget.

To compensate for these quality deficiencies, LOLCats introduces a hybrid variant with full softmax attention in a sliding window (SWA). This improves scores — bringing MMLU above random — but produces a model that is "no longer a purely recurrent model." The authors observe that even with SWA, LOLCats's relative MMLU scores "remain low compared to ours" (Table 5 vs. Table 3). The LOLCats comparison thus crystallizes the paper's core design philosophy: data efficiency must not come at the cost of architectural expressiveness or full-weight adaptation. RADLADS achieves strong downstream performance with modest (not minimal) data because it pairs a well-chosen RNN architecture with full-weight training and carefully tuned hyperparameters.

ARWKV: sharing early RADLADS techniques, revealing sensitivity to details. Yueyu et al. (2025) published ARWKV, which converts Qwen2.5-7B-Instruct to a standard RWKV-7 sequence mixer, after receiving early access to RADLADS code, techniques, and a 32B model. (The paper discloses this collaboration in Section 2.) ARWKV achieves respectable performance — 0.893 relative LAMBADA, 0.801 relative MMLU — but this is notably lower than RADLADS's own QRWKV7-7B-Instruct conversion (0.982 LAMBADA, 0.924 MMLU). The authors attribute this gap to differences in "specific choices for weight transfer, hyperparameters, dataset, and architecture," and argue that this demonstrates these choices "matter significantly." The ARWKV comparison thus serves double duty: it validates that the general approach (weight transfer + two-phase alignment/distillation to RWKV-7) works when replicated by an independent team, while simultaneously underscoring that the details of the protocol — the exact learning rate schedules, the dataset (DCLM vs. alternatives), the architectural modifications (RAD-RWKV7 vs. standard RWKV-7) — are what separate SOTA performance from merely good performance.

Llamba: concurrent work at larger scale with more data. Bick et al. (2025) distilled Llamba-8B from a Llama-3.1-70B teacher, using 8–12B tokens — roughly 15–25× more data than RADLADS. Llamba achieves strong results (0.951 LAMBADA, 0.837 MMLU relative scores), and the authors acknowledge it as "a generous comparison" since it is distilled from a 70B teacher to an 8B student (vs. RADLADS's 7B-from-7B or 7B-from-72B). RADLADS's data efficiency advantage is clear: comparable or better quality with an order of magnitude less training data.

Reconciling the Prior Landscape: What Made Conversion Hard

Synthesizing the history of prior attempts reveals that conversion quality is governed by a small set of interacting factors, none of which had been simultaneously optimized before RADLADS:

  • Architectural expressiveness of the target RNN. Traditional linear attention underperforms modern gated, data-dependent recurrence. This is not a trivial observation — it means that the choice of target architecture is a first-order determinant of conversion quality, not merely an implementation detail. The paper's development of RAD-RWKV6 and RAD-RWKV7 (Section 3, Appendix B) reflects the recognition that off-the-shelf RWKV architectures were "sometimes imperfectly matched to the needs of conversion," requiring modifications such as removal of the off-by-one decay and bonus, adoption of a Gated Linear Attention kernel, and for RAD-RWKV7, removal of tokenshift and addition of RoPE.

  • Training data volume vs. quality vs. relevance. Prior work spans from 40M tokens (LOLCats) to 100B (SUPRA), but neither extreme produced high-quality pure-RNN models — LOLCats's scores were too low, SUPRA's training was too expensive. RADLADS's 350–700M tokens on DCLM represents a middle ground, but the choice of DCLM specifically matters: the authors tried FineWeb and FineWeb-Edu but found DCLM "worked exceptionally well for us when converting Qwen models," and theorize that "the best choice of dataset may depend upon the teacher model's pretraining data distribution."

  • Weight transfer and initialization strategy. T2R established that preserving teacher weights where possible is beneficial, but the details matter: copying Q/K/V/O weights directly, initializing new parameters to mimic the teacher's behavior, and understanding which components (embeddings, MLPs) encode the bulk of factual knowledge all affect convergence speed and final quality.

  • Training schedule and learning rate design. The paper's hyperparameter philosophy (Section 3) is that "most knowledge resides in the teacher model's MLPs and embeddings," so the learning rate should start high to rapidly align attention hidden states, then anneal to the teacher's final pretraining learning rate, remaining low during distillation and finetuning "to avoid catastrophic forgetting or overly significantly changing the knowledge encoded in the original teacher MLPs."

  • Full-weight vs. parameter-efficient training. LOLCats's LoRA-based approach and the ablation results in Section 6 (showing that removing tokenshift, gating, or adding GroupNorm all degrade performance) suggest that rank-constrained or component-restricted training limits conversion quality.

How This Paper Positions Itself

RADLADS positions itself as the first approach to simultaneously achieve all of the following: (a) pure RNN (no retained softmax attention — full O(1) inference), (b) state-of-the-art downstream performance (matching or exceeding all prior pure-RNN conversions and many hybrid conversions), (c) very low training data requirements (350–700M tokens, 0.005% of pretraining), (d) low cost (< $2,000 for 72B), and (e) open-source release of both models and conversion code.

The paper's framing in Section 2 and Table 1 makes this positioning explicit: each prior work excels on some dimensions but fails on others. SUPRA gives pure RNNs but needs 100B tokens and has low MMLU. LOLCats needs minimal data but has poor scores without SWA and becomes hybrid with SWA. MOHAWK separates alignment and distillation but still requires billions of tokens. Mamba in the Llama is strong on quality but is hybrid, not pure RNN. Llamba achieves high quality but uses 8–12B tokens and distills from a larger teacher. RADLADS claims to be the first to hit all the desirable points simultaneously, and the benchmark tables are organized to make this comparison legible.

The paper also positions itself in Section 8 as transparent about negative results — sharing what didn't work (initial attention-score alignment, skipping step 1, freezing MLP weights, larger batch sizes, LoRA training, switching datasets during context extension) — which serves both as practical guidance for replicators and as evidence that the final protocol was arrived at through systematic experimentation rather than fortuitous tuning.

3. Technical Approach

This is primarily a systems and methods paper whose core idea is that converting a pretrained softmax-attention transformer into a purely recurrent linear-attention model can be made fast, cheap, and high-quality by decomposing the process into three sequential steps — attention hidden-state alignment, knowledge distillation, and context-length extension — each with carefully chosen objectives, learning rates, and data volumes, and by pairing this protocol with custom RNN architectures (RAD-RWKV6 and RAD-RWKV7) that are explicitly designed to be compatible with the conversion process rather than used off-the-shelf.

3.1 Reader Orientation

The RADLADS system is a training protocol plus a pair of architecture specifications that, given a pretrained transformer model (the teacher), produces a linear-attention decoder-only language model (the student) whose sequence-mixing layers are purely recurrent — meaning each token's computation depends on a fixed-size state rather than growing with context length — yet whose downstream performance on standard benchmarks remains close to the original transformer. The problem it solves is the cost barrier to obtaining high-quality linear-attention models: training such models from scratch requires multi-trillion-token pretraining runs accessible only to the largest organizations, whereas RADLADS converts an existing open-weight transformer (here, Qwen2.5) into a competitive linear-attention model using only 350–700 million tokens and less than $2,000 in compute for the 72B scale. The "shape" of the solution is a three-step distillation pipeline that begins by aligning the new recurrent time-mixing layers to produce the same hidden-state outputs the original attention layers would have produced, then trains the full model to match the teacher's output logits via KL divergence, and finally finetunes at longer sequence lengths — all while keeping the teacher's MLP and embedding weights largely intact to preserve factual knowledge, and using custom RWKV-derived architectures stripped of components that interfere with conversion.

3.2 Big-Picture Architecture (Diagram in Words)

The RADLADS system has five major components, seen from the perspective of the conversion process:

  1. Teacher model (frozen softmax-attention transformer) — the pretrained Qwen2.5 model at 7B, 32B, or 72B scale. All MLP layers, embedding matrices, layer norms, and non-attention parameters are preserved and transferred directly to the student. The teacher's attention layers provide two signals during training: their hidden-state outputs (used as targets in step 1) and their final logits (used as targets in step 2). The teacher is held in memory without gradient computation.

  2. Student model (trainable linear-attention decoder) — an exact structural copy of the teacher, with one critical substitution: every softmax-attention block is replaced by a RAD-RWKV6 or RAD-RWKV7 time-mixing block. All other components (MLPs, embeddings, layer norms, output head) are initialized from the teacher's weights. Where the student's recurrent block has parameters with direct analogs in the teacher (queries → receptance, keys → key, values → value, output projection → output projection), these are initialized from the teacher's corresponding weight matrices. New parameters specific to the RNN architecture (decay, tokenshift, gating mechanisms) are initialized to mimic the teacher's behavior — for instance, tokenshift parameters are set to have no immediate effect.

  3. Step 1 training loop: attention hidden-state alignment — during this step, a frozen copy of the full teacher model and the partially-initialized student model coexist in memory. For each transformer layer, the teacher's original softmax-attention block and the student's new RAD-RWKV block run in parallel on the same input hidden states. An L2 loss (or mean squared error) is computed between the teacher attention block's output and the student time-mixing block's output, summed or averaged across layers. Only the student's time-mixing parameters are updated; the teacher and all shared components (MLPs, embeddings) are frozen. At the end of this step, the teacher attention blocks are physically removed from the model, leaving a complete student model whose time-mixing layers approximate the teacher's attention outputs.

  4. Step 2 training loop: knowledge distillation — a separate, frozen copy of the full teacher model is loaded alongside the student. Both models process the same input sequences independently. The loss is the Kullback-Leibler divergence between the student's output logits and the teacher's output logits. All student parameters (time-mixing blocks, MLPs, embeddings, output head) are trainable. This step trains the full model end-to-end to recover the teacher's next-token prediction distribution.

  5. Step 3 training loop: context-length extension — the teacher is discarded entirely. The student model is trained on longer sequences (16,384 tokens vs. 512 in steps 1–2) using standard cross-entropy loss with the language modeling objective (no teacher signal). All parameters are trainable by default, though an optional memory-saving variant (step 3a) freezes all weights except decay and tokenshift parameters and raises the learning rate.

Information flows through the system sequentially: the teacher provides initialization weights and training targets → step 1 aligns per-layer hidden states → step 2 aligns full-model output distributions → step 3 adapts the model to function at longer contexts. After step 3, the result is a standalone linear-attention language model with no remaining dependency on the teacher.

3.3 Roadmap for the Deep Dive

  • First, the architectural design of RAD-RWKV6 and RAD-RWKV7 — the time-mixing blocks that replace softmax attention — because all subsequent training steps depend on these architectures' properties, and understanding why specific components were added or removed clarifies the design philosophy behind the conversion protocol.
  • Second, the weight transfer and initialization procedure (Setup), which determines the starting point for optimization and encodes assumptions about where the teacher's knowledge resides.
  • Third, step 1 (attention hidden-state alignment) in full detail — its objective function, training configuration, the rationale for the cosine learning rate schedule, and what convergence looks like.
  • Fourth, step 2 (knowledge distillation) — why KL divergence rather than cross-entropy or L2 on logits, the flat learning rate choice, the token budget, and the relationship between step 1 quality and step 2 requirements.
  • Fifth, step 3 (context-length extension) — the memory/throughput motivation for making this a separate step, the standard and low-VRAM variants, and the design choices around dataset and learning rate.
  • Sixth, the hyperparameter summary and training infrastructure (Table 7, Table 8) as a single reference point, with commentary on the logic behind each choice.
  • Seventh, the ablation studies and negative results that validate the architectural and procedural choices — what was tried and rejected, and what that teaches us about the conversion problem.

3.4 Detailed, Sentence-Based Technical Breakdown

This paper develops a three-step protocol for converting any pretrained softmax-attention transformer into a purely recurrent linear-attention decoder, along with two custom RNN time-mixing architectures (RAD-RWKV6 and RAD-RWKV7) designed to maximize conversion quality with minimal training tokens. The student model is an exact structural copy of the teacher except that every softmax-attention block is replaced by one of these recurrent blocks; all other components (MLPs, embeddings, layer norms) are initialized from the teacher's weights and fine-tuned during conversion.

RAD-RWKV6 Time Mixing ("RADFinch")

RAD-RWKV6 is a simplified variant of RWKV6-C2 ("Finch-C2," described in Goldstein et al., 2024) with three structural modifications motivated by the needs of conversion: removal of the bonus term, adoption of a Gated Linear Attention kernel, and use of state balancing (via $k_t = \tilde{k}_t(1 - w_t)d_k^{-0.5}$) to eliminate state normalization, which the authors found improves downstream performance. The architecture operates per-layer, per-timestep, transforming an input hidden state $x_t \in \mathbb{R}^D$ into an output $o_t \in \mathbb{R}^D$ through a sequence of data-dependent gating, key-value state updates, and a recurrent state $\mathbf{wkv}_t \in \mathbb{R}^{(D/h) \times (D/h)}$ maintained across timesteps.

Token shift (ddlerp). Before computing any attention-related quantities, RAD-RWKV6 applies a data-dependent linear interpolation (ddlerp) between the current input $x_t$ and the previous timestep's input $x_{t-1}$. The ddlerp mechanism is defined as:

lora□(x)=λ□+tanh⁡(xA□)B□\mathrm{lora}_{\square}(x) = \lambda_{\square} + \tanh(x \mathbf{A}_{\square}) \mathbf{B}_{\square}

ddlerp□(a,b)=a+(b−a)⊙lora□(a+(b−a)⊙μx)\mathrm{ddlerp}_{\square}(a, b) = a + (b - a) \odot \mathrm{lora}_{\square}(a + (b - a) \odot \mu_x)

where $\square$ is a placeholder for the variable being computed (r, v, g, $\tilde{w}$, or $\tilde{k}$), $\lambda_{\square} \in \mathbb{R}^D$ is a trainable bias vector, $\mu_x \in \mathbb{R}^D$ is a shared trainable vector, and $\mathbf{A}_{\square} \in \mathbb{R}^{D \times z}$, $\mathbf{B}_{\square} \in \mathbb{R}^{z \times D}$ are low-rank trainable matrices with rank $z$ (a chosen hyperparameter controlling the ddlerp bottleneck dimension). Each variable type (receptance, value, gate, decay precursor, key precursor) has its own independent $\mathbf{A}_{\square}, \mathbf{B}_{\square}, \lambda_{\square}$ but shares $\mu_x$.

What it computes: ddlerp is a learned interpolation between two consecutive hidden states. The inner term $a + (b - a) \odot \mu_x$ produces a data-dependent mixture controlled by the shared vector $\mu_x$, and the $\tanh$-gated low-rank transformation $\mathrm{lora}_{\square}(\cdot)$ generates token-specific interpolation coefficients via a bottleneck $\mathbb{R}^D \to \mathbb{R}^z \to \mathbb{R}^D$. The final operation $a + (b - a) \odot \mathrm{lora}_{\square}(\cdot)$ is an element-wise linear interpolation where the blend ratio is learned per-channel from the content. This gives each component (receptance, key, value, gate, decay) access to a token-shifted representation that mixes the current and previous hidden states in a data-driven way, introducing a form of local temporal convolution without fixed convolutional weights — the convolution kernel is input-dependent.

Why this form: standard 1D convolutions (used in Mamba and early RWKV for token mixing) have fixed kernels, meaning the same mixing coefficients apply regardless of content. The ddlerp mechanism makes the token shift content-dependent, which the authors found critical for conversion quality: it allows the recurrent block to learn, during training, how much of the previous vs. current hidden state to use for each downstream computation, adapting to the patterns that the original softmax attention would have captured. The low-rank bottleneck ($\mathbf{A}_{\square} \in \mathbb{R}^{D \times z}$ with small $z$) keeps the parameter count manageable — full-rank $\mathbb{R}^{D \times D}$ matrices would add $D^2$ parameters per variable, which would be prohibitive at the 7B+ scale.

Receptance, value, gate, decay precursor, key precursor. Each of these five vectors is computed by applying the corresponding ddlerp followed by a linear projection:

rt=ddlerpr(xt,xt−1)Wr(receptance)r_t = \mathrm{ddlerp}_r(x_t, x_{t-1}) \mathbf{W}_r \quad \text{(receptance)} vt=ddlerpv(xt,xt−1)Wv(value)v_t = \mathrm{ddlerp}_v(x_t, x_{t-1}) \mathbf{W}_v \quad \text{(value)} gt=σ(ddlerpg(xt,xt−1)Wg)(gate)g_t = \sigma(\mathrm{ddlerp}_g(x_t, x_{t-1}) \mathbf{W}_g) \quad \text{(gate)} w~t=ddlerpw~(xt,xt−1)Ww~(decay precursor)\tilde{w}_t = \mathrm{ddlerp}_{\tilde{w}}(x_t, x_{t-1}) \mathbf{W}_{\tilde{w}} \quad \text{(decay precursor)} k~t=ddlerpk~(xt,xt−1)Wk~(key precursor)\tilde{k}_t = \mathrm{ddlerp}_{\tilde{k}}(x_t, x_{t-1}) \mathbf{W}_{\tilde{k}} \quad \text{(key precursor)}

where $\mathbf{W}_r, \mathbf{W}_v, \mathbf{W}_g, \mathbf{W}_{\tilde{w}}, \mathbf{W}_{\tilde{k}} \in \mathbb{R}^{D \times D}$ (or $\mathbb{R}^{D \times (D/h)}$ when accounting for multi-head splitting) are trainable weight matrices. The gate $g_t$ passes through a sigmoid $\sigma(\cdot)$ to produce values in $(0, 1)$ for multiplicative gating.

Why these are separate projections: in standard softmax attention, Q, K, V are produced by independent linear projections of the same input, and the output passes through an output projection. RAD-RWKV6 mirrors this structure — receptance $r_t$ is analogous to the query (it reads from the recurrent state), value $v_t$ is analogous to the value (it writes to the state), key $k_t$ is analogous to the key (it determines addressing), gate $g_t$ provides a learned multiplicative filter on the output (similar to the gating in Gated Linear Attention), and decay $w_t$ controls how quickly old information is forgotten. The use of independent ddlerp transformations for each (with independent low-rank matrices) allows each to learn different temporal mixing patterns from the same pair of consecutive hidden states.

Decay computation. The decay $w_t$ controls how much of the previous recurrent state is retained at each timestep. It is computed from the decay precursor $\tilde{w}_t$ through a chain of transformations designed to constrain its range:

wt=exp⁡(max⁡(−exp⁡(loraw(w~t)),−5))w_t = \exp(\max(-\exp(\mathrm{lora}_w(\tilde{w}_t)), -5))

where $\mathrm{lora}_w(\cdot)$ is defined as in the ddlerp formula (a $\tanh$-gated low-rank transformation of $\tilde{w}_t$ with independent $\mathbf{A}_w, \mathbf{B}_w, \lambda_w$).

What it computes: the inner $\exp(\mathrm{lora}_w(\tilde{w}_t))$ produces a positive value from the decay precursor via a learned low-rank transformation followed by exponentiation. Negating this gives a negative number. The $\max(\cdot, -5)$ clamps it to be no more negative than −5 — preventing the inner exponential from producing values larger than $e^5 \approx 148$. The outer $\exp(\cdot)$ then converts this clamped negative value to a number in $(e^{-5}, 1] \approx (0.0067, 1]$. The result $w_t$ is a per-channel decay factor between roughly 0.007 and 1.0.

Why this form: the double-exponential parameterization $\exp(-\exp(\cdot))$ with clamping ensures that $w_t$ is always in $(0, 1]$ — strictly positive and never exceeding 1 — which is required for the recurrent state update to be stable (if $w_t > 1$, the state norm would grow exponentially; if $w_t = 0$, information would be instantly erased with no gradient). The $\max(\cdot, -5)$ prevents $w_t$ from becoming pathologically small (below $e^{-5}$), which would make the model essentially amnesic. The learnable transformation $\mathrm{lora}_w$ allows each channel to learn its own forgetting rate based on content, enabling the model to retain important information for varying durations.

Key computation. The final key $k_t$ is computed from the key precursor $\tilde{k}_t$ with a decay-dependent scaling and a dimensionality correction:

kt=k~t(1−wt)dk−0.5k_t = \tilde{k}_t (1 - w_t) d_k^{-0.5}

where $d_k = D/h$ is the per-head key dimension.

What it computes: the key is the key precursor scaled by two factors: a decay complement $(1 - w_t)$ and a dimensionality normalization $d_k^{-0.5}$. The $(1 - w_t)$ factor means that when the decay $w_t$ is close to 1 (slow forgetting), the key contribution to the state is small — the model is choosing to retain past information rather than write new information. When $w_t$ is small (fast forgetting), the key contribution is large — the model is overwriting the state with new content. The $d_k^{-0.5}$ factor follows the standard attention scaling convention and prevents the dot products in the state update from growing with dimensionality.

Why this form: this is one of the key modifications from standard RWKV-6. In original RWKV-6, the key scaling uses an "off-by-one" bonus term that complicates the recurrent formulation and, the authors found, interferes with the conversion process by making it harder for the student's time-mixing layer to match the teacher's attention outputs during step 1. The $(1 - w_t)d_k^{-0.5}$ formulation is simpler — it couples the write strength directly to the forget strength — and the authors report that "removing the off-by-one decay and bonus by using a Gated Linear Attention kernel allowed the model to fit the original softmax attention hidden states much more closely during step 1." This is a concrete example of how architectural choices motivated by pretraining (the bonus may help with training stability from scratch) can be suboptimal for conversion (where the objective is to match an existing computation, not learn it from random initialization).

Recurrent state update (wkv computation). The core of RAD-RWKV6 is the recurrent state $\mathbf{wkv}_t$, which accumulates key-value pairs with exponential decay. The recurrent formulation is:

wkv0=0\mathbf{wkv}_0 = \mathbf{0}

wkvt=diag(wt)⋅wkvt−1+ktT⋅vt\mathbf{wkv}_t = \mathrm{diag}(w_t) \cdot \mathbf{wkv}_{t-1} + k_t^{\mathrm{T}} \cdot v_t

where $k_t, v_t \in \mathbb{R}^{D/h}$ (after splitting into $h$ heads), $w_t \in \mathbb{R}^{D/h}$ is the per-channel decay vector, and $\mathbf{wkv}_t \in \mathbb{R}^{(D/h) \times (D/h)}$ is a matrix-valued state. The operation $\mathrm{diag}(w_t) \cdot \mathbf{wkv}_{t-1}$ multiplies each row of the state matrix by the corresponding decay factor (element-wise across channels), and the outer product $k_t^{\mathrm{T}} \cdot v_t \in \mathbb{R}^{(D/h) \times (D/h)}$ adds new key-value associations.

What it computes: at each timestep, the state is (a) decayed channel-wise by $w_t$ — old information fades at a per-channel, data-dependent rate — and (b) updated by adding the outer product of the current key and value. Because $\mathbf{wkv}_t$ is a matrix of size $(D/h) \times (D/h)$, it has $(D/h)^2$ elements per head, or $D^2/h$ total state elements. For a typical configuration (e.g., $D = 4096$, $h = 32$, so $D/h = 128$), the state has $128^2 = 16,384$ elements per head, or 524,288 elements across 32 heads — a fixed size regardless of sequence length, giving O(1) per-token computation.

Why this form: the outer-product state is what distinguishes RWKV from simpler linear attention formulations that use a vector-valued state (key-value cross-covariance). The matrix state allows the model to maintain independent associations between every pair of key and value dimensions, giving it greater expressive capacity than a vector state of the same size. The parallel scan formulation (Equation 10 in the paper) shows the equivalent non-recurrent view:

wkvt=∑i=1tdiag(∏j=it−1wj)kiTvi\mathbf{wkv}_t = \sum_{i=1}^{t} \mathrm{diag}\left(\prod_{j=i}^{t-1} w_j\right) k_i^{\mathrm{T}} v_i

This is a causally masked sum of all past key-value outer products, with each term decayed by the cumulative product of decay factors between its insertion time $i$ and the current time $t$. This directly generalizes linear attention: standard linear attention corresponds to the special case $w_t = \mathbf{1}$ for all $t$ (no decay), and the data-dependent decay is what allows the model to learn when to forget vs. retain.

Attention output, gate, and final projection. The recurrent state is queried by the receptance:

pt=LayerNorm(rtwkvt)p_t = \mathrm{LayerNorm}(r_t \mathbf{wkv}_t)

where $r_t \in \mathbb{R}^{D/h}$ (the receptance vector, analogous to a query) multiplies the state matrix from the left, producing a vector in $\mathbb{R}^{D/h}$ that represents the "read" from memory. LayerNorm is then applied for training stability. The gate $g_t$ (sigmoid-activated) modulates this output element-wise:

ot=(gt⊙pt)Woo_t = (g_t \odot p_t) \mathbf{W}_o

where $p_t$ has been reshaped from per-head to full dimension $\mathbb{R}^D$, $\odot$ is element-wise multiplication, and $\mathbf{W}_o \in \mathbb{R}^{D \times D}$ is the output projection (which can be initialized from the teacher's attention output projection).

What it computes: the receptance $r_t$ performs a learned linear read from the state matrix, producing a weighted combination of stored values based on how well the current query matches stored keys (via the dot-product implicit in $r_t \mathbf{wkv}_t = \sum_i (r_t \cdot k_i) v_i$ after the decay factors). The gate $g_t$ allows the model to learn which channels of the attention output to pass through vs. suppress — this is the "gated linear attention" mechanism that has been shown to improve training stability and model quality (Yang et al., 2024). LayerNorm on $p_t$ before gating prevents the attention output norm from growing or shrinking across layers.

Why this form: the gate $g_t$ is one of the components that the authors found "extremely beneficial only at full rank in RAD-RWKV6, but performed perfectly well in RAD-RWKV7 with reduced rank" (Section 3). This architecture-specific sensitivity to rank is an example of why careful per-architecture testing matters: a component that works well at full rank in one design may or may not transfer its benefits under rank reduction in another design, and the conversion process amplifies these differences because the model starts from a pretrained state rather than a random initialization.

Multi-head treatment and GQA. The student model retains the same number of heads and the same Grouped Query Attention (GQA) structure as the teacher. For Qwen2.5 models, this means Q and K have fewer heads than V in some configurations, with the heads repeated to match. The paper states (Section 4): "Our student models retain the runtime repetition of keys and values from Grouped Query Attention (GQA) in the teacher model, if present." This ensures structural compatibility between teacher and student — the dimensionality of the state, the number of heads, and the head sizes all match exactly.

RAD-RWKV7 Time Mixing ("RADGoose")

RAD-RWKV7 is a customized variation of RWKV-7 (Peng et al., 2025) with three structural modifications for conversion: tokenshift is removed entirely (speeding up training and inference), RoPE is applied to keys and values (inherited from the teacher model's positional encoding scheme), and the bonus term is removed (no beneficial impact on downstream performance was observed). The architecture introduces a fundamentally different state update mechanism from RAD-RWKV6: rather than a simple decayed outer-product accumulation, RAD-RWKV7 uses a learned delta rule with removal and replacement keys, enabling the model to selectively erase and overwrite stored associations — a more expressive form of recurrence that the authors found "fits even closer and more rapidly" during step 1 alignment.

Linear interpolation helper. RAD-RWKV7 defines a simpler interpolation than RAD-RWKV6's ddlerp:

lerp(a,b,x)=a+(b−a)⊙x\mathrm{lerp}(a, b, x) = a + (b - a) \odot x

where $x \in \mathbb{R}^D$ is a learned (or computed) interpolation coefficient vector and $\odot$ is element-wise multiplication. This is a standard linear blend, used in several places where content-dependent mixing is needed.

Low-rank MLP helper. RAD-RWKV7 uses a different low-rank parameterization than RAD-RWKV6's ddlerp:

loramlp□(f,x,bias)=f(xA□)B□+(λ□ if bias else 0)\mathrm{loramlp}_{\square}(f, x, \text{bias}) = f(x \mathbf{A}_{\square}) \mathbf{B}_{\square} + (\lambda_{\square} \text{ if bias else } 0)

where $f$ is an activation function (usually $\mathrm{Identity}$, $\tanh$, or $\sigma$), $\mathbf{A}_{\square} \in \mathbb{R}^{D \times z}$ and $\mathbf{B}_{\square} \in \mathbb{R}^{z \times D}$ are low-rank trainable matrices, and $\lambda_{\square} \in \mathbb{R}^D$ is an optional trainable bias vector added after the $\mathbf{B}_{\square}$ projection.

What it computes: this is a bottleneck MLP: the input is projected down to $z$ dimensions by $\mathbf{A}_{\square}$, passed through activation $f$, projected back up to $D$ dimensions by $\mathbf{B}_{\square}$, and optionally summed with a bias. When $f = \mathrm{Identity}$, this is a simple low-rank linear transformation; when $f = \tanh$ or $\sigma$, it introduces nonlinearity within the bottleneck.

Why this form: compared to RAD-RWKV6's ddlerp (which applies $\tanh$ to the down-projected representation and then uses the result as interpolation coefficients between two vectors), this formulation separates the low-rank transformation from the interpolation operation. The activation function $f$ is explicit and varies by use case — Identity for the in-context learning rate, $\tanh$ for the decay precursor, $\sigma$ for the gate. This gives more flexibility, at the cost of not directly incorporating the token-shift interpolation within the low-rank computation (since RAD-RWKV7 has removed tokenshift entirely).

In-context learning rate and removal/replacement keys. RAD-RWKV7's key architectural innovation over RAD-RWKV6 is the delta-rule state update, which requires three key-like vectors — a replacement key, a removal key, and an in-context learning rate — that together control how new associations overwrite old ones. These are computed as:

at=sigmoid(loramlpa(Identity,xt,bias=True))a_t = \mathrm{sigmoid}(\mathrm{loramlp}_a(\mathrm{Identity}, x_t, \text{bias=True}))

κt=kt⊙ξ\kappa_t = k_t \odot \xi

k~t=kt⊙lerp(1,at,α)\tilde{k}_t = k_t \odot \mathrm{lerp}(1, a_t, \alpha)

where:

  • $a_t \in \mathbb{R}^{D/h}$ is the in-context learning rate, a per-channel sigmoid-activated value in $(0, 1)$ that controls how much of an existing association is removed when a new key-value pair is written.
  • $k_t = \mathrm{RoPE}(x_t \mathbf{W}_k) \in \mathbb{R}^{D/h}$ is the base key, computed by a linear projection followed by rotary positional embedding.
  • $\xi \in \mathbb{R}^{D/h}$ is a trainable vector used to compute the removal key $\kappa_t$. The notation $\odot$ denotes element-wise multiplication, so $\kappa_t$ is the base key modulated by the learned vector $\xi$.
  • $\alpha \in \mathbb{R}^{D/h}$ is a trainable vector used in the interpolation between all-ones and the learning rate $a_t$. $\mathrm{lerp}(1, a_t, \alpha)$ produces a per-channel value between $\alpha$ (when $a_t = 0$) and 1 (when $a_t = 1$), and this interpolant is element-wise multiplied with $k_t$ to produce the replacement key $\tilde{k}_t$.
  • The removal key $\kappa_t$ is additionally normalized per head: $\hat{\kappa}_t = \kappa_t / \|\kappa_t\|_2$.

What it computes: the removal key $\kappa_t$ and replacement key $\tilde{k}_t$ are two variants of the same key that serve opposite roles in the state update. When the model writes a new value $v_t$ to the state, the removal key $\hat{\kappa}_t$ (used via its outer product with a scaled version of itself) tells the state which existing associations to erase, while the replacement key $\tilde{k}_t$ (used in the standard outer product with $v_t$) tells the state which new associations to add. The in-context learning rate $a_t$ controls the relative strength of removal vs. replacement — when $a_t$ is large, $\tilde{k}_t \approx k_t$ (full replacement), and the removal term is strong; when $a_t$ is small, $\tilde{k}_t \approx k_t \odot \alpha$ (attenuated replacement), and the removal term is weak.

Why this form: this delta-rule mechanism gives RAD-RWKV7 the ability to perform selective forgetting — rather than uniformly decaying the entire state as in RAD-RWKV6 ($\mathrm{diag}(w_t) \cdot \mathbf{wkv}_{t-1}$), the removal term $\hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)$ subtracts a rank-1 matrix from the state that is specifically targeted at the associations matching the current removal key. This is the key architectural difference that the authors credit for RAD-RWKV7's stronger performance during step 1 alignment: "RWKV-7 fits even closer and more rapidly during this step, resulting in much lower distillation loss with even less compute." The expressiveness of selective erasure — being able to say "forget what you know about concept X" rather than just "forget everything a little bit" — likely makes it easier for the recurrent block to approximate the attention patterns of the teacher, which also performs selective attention (softmax creates sharp focus on relevant keys, implicitly ignoring irrelevant ones).

Value computation with residual gating. The value $v_t$ in RAD-RWKV7 is also more sophisticated than RAD-RWKV6's single ddlerp-projected value:

νt=sigmoid(loramlpν(Identity,xt,bias=True))\nu_t = \mathrm{sigmoid}(\mathrm{loramlp}_{\nu}(\mathrm{Identity}, x_t, \text{bias=True}))

vt,l′=RoPE(xtWv)v'_{t,l} = \mathrm{RoPE}(x_t \mathbf{W}_v)

vt={vt,0′,layer l=0lerp(vt,0′,vt,l′,νt),layer l≥1v_t = \begin{cases} v'_{t,0}, & \text{layer } l = 0 \\ \mathrm{lerp}(v'_{t,0}, v'_{t,l}, \nu_t), & \text{layer } l \geq 1 \end{cases}

What it computes: each layer has two value projections — $\mathbf{W}_v$ produces $v'_{t,l}$, a layer-specific value, but there is also a "layer 0" value $v'_{t,0}$. The sigmoid gate $\nu_t$ interpolates between the global (layer 0) value and the local (current layer) value. In the first layer, only the local value is used. In subsequent layers, the effective value is a learned blend between the layer-specific and the global value representation, controlled per-token by $\nu_t$.

Why this form: this creates a value residual pathway across layers — the layer-0 value can carry information that is useful across multiple layers, while each layer can also contribute its own specialized value representation. The gate $\nu_t$ lets the model learn when to rely on shared vs. layer-specific value information. The RoPE application to values (and keys) is inherited from the teacher model — since the Qwen2.5 teacher uses RoPE on Q and K, applying RoPE to the student's K and V preserves positional information in the same representation space, which the authors found beneficial for conversion quality.

Decay computation. The decay in RAD-RWKV7 uses a different parameterization than RAD-RWKV6:

dt=loramlpd(tanh⁡,xt,bias=True)d_t = \mathrm{loramlp}_d(\tanh, x_t, \text{bias=True})

wt=exp⁡(−e−0.5σ(dt))w_t = \exp(-e^{-0.5} \sigma(d_t))

What it computes: the decay precursor $d_t$ is produced by a low-rank MLP with $\tanh$ activation. It is then passed through a sigmoid $\sigma(\cdot)$ to produce a value in $(0, 1)$, scaled by the constant $e^{-0.5} \approx 0.6065$, and exponentiated with a negative sign: $\exp(-0.6065 \cdot \sigma(d_t))$. This produces a decay factor $w_t$ in $(e^{-0.6065}, e^{0}] \approx (0.545, 1.0]$ — a much narrower and higher range than RAD-RWKV6's $(e^{-5}, 1]$.

Why this form: the higher minimum decay (0.545 vs. 0.007) means RAD-RWKV7 forgets more slowly by default, which makes sense given its additional removal mechanism — rather than relying on decay to erase old information, RAD-RWKV7 uses the explicit removal key to selectively delete, so the base decay can be more conservative (retaining information longer). The constant $e^{-0.5}$ is inherited from the RWKV-7 design (Peng et al., 2025) and serves to center the pre-exponentiation range; the authors retained it because removing the bonus term was the only modification needed — this decay parameterization already worked well for conversion.

Receptance and gate. These are simpler than in RAD-RWKV6:

rt=xtWrr_t = x_t \mathbf{W}_r

gt=loramlpg(σ,xt,bias=False)g_t = \mathrm{loramlp}_g(\sigma, x_t, \text{bias=False})

The receptance uses a direct linear projection of $x_t$ without tokenshift (since tokenshift is removed entirely in RAD-RWKV7). The gate uses a low-rank MLP with sigmoid activation and no bias term, producing per-channel gating values in $(0, 1)$.

State update (delta-rule wkv computation). This is the core difference from RAD-RWKV6. The recurrent formulation in RAD-RWKV7 is:

wkv0=0\mathbf{wkv}_0 = \mathbf{0}

wkvt=wkvt−1(diag(wt)−κ^tT(at⊙κ^t))+vtT⋅k~t\mathbf{wkv}_t = \mathbf{wkv}_{t-1} \left(\mathrm{diag}(w_t) - \hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)\right) + v_t^T \cdot \tilde{k}_t

What it computes: at each timestep, the state undergoes two transformations: (1) decay: each row (channel) of $\mathbf{wkv}_{t-1}$ is multiplied by the corresponding decay factor $w_t$ via $\mathrm{diag}(w_t)$, exactly as in RAD-RWKV6; (2) removal: the rank-1 matrix $\hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)$ is subtracted from the diagonal decay matrix. The full term $(\mathrm{diag}(w_t) - \hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t))$ is an operator that first applies per-channel decay, then selectively removes associations matching $\hat{\kappa}_t$ with strength proportional to $a_t$. After the state is transformed, the new key-value pair is added as $v_t^T \cdot \tilde{k}_t$ — note the transposition convention: $v_t^T \in \mathbb{R}^{(D/h) \times 1}$ and $\tilde{k}_t \in \mathbb{R}^{1 \times (D/h)}$, so their product is $\mathbb{R}^{(D/h) \times (D/h)}$, matching the state dimensions.

Why this form is different from RAD-RWKV6: in RAD-RWKV6, the state update is $\mathbf{wkv}_t = \mathrm{diag}(w_t) \mathbf{wkv}_{t-1} + k_t^T v_t$. The key difference is the subtraction of $\hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)$. This term is the outer product of the normalized removal key with itself, scaled element-wise by the in-context learning rate. Because it's an outer product of a vector with itself, the resulting matrix is symmetric and rank-1, with the $(i, j)$ entry equal to $\hat{\kappa}_{t,i} \cdot a_{t,j} \cdot \hat{\kappa}_{t,j}$. When this is subtracted from the state, it erases the component of the state that aligns with $\hat{\kappa}_t$ — analogous to how the delta rule (Widrow-Hoff) updates weights by subtracting $\text{learning\_rate} \times \text{input} \times \text{input}^T$ times the current weights. The key insight is that $\hat{\kappa}_t$ (the normalized removal key) determines what to forget, while $a_t$ determines how much to forget, and $\tilde{k}_t$ (the replacement key) determines what to write in its place.

Attention output, gate, and final projection. After the state update, the readout is similar to RAD-RWKV6 but operates on the transpose of the state:

pt=LayerNorm(rtwkvtT)p_t = \mathrm{LayerNorm}(r_t \mathbf{wkv}_t^T)

ot=(gt⊙pt)Woo_t = (g_t \odot p_t) \mathbf{W}_o

What it computes: the receptance queries the transposed state $\mathbf{wkv}_t^T$. Since $\mathbf{wkv}_t$ was updated as $\mathbf{wkv}_{t-1} (\ldots) + v_t^T \tilde{k}_t$, transposing gives $\mathbf{wkv}_t^T = (\ldots)^T \mathbf{wkv}_{t-1}^T + \tilde{k}_t^T v_t$. This transpose convention is a notational detail from the RWKV-7 formulation — it flips which dimension the receptance interacts with compared to RAD-RWKV6, but the functional computation (receptance reads from the state, state is updated by key-value pairs) remains the same. After LayerNorm, the gate $g_t$ modulates the output and $\mathbf{W}_o$ projects back to the model dimension.

Why this architecture fits conversion better: the authors report (Section 3) that RAD-RWKV7 "fits even closer and more rapidly during step 1 than RAD-RWKV6." The likely mechanism is the delta-rule state update's greater expressiveness: the ability to selectively remove old associations (via $\hat{\kappa}_t$) makes it easier for the recurrent block to approximate the sharp, query-dependent attention patterns of softmax attention. In softmax attention, the model computes $\mathrm{softmax}(QK^T/\sqrt{d_k})V$, which effectively zeroes out the influence of tokens with low attention scores. A standard decay-based RNN can only approximate this by letting those tokens decay away — a gradual process. The delta rule can actively delete them through the removal mechanism, producing a closer match to the sharp cutoff of softmax attention in fewer training steps.

Tokenshift removal. RAD-RWKV7 removes the tokenshift mechanism that was present in RAD-RWKV6. The authors state that tokenshift "conferred essentially no benefit in RAD-RWKV7" (Section 3). This makes architectural sense: tokenshift provides a form of local temporal convolution (mixing $x_t$ and $x_{t-1}$ with learned coefficients), but RAD-RWKV7's delta-rule mechanism already captures short-term dependencies through the explicit write-and-erase operations — the model can decide at each step which information from the immediate past to retain or discard, making an additional fixed convolutional mixing redundant.

RoPE application. RAD-RWKV7 applies rotary positional embeddings to keys and values: $k_t = \mathrm{RoPE}(x_t \mathbf{W}_k)$ and $v'_{t,l} = \mathrm{RoPE}(x_t \mathbf{W}_v)$. The paper states (Section 4) that "in cases where we include RoPE, it is applied at exactly the same point as in the teacher model." For Qwen2.5 teachers, RoPE is applied to Q and K in the attention mechanism; RAD-RWKV7 mirrors this by applying RoPE to the key (and optionally the value) before they enter the recurrent state. This preserves the positional information structure that the teacher's MLPs and embeddings were trained to expect, reducing the distribution shift that the rest of the model must adapt to during conversion.

Setup: Attention Weights Transfer

Before any training begins, the student model must be initialized. The Setup step (Section 4.1) defines the rules for initializing each parameter of the new time-mixing blocks from the teacher's attention weights.

Direct transfer of analogous weights. The teacher's softmax attention computes:

Attention(x)=softmax((xWq)(xWk)Tdk)(xWv)Wo\text{Attention}(x) = \mathrm{softmax}\left(\frac{(x\mathbf{W}_q)(x\mathbf{W}_k)^T}{\sqrt{d_k}}\right)(x\mathbf{W}_v)\mathbf{W}_o

The student's RAD-RWKV6/7 blocks compute analogous quantities — receptance $r_t$ (query-like), key $k_t$, value $v_t$, output projection $\mathbf{W}_o$ — that serve similar roles of reading from, writing to, and projecting from the sequence-mixing state. When equivalent parameters exist, they are initialized directly from the teacher:

  • $\mathbf{W}_r$ (student receptance) ← $\mathbf{W}_q$ (teacher query)
  • $\mathbf{W}_k$ or $\mathbf{W}_{\tilde{k}}$ (student key) ← $\mathbf{W}_k$ (teacher key)
  • $\mathbf{W}_v$ (student value) ← $\mathbf{W}_v$ (teacher value)
  • $\mathbf{W}_o$ (student output) ← $\mathbf{W}_o$ (teacher output)

What this accomplishes: the student's time-mixing block starts with the same linear projections the teacher used to compute queries, keys, values, and output. Even though the subsequent computation is entirely different (recurrent state update vs. softmax attention), the initial representation space for these vectors matches the teacher's. This means that, at initialization, $k_t$, $v_t$, and $r_t$ live in the same subspaces the teacher learned during pretraining — the student's job during step 1 is to learn how to combine them recurrently rather than through attention, starting from representations that are already semantically meaningful.

Initialization of new parameters. Parameters with no analog in the teacher — the low-rank ddlerp/loramlp matrices ($\mathbf{A}_{\square}, \mathbf{B}_{\square}$), the shared token-shift vector $\mu_x$ (RAD-RWKV6), the in-context learning rate components $\xi, \alpha$ (RAD-RWKV7), and all bias vectors $\lambda_{\square}$ — are initialized following standard pretraining initialization schemes. For parameters like tokenshift (in RAD-RWKV6, since removed in RAD-RWKV7), the initialization is chosen "such that they mimic the teacher model and have no immediate effect" (Section 4.1).

What "no immediate effect" means operationally: the tokenshift mechanism interpolates between $x_t$ and $x_{t-1}$. To have no immediate effect, the interpolation coefficient should be 0 (or close to it), so that $\mathrm{ddlerp}(x_t, x_{t-1}) \approx x_t$ — the block initially ignores the previous token, making its behavior closer to a feedforward transformation of the current input, which is easier to align with the teacher's per-token attention output. The ddlerp can then learn to incorporate temporal context gradually during training, as needed.

Why weight transfer matters: the ablation in Section 8 (negative results) reports that "initializing sequence mixer QKVO weights as if they were untrained, instead of copying them from the teacher model, resulted in consistently worse yet surprisingly reasonable performance. This is compatible with the previous hypothesis stating that factual knowledge is stored mainly in MLP model weights." The fact that performance is "surprisingly reasonable" even without weight transfer suggests that step 1 alignment can recover from random initialization given enough training — but the efficiency gain from weight transfer is substantial, since the student doesn't need to relearn the basic geometry of the key, query, and value spaces.

GQA head repetition. The paper explicitly states (Section 4): "Our student models retain the runtime repetition of keys and values from Grouped Query Attention (GQA) in the teacher model, if present." In GQA, multiple query heads share the same key-value head — for instance, Qwen2.5-7B has 28 query heads and 4 key-value heads, meaning each KV head is shared across 7 query heads. The student model preserves this structure: the time-mixing block's key, value, and decay parameters have the same number of heads as the teacher's KV heads, and the receptance and output projection have the same number of heads as the teacher's Q heads. This structural fidelity ensures that the weight transfer is one-to-one where applicable.

Step 1: Attention Hidden-State Alignment

Step 1 (Section 4.2) is the first and most architecturally specific training phase. Its goal is to train the student's newly inserted recurrent time-mixing blocks so that, for each transformer layer, the block produces approximately the same output hidden state that the teacher's original softmax-attention block would have produced on the same input.

Training setup. A frozen copy of the complete teacher model is held in memory. For each layer $l$, the teacher's original attention block and the student's new RAD-RWKV block both receive the same input hidden states $x_t^{(l)}$ (the output of the previous layer or the embedding layer). The teacher attention block produces output $h_t^{(l,\text{teacher})}$; the student time-mixing block produces output $o_t^{(l,\text{student})}$. The loss is the average over layers and tokens of the L2 distance between these outputs:

Lstep1=1L⋅T∑l=1L∑t=1T∥ht(l,teacher)−ot(l,student)∥22\mathcal{L}_{\text{step1}} = \frac{1}{L \cdot T} \sum_{l=1}^{L} \sum_{t=1}^{T} \| h_t^{(l,\text{teacher})} - o_t^{(l,\text{student})} \|_2^2

where $L$ is the number of transformer layers, $T$ is the sequence length, and $\|\cdot\|_2^2$ is the squared Euclidean norm (the paper notes "L2 distance (or optionally, mean squared error)" — these are equivalent up to a constant factor of $1/D$).

What it computes: for each token position in each layer, the student's time-mixing output is compared to the teacher's attention output, and the squared difference is averaged. Since only the student's time-mixing parameters are trainable (all shared components — MLPs, layer norms, embeddings — are frozen, as is the entire teacher), the optimization directly targets the discrepancy introduced by replacing softmax attention with recurrence. The loss penalizes any deviation, encouraging the recurrent block to reproduce exactly the hidden-state transformation the teacher's attention would have performed.

Why L2 loss rather than, e.g., cosine similarity or KL divergence: the L2 loss is appropriate here because the teacher's attention outputs are continuous vectors in $\mathbb{R}^D$, not probability distributions. The goal is to make the student's per-layer outputs numerically close to the teacher's, so that when these outputs are fed into the subsequent MLP and layer norm (which are initialized from the teacher's weights and initially frozen), they produce the same downstream effects. Cosine similarity would ignore magnitude differences, which matter for the MLP's nonlinearities; KL divergence requires a probability interpretation that doesn't apply to hidden states.

Training configuration (Table 7, Step 1 row). The paper provides exact hyperparameters:

  • Tokens: 100M (total across all sequences)
  • Learning rate: $1 \times 10^{-3} \to 1 \times 10^{-5}$, cosine annealed
  • Sequence length: 512 tokens
  • Batch size: 32 (number of sequences per optimizer step)
  • Optimizer: AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 1 \times 10^{-8}$
  • Dataset: DCLM (Li et al., 2024b)

Why cosine annealing from 1e-3 to 1e-5: the paper's stated rationale (Section 3) is that "most knowledge resides in the teacher model's MLPs and embeddings," so the attention-replacement layers need to adapt quickly (hence the high initial learning rate of 1e-3 — two orders of magnitude above the teacher's final pretraining LR), while the final learning rate of 1e-5 matches "the final learning rate the teacher saw during pretraining." The cosine schedule provides a smooth transition between these regimes: early in step 1, the time-mixing blocks learn rapidly to approximate the teacher's attention outputs; later in step 1, the learning rate drops to a level that won't disturb the pretrained representations too much, preparing the model for step 2 where the full model will be trained together.

Why 100M tokens is sufficient: the authors find that "100M tokens is enough to converge to a low stable loss during step 1" (Section 4.2). The convergence speed depends on the quality of the target architecture: RAD-RWKV7 converges faster than RAD-RWKV6, meaning it reaches a lower L2 loss with the same token budget, or equivalently reaches the same loss with fewer tokens. This is consistent with the hypothesis that the delta-rule mechanism makes it easier to fit the teacher's attention patterns.

Alternative per-layer training. The paper notes that step 1 "can be done sequentially layer by layer, or in parallel with all layers at once." In practice, the authors train all layers simultaneously with a single optimizer and a single forward/backward pass. Sequential training — training layer 1, freezing it, then training layer 2, etc. — would allow each layer to be aligned to a "perfect" input distribution (since lower layers would already match the teacher), but is computationally much slower (requiring $L$ separate training runs). The parallel approach allows all layers to be trained in one pass, accepting that upper layers see somewhat noisy inputs from lower layers that haven't fully converged yet.

What happens at the end of step 1. After the 100M tokens are processed, the teacher attention layers are physically removed from the model — the student model now consists of RAD-RWKV blocks (trained to approximate attention outputs), MLPs (still frozen with teacher weights), embeddings (still frozen), and layer norms (still frozen). The output of step 1 is a complete, standalone linear-attention model whose per-layer hidden states approximately match the teacher's, but whose end-to-end next-token prediction may still be suboptimal because the MLPs and embeddings haven't been fine-tuned to work with the slightly different hidden-state distributions the recurrent blocks produce.

Step 2: Knowledge Distillation (Model-Wise Approximation of Logits)

Step 2 (Section 4.3) trains the complete student model end-to-end to match the teacher's output distribution over the vocabulary. While step 1 aligned per-layer hidden states, step 2 addresses the accumulated discrepancies across layers and adapts the MLPs and embeddings to the new time-mixing blocks.

Training setup. A separate frozen copy of the full teacher model is loaded (separate from the step 1 copy, though in practice this can be the same weights reloaded). Both teacher and student process the same input sequences independently. The loss is the Kullback-Leibler divergence between the student's output logits and the teacher's output logits:

Lstep2=1T∑t=1TDKL(pt(teacher)∥pt(student))=1T∑t=1T∑v∈Vpt(teacher)(v)log⁡pt(teacher)(v)pt(student)(v)\mathcal{L}_{\text{step2}} = \frac{1}{T} \sum_{t=1}^{T} D_{\mathrm{KL}}(p_t^{\text{(teacher)}} \| p_t^{\text{(student)}}) = \frac{1}{T} \sum_{t=1}^{T} \sum_{v \in V} p_t^{\text{(teacher)}}(v) \log \frac{p_t^{\text{(teacher)}}(v)}{p_t^{\text{(student)}}(v)}

where $V$ is the vocabulary, and $p_t^{\text{(teacher)}}(v) = \mathrm{softmax}(\text{logits}_t^{\text{(teacher)}})_v$ is the teacher's predicted probability for token $v$ at position $t$, and similarly for the student. All student parameters — time-mixing blocks, MLPs, embeddings, output head, layer norms — are trainable.

What it computes: at each token position, the teacher and student each produce a probability distribution over the vocabulary (via softmax of their logits). KL divergence measures how much additional information would be needed to encode samples from the teacher distribution using the student distribution — it is minimized (to 0) when the distributions are identical. By minimizing the average KL across all positions, the student learns to replicate the teacher's next-token prediction behavior, which implicitly captures the teacher's full conditional language model: $P_{\text{teacher}}(\text{next\_token} | \text{context})$.

Why KL divergence rather than cross-entropy with the teacher's argmax token: soft labels (the full distribution) carry more information than hard labels (the single highest-probability token). The teacher may assign, say, 80% probability to the correct token, 15% to a plausible alternative, and 5% distributed across other tokens. Training on the hard label ("token A is correct") discards the information that "token B is almost correct." Training on the full distribution via KL divergence preserves this secondary information, which helps the student learn the teacher's uncertainty calibration and prevents overfitting to the teacher's occasional errors. This is the standard knowledge distillation approach (Hinton et al., 2015) applied to language model logits rather than classification logits.

Why not continue with L2 loss on hidden states: the step 1 L2 loss operates in the hidden-state space, aligning per-layer representations. But small per-layer errors compound across 28–80 layers: if each layer's output is off by $\epsilon$, the final-layer hidden state (and thus the logits) could be off by $O(L\epsilon)$. Step 2 directly optimizes the end-to-end objective — matching output distributions — which can compensate for accumulated per-layer discrepancies by adjusting the MLPs and later layers to work with the specific error patterns the recurrent blocks produce.

Training configuration (Table 7, Step 2 row).

  • Tokens: 250M–700M (250M is "generally sufficient"; the paper uses 500M for most models)
  • Learning rate: $1 \times 10^{-5}$, flat (constant throughout the step)
  • Sequence length: 512 tokens
  • Batch size: 96
  • Optimizer: AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 1 \times 10^{-8}$
  • Dataset: DCLM

Why a flat learning rate of 1e-5: the paper states (Section 3) that the learning rate during steps 2 and 3 is kept fixed and low — "similar to the final learning rate the teacher saw during pretraining" — "to avoid catastrophic forgetting or overly significantly changing the knowledge encoded in the original teacher MLPs." The constant rate means the model takes small, steady steps toward matching the teacher's logits without the aggressive early exploration of a decaying schedule. The specific value 1e-5 is one to two orders of magnitude lower than typical pretraining peak learning rates, consistent with the philosophy that the MLPs already contain most of the factual knowledge and need only minor adaptation.

Why 250M–700M tokens, and why more isn't always better: the paper explicitly states that "training more than around 500M tokens during step 2 does not improve English language benchmarks," though "it's possible that less frequently encountered tokens still improve beyond this point" (Section 4.3). The observation that benchmark performance saturates at ~500M tokens is consistent with the idea that the model's factual knowledge (in MLPs and embeddings) is already present and the time-mixing mechanism is already well-aligned from step 1 — step 2 is mainly integrating these components, which requires relatively little data. The difference between 250M and 500M tokens matters more for RAD-RWKV6 than RAD-RWKV7: since RAD-RWKV7 "reaches better alignment during step 1 than RAD-RWKV6," it "can therefore tolerate shorter step 2 training runs" (Section 4.3).

Relationship to step 1 quality: the better the step 1 alignment, the less work step 2 needs to do. If the time-mixing blocks produce outputs almost identical to the teacher's attention outputs, the MLPs and embeddings see essentially the same distribution as during pretraining, and step 2 just needs to polish the remaining small discrepancies. If step 1 alignment is poor, step 2 must compensate for larger distribution shifts, requiring more tokens or resulting in lower final quality. This is the mechanistic explanation for why RAD-RWKV7's stronger step 1 performance translates to better final downstream scores at the same step 2 token budget.

Step 3: Context-Length Extension

Step 3 (Section 4.4) adapts the model to function at longer sequence lengths than the 512 tokens used in steps 1 and 2. The motivation is that steps 1 and 2 require keeping both teacher and student models in memory, which limits the maximum feasible sequence length on affordable hardware. Step 3 discards the teacher entirely, freeing memory for longer sequences.

Training setup. The student model is trained with standard language modeling loss (cross-entropy with the ground-truth next token) on sequences of length 16,384 tokens — a 32× increase from steps 1–2. There is no teacher model in memory. All student parameters are trainable.

Lstep3=−1T∑t=1Tlog⁡pt(student)(yt)\mathcal{L}_{\text{step3}} = -\frac{1}{T} \sum_{t=1}^{T} \log p_t^{\text{(student)}}(y_t)

where $y_t$ is the ground-truth next token from the training corpus.

What it computes: standard autoregressive language modeling loss. The model predicts the next token at each position and is penalized for incorrect predictions. Unlike step 2 (which matched the teacher's distribution), step 3 trains directly on the data, which means the model can potentially recover from any teacher-specific biases or errors introduced during distillation, while simultaneously learning to handle the longer-range dependencies that 512-token contexts cannot capture.

Why standard cross-entropy rather than continued distillation: at 16K context length, running the teacher model alongside the student for step 2-style distillation would require holding two full models in memory for 32× longer sequences — approximately 64× the memory of step 1 with a single model (two models, 32× sequence length). For a 72B model, this far exceeds typical GPU memory. By switching to standard LM training, step 3 requires only one model (the student), making 16K sequences feasible. Additionally, the teacher model was pretrained with a certain context length (typically 4K–32K for Qwen2.5), and its behavior beyond its training context may be poorly defined — distillation at lengths the teacher wasn't trained for could propagate undefined behavior.

Training configuration (Table 7, Step 3 row).

  • Tokens: 100M (total)
  • Learning rate: $1 \times 10^{-5}$, flat
  • Sequence length: 16,384 tokens
  • Batch size: 96
  • Optimizer: AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 1 \times 10^{-8}$
  • Dataset: DCLM

Why 100M tokens for context extension: the context extension phase is lighter-weight than the distillation phase because the primary challenge is adapting the time-mixing decay and state-management mechanisms to longer sequences, not learning new knowledge. The recurrent state update in RAD-RWKV6/7 is governed by data-dependent decay factors $w_t$, and these factors were trained on 512-token sequences in step 1 and 2. At 16K tokens, the cumulative product $\prod_{j=i}^{t-1} w_j$ can decay to much smaller values than at 512 tokens, meaning the model may effectively "forget" information it should retain over long contexts. The 100M tokens of step 3 give the decay mechanisms enough signal to recalibrate for the longer horizon, without requiring the full distillation pipeline.

Low-VRAM alternative (step 3a). The paper provides an alternative for situations where 16K context lengths on full model weights exceed available memory:

"In this alternative step 3a, we freeze all weights except for decay and tokenshift (if present), and increase the learning rate to $1 \times 10^{-4}$."

What this does: by freezing all weights except the parameters that directly control temporal dynamics (decay parameters, and tokenshift if the architecture uses it), the memory footprint drops because only a small fraction of the model's parameters need optimizer states (Adam maintains first and second moment estimates for each trainable parameter). The learning rate increase to 1e-4 (10× the standard step 3 LR) compensates for the reduced parameter count and the fact that only a narrow set of temporal parameters needs to adapt. The standard step 3 is preferred when memory allows, since full-weight training can adapt the MLP representations to long-context patterns as well, but step 3a provides a practical fallback.

Why context extension is done as a separate step rather than by increasing sequence length in steps 1–2: the paper explains this as "for efficiency and memory reasons" (Section 4.4). Steps 1 and 2 require holding teacher model(s) in memory alongside the student and (for step 1) training only a subset of parameters. Increasing sequence length would multiply the memory cost of both the teacher and any intermediate activations. By separating context extension into step 3 — where only the student model is present and all parameters can be trained — the memory budget can be devoted to longer sequences rather than teacher model storage.

Dataset considerations for step 3. The paper notes in Section 8 (negative results) that using custom datasets during context-length extension "seemed to create a confused model that would use more and more adjectives as generation progressed." This is an interesting negative result: the choice of dataset for long-context training affects not just long-range coherence but also local generation quality, possibly because the model learns spurious correlations between sequence length and stylistic patterns in the training data. The default use of DCLM for all three steps avoids this issue.

Hyperparameter and Infrastructure Summary

Table 7 in the paper provides a consolidated reference for all training hyperparameters. Reproduced as a structured summary:

Step 1 — Attention Hidden-State Alignment:

  • Tokens: 100M
  • Learning rate schedule: cosine from $1 \times 10^{-3}$ to $1 \times 10^{-5}$
  • Batch size: 32 sequences
  • Sequence length: 512
  • Optimizer: AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$)
  • Dataset: DCLM

Step 2 — Knowledge Distillation:

  • Tokens: 250M–700M (500M used for main models)
  • Learning rate: flat $1 \times 10^{-5}$
  • Batch size: 96 sequences
  • Sequence length: 512
  • Optimizer: AdamW (same betas, epsilon)
  • Dataset: DCLM

Step 3 — Context-Length Extension:

  • Tokens: 100M
  • Learning rate: flat $1 \times 10^{-5}$
  • Batch size: 96 sequences
  • Sequence length: 16,384
  • Optimizer: AdamW (same betas, epsilon)
  • Dataset: DCLM

Alternative Step 3a (low VRAM):

  • All weights frozen except decay and tokenshift
  • Learning rate: flat $1 \times 10^{-4}$

Why the batch size increases from 32 to 96 between steps 1 and 2: step 1 has both teacher and student in memory, limiting per-GPU batch size. Step 2 also has both models, but uses DeepSpeed ZeRO stage 1 or FSDP to distribute memory, allowing larger effective batches. Larger batch sizes in step 2 improve gradient signal quality for the full-model distillation, since the KL divergence over a vocabulary of 150K+ tokens produces noisy per-token gradients that benefit from averaging over more tokens. The paper explicitly notes in Section 8 that "the number of optimizer steps appears to be of key importance during conversion," and that "increasing batchsize did not help the model converge faster" — meaning throughput (tokens/second) can be increased with larger batches, but the number of optimizer updates (steps) matters for convergence, not just total tokens.

Infrastructure and timing (Table 8). All experiments use 8× AMD Mi300X GPUs (192GB VRAM each at the time of the experiments). The choice between DeepSpeed ZeRO stages and FSDP is manual, based on VRAM availability:

  • 7B models: DeepSpeed ZeRO stage 1 for steps 1 and 2 (faster training when VRAM permits); step 3 also uses DS1. Step 1 takes ~0.75 hours, step 2 takes ~5.5 hours (500M tokens), step 3 takes ~1 hour. Total: ~7.25 GPU-hours on 8 GPUs.
  • 32B models: DeepSpeed ZeRO stage 1 for step 1 (~2.5 hours); FSDP for steps 2 and 3 (~27 hours and ~3 hours respectively). Total: ~32.5 GPU-hours.
  • 72B models: FSDP for all three steps (~7.5 hours, ~54 hours, ~6 hours). The paper notes that "16,384 ctxlen does not fit on a single node at 72B scale" for step 3, requiring multi-node or gradient accumulation tricks. Total: ~67.5 GPU-hours.

At cloud GPU pricing of roughly 2–3/GPU−hour,the72Bconversioncostisapproximately‘2–3/GPU-hour, the 72B conversion cost is approximately `67.5 × 3×8≈3 × 8 ≈ 1,620, consistent with the "< $2,000" claim. The 7B conversion costs roughly 7.25×7.25 × 3 × 8 ≈ $174`.

Ablation Studies and Negative Results as Design Validation

Section 6 presents controlled ablations on RAD-RWKV6 at 7B scale, each removing or adding a single architectural component and measuring the impact on downstream benchmarks. These results validate the architectural choices described above.

Full RAD-RWKV6 (baseline): LAMBADA 0.6748, MMLU 0.6572, ARC-C 0.5631, ARC-E 0.8136, HellaSwag 0.7901, PIQA 0.8025, Winogrande 0.7111.

Adding RoPE (RAD-RWKV6 RoPE): minimal change across all benchmarks (e.g., MMLU 0.6610 vs. 0.6572). This is notable because the authors' main RAD-RWKV6 models do not use RoPE (unlike RAD-RWKV7, which does). The ablation suggests that for RAD-RWKV6, the tokenshift mechanism provides sufficient positional information without RoPE, consistent with the design philosophy of removing unnecessary components.

Removing tokenshift: LAMBADA drops to 0.6707 (from 0.6748), Winogrande drops to 0.6875 (from 0.7111). The Winogrande degradation is the largest single-benchmark effect in the ablations, suggesting that tokenshift contributes meaningfully to coreference resolution and entity tracking — tasks central to Winogrande's pronoun-disambiguation format. This validates the inclusion of tokenshift in RAD-RWKV6.

Removing gate: LAMBADA drops to 0.6590, MMLU to 0.6417, ARC-C to 0.5444 — consistently lower across most benchmarks. This confirms that the sigmoid gate is important for model quality in RAD-RWKV6, consistent with the authors' observation that gating was "extremely beneficial only at full rank in RAD-RWKV6."

Adding GroupNorm (replacing state balancing): LAMBADA drops to 0.6559, MMLU to 0.6340 — the largest MMLU degradation among the ablations. This validates the choice of the $k_t = \tilde{k}_t(1 - w_t)d_k^{-0.5}$ state-balancing mechanism over GroupNorm, as the authors claimed it "improved downstream performance."

Section 8 catalogs approaches that were tried and rejected, providing context for why the final protocol takes its specific form:

  • Initial attention-score alignment ("step 0"): training to match attention-score matrices before hidden-state alignment did not improve convergence and, when extended, incurred higher final loss. This suggests that directly matching the teacher's attention scores is a harder optimization problem than matching hidden states — the score space is higher-dimensional ($T \times T$ vs. $D$), and small errors in score reproduction may not matter for downstream hidden-state quality.

  • Skipping step 1 entirely: starting directly with step 2 distillation resulted in "a much lower performance model" with loss "plateauing at a higher minimum, even with longer training." This validates the two-phase design: without step 1's hidden-state alignment, the end-to-end distillation signal is too weak to guide the randomly initialized time-mixing blocks to a good solution.

  • Freezing MLPs during step 2: "this results in significantly reduced model performance." The authors theorize that "the internal embedding and hidden state representations need to adapt somewhat to the new RNN sequence mixer's use of channels" and that "the MLPs may learn to route this information differently to avoid conflicts with pre-existing information flow from the teacher model."

  • Larger batch sizes: "the number of optimizer steps appears to be of key importance during conversion. Consequently, increasing batchsize did not help the model converge faster." Combined with the learning rate being kept low to avoid disturbing MLPs, this means there is no easy shortcut: more tokens don't help if they're processed in fewer optimizer steps.

  • LoRA training: "rank reduction was generally quite detrimental to performance. The one place we found it could work without causing problems was on the embeddings." This directly contrasts with LOLCats's LoRA-based approach and explains their lower quality — conversion requires full-rank adaptation of the time-mixing parameters to match the teacher's attention patterns.

  • Switching datasets during context extension: using custom datasets produced "a confused model that would use more and more adjectives as generation progressed." This is a caution about dataset consistency across conversion steps.

4. Key Insights and Innovations

Innovation 1: Conversion Quality Is Governed by Architectural Match, Not Just Training Budget

The dominant assumption across prior conversion work — from SUPRA's 100B-token training runs to LOLCats's 40M-token LoRA-based approach — has been that conversion is primarily a data problem: give the student model enough tokens with the right distillation objective, and it will recover the teacher's capabilities regardless of the target architecture. The field has treated the choice of recurrent architecture (Mamba vs. linear attention vs. RWKV) as an implementation detail — something you plug in and train around — rather than as a first-order determinant of conversion success.

RADLADS reframes conversion as an architectural compatibility problem. The paper's central diagnostic move is the observation, in Section 3, that "pre-existing RWKV architectural choices were sometimes imperfectly matched to the needs of conversion." This is not a statement about absolute architectural quality — RWKV-6 and RWKV-7 are strong architectures for pretraining — but about a specific kind of mismatch: components designed to aid training from random initialization (the off-by-one decay, the bonus term, tokenshift) can actively interfere with the goal of matching a pretrained softmax-attention computation at each layer.

The evidence for this reframing comes from a tight chain of observations. The authors found that removing the off-by-one decay and bonus from RWKV-6 "allowed the model to fit the original softmax attention hidden states much more closely during step 1" — the same architecture, stripped of pretraining-oriented features, becomes a better substrate for conversion. Similarly, tokenshift was "useful in RAD-RWKV6, but conferred essentially no benefit in RAD-RWKV7" — the same component has different conversion-relevance depending on what other mechanisms are present. And RAD-RWKV7, whose delta-rule state update is inherently more expressive (it can selectively erase associations rather than only decaying them), "fits even closer and more rapidly during step 1," reducing the burden on downstream distillation. These are not universal architectural truths — they are conversion-specific compatibility findings.

This insight matters beyond RADLADS because it changes the search problem for future conversion work. Rather than asking "which RNN architecture is best in general?" and then training it on lots of data, the right question is "which architectural components help or hinder the specific task of matching a frozen attention computation at each layer, starting from a pretrained initialization?" Concretely, this means future conversion efforts should ablate architectural components by measuring step 1 alignment loss — a cheap proxy for conversion quality — rather than by running full end-to-end distillation for every candidate. RADLADS's development of RAD-RWKV6 and RAD-RWKV7 demonstrates this methodology: start from a strong pretraining architecture, remove components that the ablation shows don't help step 1 convergence (bonus, tokenshift in RWKV-7), keep or add components that do (Gated Linear Attention kernel, learned delta rule), and verify with the final benchmark evaluation.

This is a fundamental reframing, not an incremental refinement. Prior work treated architecture as fixed and data volume as the variable to optimize; RADLADS treats architecture as the variable and data volume as modest and sufficient. The 350–700M token budget is not the achievement — it's a consequence of getting the architecture right.

Innovation 2: The Three-Step Decomposition as a Diagnostic Framework, Not Just a Recipe

Many prior conversion methods employ a subset of the RADLADS steps. MOHAWK and LOLCats use attention alignment followed by distillation (a two-phase approach). T2R and SUPRA use weight transfer followed by full-model finetuning. Mamba in the Llama uses progressive distillation with supervised finetuning. At a surface level, RADLADS's three-step protocol — hidden-state alignment, knowledge distillation, context extension — appears to be simply a well-tuned combination of known ingredients, arranged in a sensible order.

But the deeper contribution is that RADLADS's step decomposition serves as a diagnostic framework for understanding where conversion quality comes from, and where it breaks down. Each step's function is conceptually distinct, and — critically — the paper provides evidence (Section 8, negative results) for what happens when each step is omitted or altered, enabling a causal attribution of conversion difficulty to specific sub-problems.

Step 1 isolates the architecture-matching problem. The fact that "skipping step 1 entirely resulted in a much lower performance model" with loss "plateauing at a higher minimum, even with longer training" tells us something fundamental: end-to-end logit distillation is an insufficient signal for training the time-mixing layers from scratch, even when the MLPs and embeddings are already pretrained. The gradient signal through 28–80 layers of frozen MLPs and layer norms is too attenuated or too high-variance to efficiently guide randomly initialized recurrent blocks toward the teacher's attention computation. Step 1 exists not because alignment is a nice-to-have, but because the optimization landscape for end-to-end logit matching has local minima that hidden-state alignment can bypass. This is a diagnostic insight about the geometry of the distillation problem, not merely a recipe step.

Step 2 isolates the integration problem. The finding that freezing MLPs during step 2 "results in significantly reduced model performance" — and the authors' hypothesis that "MLPs may learn to route this information differently to avoid conflicts with pre-existing information flow" — reveals that the hidden-state alignment from step 1, even when converged to low L2 loss, produces subtle distribution shifts that compound across layers. The MLPs, which encode the bulk of factual knowledge, were trained to expect hidden states from softmax attention blocks. When those hidden states are produced by RAD-RWKV blocks instead — even well-aligned ones — certain channels or subspaces may carry the same information at slightly different magnitudes or in slightly different directions. Step 2 allows the MLPs to adapt to these shifts without forgetting their stored knowledge, enabled by the flat, low learning rate.

Step 3 isolates the temporal-horizon problem. By separating context extension from distillation, RADLADS makes visible a phenomenon that is easy to overlook: the time-mixing parameters trained at 512-token context may not generalize to 16K tokens, not because they are "wrong" in any absolute sense, but because the cumulative decay product $\prod w_j$ over 16K tokens can be many orders of magnitude smaller than over 512 tokens. The model trained at short context may have learned decay rates that are appropriate for a 512-token horizon but too aggressive for 16K tokens (causing premature forgetting of earlier information). Step 3 exists because the temporal dynamics of recurrent state management are context-length-dependent, and this dependence cannot be addressed during steps 1–2 due to memory constraints from the teacher model.

The negative result on initial attention-score alignment ("step 0") is particularly illuminating within this framework. The fact that matching attention-score matrices — the most direct possible alignment objective — not only fails to accelerate training but increases final loss when extended tells us that the optimization problem of score matching is harder than hidden-state matching, likely because the score space has dimension $T \times T$ (per-head, per-layer) versus the hidden-state dimension $D$. This is a non-obvious finding: one might intuitively expect that matching attention scores would provide a richer training signal than matching attention outputs, but the opposite holds in practice because the higher-dimensional objective introduces more noise and optimization difficulty per training token.

This decomposition is a conceptual advance that enables future work to target specific bottlenecks. If context extension is the limiting factor in a new conversion, step 3 can be improved independently (better long-context datasets, alternative training objectives) without redesigning steps 1–2. If architecture matching is the bottleneck, step 1 ablation (measuring alignment loss for different architectural variants) provides a fast feedback loop without requiring full distillation. This modularity is the intellectual legacy of the three-step framework, not the specific hyperparameter values in Table 7.

Innovation 3: Conversion Cost Can Be Decoupled from Teacher Training Cost — by a Factor of 20,000×

The most striking number in this paper is not a benchmark score but a ratio: 0.005% — the fraction of the teacher's pretraining tokens needed for conversion. For Qwen2.5 models trained on multi-trillion-token corpora (the Qwen2.5 technical report describes training on 18T tokens), RADLADS's 350–700M token budget represents a reduction of roughly 20,000× in the data required to produce a model of comparable quality in the linear-attention regime.

Prior conversion work established that distillation is cheaper than pretraining from scratch, with improvements ranging from SUPRA's ~10× reduction (100B tokens vs. 1T+ pretraining) to LOLCats's ~25,000× reduction (40M tokens vs. 1T+ pretraining). But the critical difference between RADLADS and LOLCats is that RADLADS achieves this data efficiency without sacrificing model quality to the point of unusability. LOLCats's pure-RNN models produced MMLU scores worse than random guessing (relative score −0.288 in Table 1), forcing the authors to resort to hybrid architectures (sliding-window softmax attention) to recover acceptable performance. RADLADS's pure-RNN models achieve relative MMLU scores of 0.924 (RAD-RWKV7, 7B) and 0.899 (RAD-RWKV6, 72B) — within 8–10% of the teacher and far above any prior pure-RNN conversion.

This decoupling — data efficiency and quality preservation are not inherently in tension — is the conceptual contribution. The field had implicitly accepted a tradeoff curve where lower token budgets meant lower quality (SUPRA at one extreme, LOLCats at the other), with MOHAWK and Llamba occupying intermediate positions. RADLADS's results shift this curve dramatically: it achieves quality above Llamba (which used 8–12B tokens) with an order of magnitude less data (350–700M tokens). The mechanism is not a single trick but the combination of architecture-compatibility optimization (Innovation 1) and the three-step framework (Innovation 2) — removing the right components from the target architecture makes step 1 alignment converge faster and better, which reduces the token budget needed in step 2.

The economic implication — converting a 72B model for under $2,000 — is the concrete instantiation of this decoupling, but the intellectual significance goes beyond cost. It means that the relationship between pretraining compute and model capability is not symmetric with respect to architecture. The knowledge embedded in the teacher's MLPs and embeddings — which the authors hypothesize contains "most" of the factual knowledge — is architecture-independent in the sense that it transfers cleanly to a new sequence mixer. The sequence-mixing computation itself, while architecturally specific, can be approximated well by a sufficiently expressive recurrent mechanism with relatively little data. This division — architecture-independent knowledge in MLPs, architecture-specific computation in the sequence mixer, with the latter being cheap to adapt — is an empirical claim about where language model capabilities reside, supported by the conversion efficiency but not proven by it (it remains a hypothesis that the authors present cautiously in Section 3).

This insight is fundamental in the sense that it challenges the default assumption that capabilities are diffusely distributed across all model parameters. If the hypothesis holds — that factual knowledge is concentrated in MLPs and embeddings, and sequence mixers mainly control how that knowledge is accessed and composed — then it has implications beyond conversion, including for model merging, pruning, and architecture design. It suggests that architectural innovation in sequence mixing can proceed largely independently of knowledge acquisition, with conversion serving as the bridge.

Innovation 4: Selective Forgetting via Delta-Rule State Updates Improves Conversion Fidelity

The architectural difference between RAD-RWKV6 and RAD-RWKV7 — the delta-rule state update with removal and replacement keys — constitutes a diagnostic finding about what makes a recurrent architecture good for approximating softmax attention. The paper does not present this as a theoretical claim, but the empirical evidence from step 1 alignment (RAD-RWKV7 "fits even closer and more rapidly") and downstream performance (RAD-RWKV7 achieves higher relative scores across all benchmarks at 7B: 0.982 LAMBADA vs. 0.970, 0.924 MMLU vs. 0.871) supports a specific mechanistic hypothesis: softmax attention's sharp query-dependent focus is better approximated by recurrent mechanisms that can actively erase old associations than by mechanisms that rely solely on decay.

In softmax attention, the output at position $t$ is $\sum_i \alpha_{ti} v_i$, where $\alpha_{ti} = \text{softmax}(q_t \cdot k_i / \sqrt{d_k})$. For tokens $i$ that are irrelevant to query $q_t$, $\alpha_{ti} \approx 0$ — their values are effectively excluded from the output. A decay-only recurrent mechanism (RAD-RWKV6) approximates this by letting the influence of token $i$ decay as $\prod_{j=i}^{t-1} w_j$. But this decay is content-agnostic: it applies the same per-channel decay to all stored associations, regardless of whether they are relevant to the current query. To suppress an irrelevant token's influence quickly, the model must set high decay rates (low $w_t$), which also suppresses potentially relevant older information — a form of collateral forgetting.

The delta-rule mechanism in RAD-RWKV7 solves this by introducing the removal term $-\hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)$ in the state update. This term subtracts a rank-1 matrix from the state that is specifically targeted at associations matching the removal key $\hat{\kappa}_t$. The model can learn to set $\hat{\kappa}_t$ to match the keys of information that should be forgotten (because it's no longer relevant to the current query) while preserving information matching other keys. The replacement key $\tilde{k}_t$ simultaneously writes new information into the now-vacated subspace. This is qualitatively closer to how softmax attention operates: it doesn't gradually forget irrelevant tokens — it ignores them completely from the start, focusing the value computation on the tokens with the highest attention weights.

The evidence that this matters for conversion, rather than just being a general architectural improvement, comes from the step 1 alignment comparison. RAD-RWKV7 achieves lower L2 loss to the teacher's attention outputs in fewer tokens, suggesting that the delta rule can more easily reproduce the sharp, query-conditioned output patterns of softmax attention than decay-only mechanisms can. This is a novel empirical finding — prior work on delta-rule architectures (DeltaNet, Gated DeltaNet) demonstrated their effectiveness for pretraining and associative recall tasks, but their specific advantage for approximating a frozen softmax-attention computation during distillation had not been characterized.

The significance of this finding extends beyond RADLADS. It suggests a design principle for future recurrent architectures intended for conversion: the mechanism should provide explicit, content-addressable erasure capability, not just content-agnostic decay. This principle can guide the development of new architectures optimized specifically for conversion fidelity (which may differ from those optimized for pretraining stability or training throughput), establishing a sub-field of conversion-aware RNN design.

This is an incremental but practically significant advance — it refines the understanding of which architectural properties matter for conversion, building on the known expressiveness of RWKV-7 while identifying the specific mechanism (selective removal) that drives the improvement and the specific component (tokenshift) that becomes redundant when this mechanism is present.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All models are evaluated on a set of standard English language benchmarks: LAMBADA (Paperno et al., 2016), MMLU (Hendrycks et al., 2021), ARC-C and ARC-E (Clark et al., 2018), PIQA (Bisk et al., 2020), Winogrande (Sakaguchi et al., 2021), and HellaSwag (Zellers et al., 2019). For the conversion training itself, the DCLM dataset (Li et al., 2024b) is used across all three steps; the authors report having tried FineWeb, FineWeb-Edu, and custom datasets before settling on DCLM, which "worked exceptionally well for us when converting Qwen models" (Section 3). The paper does not specify the exact DCLM subset or preprocessing pipeline beyond naming the dataset. Evaluation is conducted in a zero-shot setting for all benchmarks except where noted (LOLCats MMLU is 5-shot).

  • Base model(s). All RADLADS conversions start from the Qwen2.5 family of open-source transformer models (Qwen et al., 2025): Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, Qwen2.5-72B-Instruct, and Qwen2.5-QwQ-32B (a reasoning-focused variant). One additional model — QRWKV7-7B-Instruct-from72B — is distilled from the 72B teacher to a 7B student, enabling a cross-scale distillation comparison. The choice of Qwen2.5 is not defended at length; the authors treat these as representative state-of-the-art open-weight transformers. The paper does not report experiments on other model families (e.g., Llama, Mistral) except as teacher baselines for prior work cited in Table 2 and Table 3.

  • Metrics. The primary metrics are per-benchmark accuracy scores (e.g., MMLU accuracy, LAMBADA accuracy, etc.) and a derived relative score defined as:

    Relative Score=s−rt−r\text{Relative Score} = \frac{s - r}{t - r}

    where s is the student model's accuracy, t is the teacher model's accuracy, and r is the expected accuracy from random guessing on that benchmark (e.g., 0.25 for 4-way multiple choice, 0.10 for 10-way, etc. — the exact values of r are not enumerated in the paper but follow standard practice for each benchmark). A relative score of 1.0 means the student matches the teacher exactly; above 1.0 means the student outperforms the teacher; negative values indicate performance worse than random guessing. This normalization allows cross-comparison of conversion quality across different teacher models and scales, since it controls for the teacher's absolute performance.

  • Baselines. The paper compares against the following prior conversion methods, each producing either pure-RNN or hybrid models from transformer teachers: SUPRA (Mercat et al., 2024) — pure-RNN, 100B tokens, Mistral-7B teacher; Mamba2-Llama3.0-8B-Instruct (Wang et al., 2024) — hybrid with 12.5%/25%/50% softmax attention retention, 20B tokens; MOHAWK-Phi1.5-1.3B (Bick et al., 2024b) — pure-RNN, 3–5B tokens; DiJiang-Llama2.0-7B (Chen et al., 2024) — pure-RNN, 40B tokens; Llamba-8B (Bick et al., 2025) — pure-RNN, 8–12B tokens, distilled from Llama-3.1-70B; LOLCatsHedgehog-Llama3.0-8B (Zhang et al., 2024a) — pure-RNN (40M tokens) and hybrid SWA variants; and ARWKV-7B (Yueyu et al., 2025) — pure-RNN, converting Qwen2.5-7B-Instruct to RWKV-7 using early RADLADS techniques. Teacher models for each baseline are also reported where available. The paper additionally reports hybrid LOLCats variants (Tables 4 and 5) with 100% sliding-window softmax attention retained, and Mamba2-Llama hybrid variants with varying attention retention percentages, for comparison against pure-RNN RADLADS models.

  • Generation budget / compute accounting. The paper does not use test-time generation budgets in the evaluation sense (these are not inference-time scaling experiments). Compute comparisons between methods are instead made on the basis of training tokens required for conversion: the RADLADS protocol uses 350–700M tokens total (100M step 1, 250–500M step 2, 100M step 3), while prior methods range from 40M (LOLCats) to 100B (SUPRA). The paper's central efficiency claim — converting a 72B model for less than $2,000 — is based on Table 8, which reports approximate GPU-hours on 8× AMD Mi300X GPUs for each model scale and step. No formal FLOPs accounting is provided; the token count serves as the primary compute metric, with GPU-hours as a practical cost proxy.

  • Cross-validation / statistical protocol. None reported. All benchmark evaluations appear to be single-run accuracy measurements on standard test sets. The paper does not describe any statistical significance testing, confidence intervals, or multiple-run averaging for the reported scores. For ablation studies (Table 6), each variant is the result of a single conversion run at 7B scale with 100M step 1 tokens and 500M step 2 tokens. The paper does not discuss variance across random seeds, data order, or initialization.

Main Quantitative Results

The experimental results are organized around two comparison axes: (1) RADLADS models vs. prior pure-RNN and hybrid conversion methods, benchmarked on standard downstream tasks, and (2) within the RADLADS family, the effect of architectural choices (RAD-RWKV6 vs. RAD-RWKV7, with and without RoPE, with and without cross-scale distillation).

RADLADS vs. Prior Pure-RNN Conversion Methods

Headline result (Table 1, Table 3): RADLADS models achieve state-of-the-art downstream performance among pure RNN language models at every tested scale, with the 72B QRWKV6-Instruct reaching relative scores of 1.004 on LAMBADA, 0.899 on MMLU, and 1.015–1.123 on ARC-C, ARC-E, HellaSwag, PIQA, and Winogrande — indicating the converted model matches or slightly exceeds the teacher on LAMBADA and comes within ~10% on MMLU. At 7B, QRWKV7-7B-Instruct achieves relative scores of 0.982 on LAMBADA and 0.924 on MMLU, substantially above all prior pure-RNN methods at comparable scale.

Table 1 (relative scores, pure-RNN methods ≤ 8B): The comparison is structured to make cross-method evaluation possible despite different teacher models. ARWKV-7B — the most directly comparable prior method since it also converts Qwen2.5-7B-Instruct — achieves 0.893 LAMBADA and 0.801 MMLU relative scores. RADLADS's QRWKV7-7B-Instruct achieves 0.982 and 0.924 respectively — a gain of 8.9 percentage points on LAMBADA and 12.3 points on MMLU over the independent replication using shared techniques. This demonstrates that the specific architectural modifications (RAD-RWKV7 vs. standard RWKV-7), hyperparameter choices, and DCLM dataset selection account for substantial quality differences even when the general approach is similar.

Llamba-8B, which the authors describe as "a generous comparison" since it distills from a 70B teacher to an 8B student (vs. RADLADS's 7B-from-7B), achieves 0.951 LAMBADA and 0.837 MMLU. The QRWKV7-7B-Instruct-from72B model — distilled from Qwen2.5-72B-Instruct to a 7B student — achieves 1.016 LAMBADA and 0.893 MMLU, exceeding Llamba on both while using 20–30× fewer training tokens (350–700M vs. 8–12B). This directly supports the paper's central claim that high-quality conversion does not require large training budgets when architecture and protocol are well-matched.

The older methods — SUPRA (0.913 LAMBADA, 0.216 MMLU), MOHAWK (−0.047 MMLU), DiJiang (relative scores mostly missing but MMLU negative), and LOLCats/Hedgehog (−0.288 MMLU) — all exhibit MMLU scores near or below random guessing, demonstrating that prior methods either failed to preserve the teacher's knowledge or required hybrid architectures to achieve usable quality. The raw accuracy values in Table 3 confirm this: SUPRA achieves 0.331 absolute MMLU accuracy from a Mistral-7B teacher scoring 0.624; LOLCats/Hedgehog achieves 0.238 MMLU from a Llama3.0-8B teacher scoring 0.533. RADLADS's QRWKV7-7B-Instruct, by contrast, achieves 0.682 MMLU from a Qwen2.5-7B-Instruct teacher scoring 0.717 — a drop of only 3.5 absolute percentage points.

Table 3 (absolute accuracy scores): The raw numbers reveal that absolute performance varies substantially by teacher quality and model scale, making the relative scores essential for cross-comparison. At 72B, QRWKV6-72B-Instruct achieves LAMBADA 0.754 (teacher: 0.751), MMLU 0.775 (teacher: 0.834), ARC-C 0.638 (teacher: 0.632), and Winogrande 0.796 (teacher: 0.763). The student exceeds the teacher on LAMBADA, ARC-C, and Winogrande — a finding the paper does not elaborate on, but which suggests that the distillation process (steps 1–2) and DCLM finetuning (step 3) may provide a form of continued training on high-quality data that slightly improves some capabilities beyond the original teacher. The MMLU drop (0.834 → 0.775, −5.9 absolute points) is the most significant gap between student and teacher at 72B, indicating that multi-task factual knowledge is the dimension most affected by conversion. This is consistent with the hypothesis that the sequence mixer replacement most impacts the model's ability to access and compose knowledge across its training distribution, while factual storage in MLPs is largely preserved.

At 32B, QRWKV6-32B-Instruct achieves MMLU 0.766 (teacher: 0.818) — a drop of 5.2 points — and Winogrande 0.782 (teacher: 0.729), meaning the student outperforms the teacher by 5.3 points on Winogrande. At 7B, QRWKV7-7B-Instruct achieves MMLU 0.682 (teacher: 0.717, drop of 3.5 points) and LAMBADA 0.684 (teacher: 0.696, drop of 1.2 points). The consistent pattern across scales is that MMLU shows the largest absolute degradation from conversion, while LAMBADA, ARC-C, ARC-E, PIQA, and Winogrande show smaller drops or occasional improvements. This is interpretable: MMLU is a broad factual-knowledge benchmark requiring the model to retrieve and apply specific information across dozens of domains, which stresses the sequence mixer's ability to route information from the context to the relevant stored knowledge. The recurrent mechanism, even when well-aligned, may not perfectly replicate the softmax attention's information-routing patterns for this diverse retrieval task.

RADLADS vs. Hybrid Conversion Methods

Headline result (Tables 4 and 5): Even when compared against hybrid models that retain some proportion of softmax attention (and therefore do not achieve O(1) inference), RADLADS's pure-RNN models achieve competitive or superior relative MMLU scores. This is notable because it undermines the implicit assumption in prior work (e.g., Mamba in the Llama, LOLCats with SWA) that retaining some softmax attention is necessary to preserve downstream quality — RADLADS shows that a well-matched pure-RNN architecture with proper training can match or exceed hybrids on many benchmarks.

Table 4 (relative scores, hybrid models): The Mamba2-Llama3.0-8B-Instruct hybrids show a gradient: at 12.5% softmax attention retention, MMLU relative score is 0.663; at 25%, 0.738; at 50%, 0.789 — never reaching RADLADS's 0.924 (QRWKV7-7B-Instruct) even with half the attention layers retained. This is a strong result because it shows that the RADLADS protocol with a pure-RNN architecture outperforms a hybrid that keeps substantial softmax attention, despite the hybrid having strictly more representational capacity (it can fall back to exact softmax computation in some layers).

The LOLCats sliding-window models (100% softmax attention in a window) achieve MMLU relative scores of 0.706 (Mistral-7B), 0.727 (Llama3.1-8B), and 0.794 (Llama3.1-70B) — all below RADLADS's pure-RNN scores. However, this comparison is confounded by different teacher models: LOLCats uses Mistral and Llama teachers, while RADLADS uses Qwen2.5. The relative score normalization is designed to remove this confound by measuring recovery of the teacher's own performance, but differences in how amenable different teacher architectures are to conversion cannot be ruled out.

Table 5 (absolute accuracy scores, hybrid models): The absolute numbers provide additional context. The LOLCats-Llama3.1-70B SWA model achieves MMLU 0.677 (teacher: 0.788) — a substantial 11.1-point absolute drop despite retaining softmax attention in a sliding window. RADLADS's QRWKV6-72B-Instruct achieves MMLU 0.775 (teacher: 0.834) — a drop of 5.9 points with no softmax attention retained. The pure-RNN RADLADS model outperforms the hybrid on absolute MMLU (0.775 vs. 0.677) despite the latter having access to exact attention computation on local windows. This is evidence that the choice of RNN architecture and training protocol matters more than whether softmax attention is partially retained.

Architectural Comparisons Within RADLADS

Headline result (Tables 3 and 6): RAD-RWKV7 consistently outperforms RAD-RWKV6 at the same scale, with the 7B comparison showing RAD-RWKV7 achieving MMLU 0.682 vs. 0.657 for RAD-RWKV6 (absolute), and relative scores of 0.924 vs. 0.871. The addition of RoPE to RAD-RWKV6 produces minimal changes (MMLU 0.661 vs. 0.657), suggesting that the tokenshift mechanism in RAD-RWKV6 already provides sufficient positional information and that RoPE is not a critical component for conversion quality in that architecture — though it is used in RAD-RWKV7 due to its native compatibility with the Qwen2.5 teacher's RoPE scheme.

Table 3, RAD-RWKV6 vs. RAD-RWKV7 at 7B: QRWKV7-7B-Instruct outperforms QRWKV6-7B-Instruct on every benchmark except ARC-C (1.026 vs. 1.043 relative) and PIQA (1.013 vs. 1.030 relative). The MMLU gap (0.924 vs. 0.871 relative, or 0.682 vs. 0.657 absolute) is the largest single-benchmark difference, consistent with the hypothesis that the delta-rule mechanism's selective forgetting capability is most beneficial for tasks requiring precise retrieval and composition of stored knowledge — exactly the demands of MMLU.

QRWKV6-7B-Instruct RoPE variant: The addition of RoPE changes MMLU relative score from 0.871 to 0.880 and LAMBADA from 0.970 to 0.969 — differences within 1 percentage point. This ablation (which the paper treats as a variant, not a formal ablation in Table 6) supports the claim that RoPE is not essential for RAD-RWKV6 conversion quality, since the tokenshift mechanism provides sufficient positional signal. The authors do not explain why RoPE was included in RAD-RWKV7 but not RAD-RWKV6, but the likely reason is architectural: RAD-RWKV7 removed tokenshift entirely, making RoPE necessary for positional encoding, while RAD-RWKV6 retained tokenshift, making RoPE redundant.

Cross-scale distillation (QRWKV7-7B-Instruct-from72B): This model, distilled from the 72B teacher instead of the 7B teacher, achieves the highest relative scores in the entire paper: LAMBADA 1.016, MMLU 0.893, and notably high scores across ARC (1.071, 1.033), PIQA (1.026), and Winogrande (1.139). The fact that a 7B student distilled from a 72B teacher outperforms a 7B student distilled from a 7B teacher on most benchmarks is expected — the 72B teacher contains more knowledge — but the magnitude is instructive. The 7B-from-72B student achieves absolute LAMBADA 0.707 (vs. 7B-from-7B's 0.684), MMLU 0.667 (vs. 0.682 — here the 7B-from-7B actually does slightly better), and Winogrande 0.739 (vs. 0.718). The MMLU result is the exception: the larger teacher's MMLU knowledge does not transfer proportionally to the smaller student via RADLADS, suggesting that MMLU capability depends partly on model capacity (number of parameters) and not just on the teacher's knowledge base. This is a well-known phenomenon in knowledge distillation — the student's capacity constrains what can be transferred — and its appearance here validates that RADLADS does not magically circumvent this fundamental limit.

Inference Efficiency (Implicit Results)

While the paper's stated contributions center on conversion cost and downstream quality, the motivation includes inference efficiency: linear attention offers O(1) per-token computation vs. O(N) for softmax attention. The paper does not present empirical inference benchmarks (tokens/second, latency measurements, memory usage comparisons) for the converted QRWKV models vs. their Qwen2.5 teacher counterparts. The efficiency claims are theoretical, based on the known asymptotic properties of linear attention:

"linear attention is computable in O(1) time per token instead of O(N) time for softmax transformers, and avoids the expensive memory bandwidth usage of a Key Value cache." (Section 1)

The paper provides no measurements of actual inference speedup, memory savings, or latency reduction on specific hardware. This is a notable gap: while the asymptotic advantage is well-established, practical factors — kernel implementation quality, hardware utilization for recurrent vs. attention operations, and the overhead of the specific RAD-RWKV operations (ddlerp, delta-rule state updates) — determine whether the theoretical advantage translates to wall-clock improvements at the scales tested (7B–72B). The authors' claim that RAD-RWKV7's tokenshift removal "speeds up training and inference" (Appendix B.2) is asserted without timing data.

Ablation Studies and Robustness Checks

All ablations are conducted on the RAD-RWKV6 architecture at 7B scale, converting Qwen2.5-7B-Instruct, with 100M tokens of step 1 training followed by 500M tokens of step 2 training. Results are reported in Table 6.

  • Full RAD-RWKV6 (no ablation): The baseline configuration achieves LAMBADA 0.6748, MMLU 0.6572, ARC-C 0.5631, ARC-E 0.8136, HellaSwag 0.7901, PIQA 0.8025, Winogrande 0.7111. These are absolute accuracy scores on the respective benchmarks.

  • Use RoPE: Adding rotary positional embeddings produces LAMBADA 0.6740 (nearly identical), MMLU 0.6610 (slight improvement of 0.38 points), ARC-C 0.5666 (slight improvement), ARC-E 0.8165 (slight improvement), PIQA 0.7992 (slight decline), Winogrande 0.7056 (decline of 0.55 points). No benchmark changes by more than 0.6 points. This confirms that RoPE is approximately neutral for RAD-RWKV6 conversion quality — it neither helps nor hurts substantially, which justifies the design choice to omit it from the main RAD-RWKV6 models (since RAD-RWKV6's tokenshift already handles positional information, RoPE is redundant rather than harmful).

  • No tokenshift: Removing the entire ddlerp/tokenshift mechanism produces LAMBADA 0.6707 (drop of 0.41 points), MMLU 0.6584 (essentially unchanged), ARC-C 0.5546 (drop of 0.85 points), ARC-E 0.8043 (drop of 0.93 points), HellaSwag 0.7874 (drop of 0.27 points), PIQA 0.8036 (essentially unchanged), Winogrande 0.6875 (drop of 2.36 points, the largest effect in any ablation). The Winogrande degradation is the headline finding: Winogrande tests pronoun resolution and entity tracking across sentences, tasks that rely on the model's ability to maintain and update representations of entities over a discourse context. The disproportionate Winogrande impact suggests that tokenshift's local temporal convolution is particularly important for short-range entity tracking — the model needs to know how the representation of "the trophy" changed between consecutive sentences to correctly resolve "it" in a Winogrande example. This validates the inclusion of tokenshift in RAD-RWKV6 and explains why it was retained despite being removed in RAD-RWKV7 (where the delta-rule mechanism may serve a similar discourse-tracking function through selective state updates).

  • No gate: Removing the sigmoid gate (making $o_t = p_t \mathbf{W}_o$ directly, without the $g_t \odot p_t$ modulation) produces LAMBADA 0.6590 (drop of 1.58 points), MMLU 0.6417 (drop of 1.55 points), ARC-C 0.5444 (drop of 1.87 points), ARC-E 0.8077 (drop of 0.59 points), HellaSwag 0.7842 (drop of 0.59 points), PIQA 0.7971 (drop of 0.54 points), Winogrande 0.6930 (drop of 1.81 points). The degradation is broadly distributed across all benchmarks rather than concentrated on one task, indicating that the gate serves a general quality-control function — it allows the model to learn which channels of the attention output to trust and which to suppress, and removing it degrades performance uniformly. This ablation confirms the authors' claim that gating is "extremely beneficial only at full rank in RAD-RWKV6" (Section 3) — here tested at full rank and found essential.

  • Use GroupNorm (replacing state balancing): Substituting the $k_t = \tilde{k}_t(1 - w_t)d_k^{-0.5}$ state-balancing mechanism with GroupNorm applied to the recurrent state produces LAMBADA 0.6559 (drop of 1.89 points), MMLU 0.6340 (drop of 2.32 points, the largest MMLU degradation) , ARC-C 0.5648 (slight increase of 0.17 points), ARC-E 0.8186 (slight increase of 0.50 points), HellaSwag 0.7865 (drop of 0.36 points), PIQA 0.7905 (drop of 1.20 points), Winogrande 0.7032 (drop of 0.79 points). The MMLU degradation is the key finding: state balancing — the simple mechanism of scaling keys by $(1 - w_t)d_k^{-0.5}$ — is more effective at maintaining stable recurrent state norms than explicit normalization via GroupNorm, and this stability matters particularly for the diverse retrieval task that MMLU represents. The authors attribute this improvement to "the use of the state balancing technique from RWKV6-C2 [which] enables us to remove state normalization, improving downstream performance" (Appendix B.1).

Non-obvious patterns across ablations:

  • MMLU is the most sensitive benchmark to architectural changes. The MMLU score varies from 0.6340 (GroupNorm) to 0.6610 (RoPE), a range of 2.7 points, while LAMBADA varies only 1.9 points across the same ablations. This pattern — MMLU being the most architecture-sensitive benchmark — is consistent with the main results showing MMLU as the dimension where the student-teacher gap is largest. It suggests that MMLU's multi-domain factual retrieval task stresses the sequence mixer's information-routing capabilities more than any other benchmark tested.

  • Most ablations cause broad degradation rather than task-specific effects. With the exception of Winogrande's sensitivity to tokenshift removal, removing any component (gate, tokenshift, state balancing) causes declines across most benchmarks rather than creating a task-specific tradeoff. This suggests that these components are genuinely necessary for the recurrent mechanism to function properly — they are not specialized features that help on some tasks while being neutral on others.

  • No ablation results in catastrophic collapse. Even the worst-performing ablation (GroupNorm) achieves MMLU 0.6340 — well above random (0.25) and substantially above prior methods like SUPRA (0.331) and MOHAWK (−0.047 relative). This indicates that the RADLADS conversion process is robust: even suboptimal architectural choices produce usable models, even if quality is noticeably degraded. This robustness is a practical strength — conversion succeeds (produces above-random models) under a range of configurations, even if optimal performance requires careful component selection.

Negative results (Section 8): The paper reports several experiments that were tried and abandoned, which serve as implicit ablations of the protocol itself:

  • Initial attention-score alignment ("step 0"): Training time-mixing layers to match the teacher's attention-score matrices before hidden-state alignment "did not improve total training time nor lower final loss" and "extended training of step 0 incurred higher final loss." This negative result validates the choice to omit explicit attention-score matching from the protocol.

  • Skipping step 1: "Starting directly with step 2 distillation and skipping step 1 entirely resulted in a much lower performance model. The same level of convergence simply did not occur, with loss plateauing at a higher minimum, even with longer training." This validates the two-phase design.

  • De-novo initialization of attention weights: Initializing QKVO weights randomly instead of copying from the teacher produced "consistently worse yet surprisingly reasonable performance." The "surprisingly reasonable" qualifier is important — it shows that step 1 alignment can partially recover from random initialization, but weight transfer provides a substantial head start.

  • Freezing model weights during step 2: "This results in significantly reduced model performance," motivating the choice to train all parameters during distillation despite the hypothesis that MLPs store the bulk of factual knowledge.

  • Larger batch sizes: Did not accelerate convergence because "the number of optimizer steps appears to be of key importance during conversion."

  • LoRA training: "Rank reduction was generally quite detrimental to performance" except for embeddings, explaining why full-weight training is preferred over parameter-efficient methods like those used in LOLCats.

  • Switching datasets during step 3: Using custom datasets instead of DCLM for context-length extension "seemed to create a confused model that would use more and more adjectives as generation progressed."

These negative results are reported transparently but without quantitative detail (no loss curves, no benchmark scores for the failed configurations). They function as design rationale rather than as formal ablations, explaining why the protocol takes its final form but not enabling independent verification of the claimed failure modes.

Critical Assessment

The experiments demonstrate a clear and internally consistent pattern: RADLADS consistently outperforms prior conversion methods on downstream benchmarks when evaluated on Qwen2.5 teacher models, across multiple scales and both architectural variants. The relative score metric and the inclusion of teacher baseline scores in Table 3 enable meaningful cross-method comparison despite different teacher models. The ablation studies (Table 6) provide controlled evidence that the specific architectural choices — gating, tokenshift, state balancing — each contribute measurably to conversion quality. However, several aspects of the experimental design limit the generality and strength of the conclusions.

Claim: RADLADS achieves state-of-the-art performance among pure RNN language models. This claim is well-supported for the specific combination of Qwen2.5 teacher models and the DCLM dataset. All RADLADS models outperform all cited prior pure-RNN models on the reported benchmarks, and the relative score metric controls for teacher quality differences. However, the claim's scope is narrower than it appears because:

  • The comparison set is small and heterogeneous. Only 7 prior pure-RNN methods with published benchmark scores are available for comparison, spanning different teacher models, scales (1.3B to 8B for the direct competitors; RADLADS additionally reports 32B and 72B), and training paradigms. ARWKV is the only method that converts the same teacher model (Qwen2.5-7B-Instruct) to a comparable architecture (RWKV-7), and RADLADS's advantage over ARWKV — while clear — reflects differences in architectural details, hyperparameters, and dataset that are hard to disentangle. A fairer evaluation would include a RADLADS conversion using the exact standard RWKV-7 architecture (without the RAD modifications) as a controlled baseline, isolating the effect of the protocol from the effect of the architecture. The paper does not report this.

  • No results on non-Qwen teachers. The claim that RADLADS is a general conversion protocol is supported only by Qwen2.5 conversions. The authors state that the code can be adapted to "a transformer of their choice" (Section 1), but no Llama, Mistral, or other conversions are reported. The finding that DCLM "worked exceptionally well when converting Qwen models" and the hypothesis that "the best choice of dataset may depend upon the teacher model's pretraining data distribution" (Section 3) imply that the specific configuration may not transfer directly. Without at least one non-Qwen conversion, the generality claim is aspirational.

  • The paper does not compare against training a linear-attention model from scratch at comparable scale. The motivation (Section 1) is that pretraining from scratch is cost-prohibitive, but this is an economic argument, not an experimental one. It is possible that a 7B RWKV-7 model pretrained from scratch on, say, 100B tokens (still far less than the multi-trillion-token transformer pretraining runs) could match or exceed the converted QRWKV7-7B quality. The conversion approach is only strictly superior if it achieves higher quality per training token than from-scratch pretraining, which is not tested.

Claim: Conversion cost is under $2,000 for 72B models. This claim is supported by Table 8's timing estimates and reasonable cloud GPU pricing assumptions, but:

  • The cost calculation excludes development cost. The 2,000figurerepresentsonesuccessfulconversionrunafterthearchitecturaldesign,hyperparametertuning,andnegativeresultexplorationdescribedinSections3and8.ThetotalcomputespentdevelopingRADLADS—includingfailedexperiments,architectureiterations,andhyperparametersweeps—isnotreportedbutiscertainlymuchhigher.Thisisstandardforresearchpapers(thereportedcostisthemarginalcostofapplyingthemethod,notthecostofinventingit),butitmeansthe2,000 figure represents one successful conversion run after the architectural design, hyperparameter tuning, and negative result exploration described in Sections 3 and 8. The total compute spent developing RADLADS — including failed experiments, architecture iterations, and hyperparameter sweeps — is not reported but is certainly much higher. This is standard for research papers (the reported cost is the marginal cost of applying the method, not the cost of inventing it), but it means the 2,000 figure should not be interpreted as "anyone can convert a 72B model for $2,000 without additional experimentation."

  • VRAM requirements limit accessibility. Table 8 reports that the 72B conversion uses 8× AMD Mi300X GPUs (192GB each), and notes that 16,384-token context length in step 3 "does not fit on a single node at 72B scale." The GPU hardware required (8× Mi300X represents a capital cost well into six figures) is not available to the "small team with modest compute" the paper envisions (Section 2). The $2,000 figure reflects cloud rental cost, which assumes access to cloud instances with sufficient GPU availability — a realistic assumption for well-funded labs but not for individual researchers or small academic groups.

  • Token counts are modest but not trivially small. 700M tokens is 0.005% of the teacher's pretraining data, but it still represents a non-trivial training run. At 72B scale, step 2 alone takes 54 GPU-hours on 8× Mi300X — several days on a single node. This is not "rapid" in the sense of minutes-to-hours turnaround that would enable the fast architectural iteration cycle the paper envisions. The "rapid" in RADLADS is relative to from-scratch pretraining, not relative to typical fine-tuning or LoRA-based approaches.

Claim: The three-step decomposition is critical to conversion success. The negative results in Section 8 provide qualitative evidence that each step matters (skipping step 1 hurts, freezing MLPs in step 2 hurts), but:

  • No quantitative ablation of the full three-step vs. two-step vs. one-step protocol is reported. We do not know the exact benchmark penalty of skipping step 1 (only that loss "plateaus at a higher minimum"), the penalty of skipping step 3 (using the step 2 model at 16K context without dedicated context-length training), or the penalty of combining steps 2 and 3 (distilling at long context directly). These ablations would quantify the value of the decomposition itself, rather than just the value of individual components within steps.

  • The negative results are described qualitatively without benchmark scores or loss curves. "Much lower performance model," "significantly reduced model performance," and "consistently worse yet surprisingly reasonable" are subjective descriptions. The skeptical reader cannot assess whether "much lower" means a 2% or 20% absolute MMLU degradation. This is a missed opportunity: quantitative negative results are often more informative than positive ones for understanding failure modes.

Weaknesses in the evaluation methodology:

  • No confidence intervals or statistical testing. All benchmark scores are single-point estimates from a single conversion run. Given that benchmark scores for 7B models on MMLU typically vary by ±1–2 points across runs (due to data order, hardware nondeterminism, and benchmark sampling when not all test examples are used), some of the smaller differences in Table 6 (e.g., LAMBADA 0.6748 vs. 0.6740 for the RoPE ablation) may not be statistically significant. Without variance estimates, readers cannot distinguish signal from noise in the finer comparisons.

  • No held-out validation protocol for hyperparameter selection. The paper reports that learning rates, token counts, and dataset choices were arrived at through iteration, but there is no description of a validation set or cross-validation procedure to prevent overfitting the conversion protocol to the test benchmarks. Since the test benchmarks (MMLU, LAMBADA, etc.) are public and widely used, and the conversion protocol was developed and refined using Qwen2.5 models evaluated on these exact benchmarks, there is risk that the hyperparameters are implicitly tuned to these specific evaluation sets.

  • Context-length evaluation is absent. The paper extends context to 16K tokens in step 3 but reports zero-shot evaluations on standard benchmarks (LAMBADA, MMLU, etc.) that do not require long-context processing. There is no evaluation of the model's quality at long context lengths — e.g., perplexity on long documents, needle-in-a-haystack retrieval accuracy, or long-context QA tasks. The claim that step 3 produces a model with "enhanced long-context modeling capabilities" (Section 4.4) is unsupported by any long-context evaluation. The RADLADS contribution is primarily about preserving short-context benchmark performance after conversion; the long-context extension is aspirational and unevaluated.

  • No comparison to the teacher at long context. Even if long-context evaluations were included, comparing the student's long-context performance to the teacher's would be necessary to assess whether conversion preserves or degrades the teacher's long-context capabilities. The Qwen2.5 teachers are reported to support up to 128K tokens (Qwen et al., 2025); whether the converted QRWKV models match this is unknown.

  • Missing efficiency benchmarks. As noted above, the paper provides no inference throughput, latency, or memory measurements. The entire motivation — linear attention is more efficient at inference — is taken as given from prior work (Arora et al., 2023) without verifying that the specific RAD-RWKV architectures, with their ddlerp mechanisms, low-rank MLP helpers, and (in RWKV-7) delta-rule state updates, actually achieve the theoretical O(1) advantage in practice on modern GPU hardware. The ddlerp computations involve multiple low-rank matrix multiplications and element-wise operations per token, and the state update in RAD-RWKV7 involves outer products and matrix multiplications of $(D/h) \times (D/h)$ matrices — operations whose constant factors may be substantial. Without measured throughput against the softmax-attention baseline, the practical efficiency gain is unquantified.

Experiments that would have strengthened the paper:

  • A controlled comparison converting the same Qwen2.5-7B-Instruct model using standard RWKV-7 (without RAD modifications) to isolate the architecture effect from the protocol effect.
  • At least one non-Qwen conversion (e.g., Llama-3.1-8B) with the identical protocol to test generality.
  • Long-context evaluations (perplexity at 8K/16K/32K, needle-in-haystack retrieval) for the step 3 model vs. the step 2 model and vs. the teacher.
  • Inference throughput benchmarks (tokens/second, memory usage) comparing QRWKV models to their Qwen2.5 teacher counterparts at matching batch sizes and sequence lengths.
  • Quantitative negative-result scores (e.g., what is the MMLU score when step 1 is skipped? when MLPs are frozen? etc.) to calibrate the qualitative descriptions.
  • Multiple conversion runs with different random seeds to assess variance in the reported benchmark scores.

Overall, the experiments convincingly demonstrate that RADLADS produces higher-quality pure-RNN conversions than prior published methods, and that the architectural choices embodied in RAD-RWKV6 and RAD-RWKV7 each contribute meaningfully to this quality. The evidence is strong for the specific configuration tested (Qwen2.5 + DCLM + 7B–72B). The evidence is weaker for generality (no non-Qwen conversions), for practical efficiency (no inference benchmarks), for long-context capability (no long-context evaluations), and for statistical reliability (no confidence intervals or multi-run variance estimates). The central efficiency claim — 350–700M tokens, under $2,000 — is accurate for the marginal cost of one conversion run on the reported hardware, but excludes development cost and assumes cloud GPU access at scale.

6. Limitations and Trade-offs

Limitation 1: No Empirical Validation of Inference Efficiency Gains

The assumption: The paper’s entire motivation — converting transformers to linear attention — rests on the theoretical O(1) per-token inference complexity and elimination of the KV cache, which the authors state as established fact:

"linear attention is computable in O(1) time per token instead of O(N) time for softmax transformers, and avoids the expensive memory bandwidth usage of a Key Value cache." (Section 1)

The consequence: The paper provides zero empirical measurement of inference throughput, latency, or memory consumption for the converted QRWKV models relative to their Qwen2.5 teacher counterparts. This is not a minor omission — it is a gap at the core of the paper’s value proposition. The RAD-RWKV architectures introduce several mechanisms that could erode the theoretical O(1) advantage in practice:

  • The ddlerp operation in RAD-RWKV6 requires multiple low-rank matrix multiplications ($$\mathbf{A}_{\square} \in \mathbb{R}^{D \times z}$$, $$\mathbf{B}_{\square} \in \mathbb{R}^{z \times D}$$) per timestep per variable (r, v, g, $$\tilde{w}$$, $$\tilde{k}$$), plus element-wise $$\tanh$$, interpolation, and multiplication operations.
  • The RAD-RWKV7 delta-rule state update involves outer products and matrix multiplications of $$(D/h) \times (D/h)$$ matrices — at 72B scale with typical head dimensions, this is substantial computation.
  • The removal key normalization ($$\hat{\kappa}_t = \kappa_t / \|\kappa_t\|_2$$) requires an L2 norm computation per head per timestep.

Whether the constant factors from these operations make the recurrent models faster or slower than optimized FlashAttention implementations at practical sequence lengths (512–16K tokens) is entirely unknown. A practitioner considering deploying QRWKV models instead of the original Qwen2.5 models has no data to determine whether the conversion actually saves inference time or memory.

What evidence exists in the paper: None. The only efficiency-related claim is qualitative: the authors assert that removal of tokenshift in RAD-RWKV7 "speeds up training and inference" (Appendix B.2), but provide no timing measurements for either. Table 8 reports training GPU-hours for conversion, not inference benchmarks for the resulting models.

Mitigation status: Not addressed. The paper treats the inference efficiency advantage as axiomatic based on asymptotic complexity, which is standard in the linear-attention literature but insufficient for a methods paper whose primary contribution is producing deployable models. A minimal evaluation would report tokens/second and peak memory usage for QRWKV vs. Qwen2.5 at identical batch sizes across a range of sequence lengths (512, 2K, 8K, 16K). The authors do not flag this as future work.

Limitation 2: Conversion Protocol Validated on a Single Model Family

The assumption: The RADLADS protocol is presented as a general method for converting "a transformer of [the user's] choice" (Section 1) to linear attention. The paper states:

"we settled on DCLM for all our conversions. We theorize that the best choice of dataset may depend upon the teacher model’s pre-training data distribution. DCLM worked exceptionally well for us when converting Qwen models." (Section 3)

The consequence: Every model released and evaluated in this paper is converted from the Qwen2.5 family. No Llama, Mistral, Gemma, or other architecture is tested. This matters for three distinct reasons:

  1. Teacher architecture dependence. Qwen2.5 uses a specific attention implementation (GQA with RoPE, specific head counts and ratios), specific MLP architectures, and specific training data distributions. Whether the RAD-RWKV time-mixing blocks can align equally well to, say, Llama-3.1’s attention patterns — which use a different GQA configuration, different RoPE frequencies, and different pretraining data — is unknown. Components like the value residual gating in RAD-RWKV7 ($$\nu_t$$ interpolating between layer-0 and layer-local values; Appendix B.2) rely on the teacher having a specific per-layer structure that may not transfer.

  2. Dataset–teacher interaction. The authors’ hypothesis that dataset choice depends on the teacher’s pretraining distribution implies that DCLM may not be optimal for non-Qwen teachers. A practitioner converting a Llama model might need to run their own dataset ablation — fineweb, fineweb-edu, DCLM, and custom mixes — to find what works. The paper provides no guidance for this selection beyond the Qwen-specific result.

  3. Interaction between the conversion protocol and the teacher’s training recipe. Qwen2.5 was trained with specific learning rate schedules, data mixtures, and architectural choices (e.g., the specific implementation of RoPE, the specific GQA head ratios). The RADLADS protocol’s learning rate choices (cosine to 1e-5 matching "the final learning rate the teacher saw during pretraining," Section 4.2) assume knowledge of and compatibility with the teacher’s training recipe. A different teacher trained with a different final LR, or with different optimizer settings, may require different step 1/2 LR configurations. The paper provides no ablation across teacher LRs or a method for determining the appropriate student LRs for an arbitrary teacher.

What evidence exists in the paper: The paper reports results exclusively for Qwen2.5 teachers at 7B, 32B, and 72B (Table 3). The ARWKV comparison (which also converts Qwen2.5-7B-Instruct, but to standard RWKV-7 rather than RAD-RWKV7) provides a same-teacher, different-architecture comparison that isolates the effect of architectural modifications and protocol details, but does not test cross-family generality.

Mitigation status: The paper partially acknowledges this (Section 3, dataset discussion; Section 9, "New varieties of model interactions will test these conversions in ways that we cannot predict"). The open-source code release enables community testing on other model families, but the paper itself provides no evidence that the protocol transfers. A single non-Qwen conversion (e.g., Llama-3.1-8B) with comparable benchmark results would have substantially strengthened the generality claim.

Limitation 3: Long-Context Capability Is Claimed but Not Evaluated

The assumption: Step 3 (context-length extension, Section 4.4) trains the student model on sequences of 16,384 tokens — a 32× increase over the 512-token sequences used in steps 1–2 — with the stated goal of "enhancing its long-context modeling capabilities." The paper’s framing implies that the converted model should function well at these extended context lengths.

The consequence: Not a single evaluation in the paper tests the model’s behavior at long context. All benchmark evaluations (LAMBADA, MMLU, ARC-C, ARC-E, HellaSwag, PIQA, Winogrande — Tables 1–6) are standard short-context tasks that do not require processing sequences longer than a few hundred tokens. A practitioner deploying QRWKV models for a long-context application (document QA, summarization of long texts, multi-turn dialogue with extensive history) has no evidence that the model performs competently at the 16K context length it was supposedly trained for — and certainly none that it can approach the Qwen2.5 teacher’s reported 128K context capability.

There are specific technical reasons to expect degraded long-context performance after conversion:

  • Cumulative decay dynamics. The recurrent state update $$\mathbf{wkv}_t = \mathrm{diag}(w_t) \cdot \mathbf{wkv}_{t-1} + \ldots$$ (RAD-RWKV6) applies per-timestep decay. The cumulative product $$\prod_{j=i}^{t-1} w_j$$ over 16K timesteps can be many orders of magnitude smaller than over 512 timesteps, even with the bounded decay ranges (RAD-RWKV6: $$w_t \in (e^{-5}, 1]$$; RAD-RWKV7: $$w_t \in (0.545, 1]$$). Whether 100M tokens of step 3 training (only ~6,100 sequences of 16K tokens) is sufficient to recalibrate the decay parameters for stable long-range retention is unverified.

  • State capacity limits. The recurrent state in RAD-RWKV6/7 has $$D^2/h$$ elements — a fixed capacity regardless of sequence length. At 7B with, e.g., $$D = 4096, h = 32$$, this is approximately 524K scalars per layer. Whether this fixed capacity can faithfully represent the information from 16K tokens of context (vs. 512 tokens in steps 1–2) without destructive interference from the recurrent updates is unknown.

  • No comparison to the teacher at long context. Even if step 3 produces a model that functions at 16K tokens, we do not know whether the converted model’s long-context quality degrades relative to the teacher more than its short-context quality does. The relative score metric — which shows preservation of short-context capabilities — may not extend to long-context behavior.

What evidence exists in the paper: None. Step 3’s effectiveness is asserted, not demonstrated. The alternative low-VRAM step 3a (freezing all weights except decay and tokenshift; Section 4.4) is described as an option but also not evaluated.

Mitigation status: Not addressed. The paper does not flag the absence of long-context evaluation as a limitation. Standard long-context evaluations (perplexity on long documents, needle-in-a-haystack retrieval accuracy, long-context QA benchmarks) are well-established in the literature and their omission is a significant gap for a method that includes an explicit context-extension step and whose primary practical motivation is efficient long-context inference.

Limitation 4: Difficulty Estimation Cost and Practical Deployment Overhead Are Unaccounted For

The assumption: The paper frames RADLADS as a recipe that a practitioner can follow to convert an existing transformer: pick a teacher, run the three steps, obtain a linear-attention model. The headline numbers — 350–700M tokens, under $2,000 for 72B — represent the marginal cost of one successful conversion run after all design choices are finalized.

The consequence: The cost of developing a working conversion for a new model family is not captured by the headline numbers, and the paper provides no principled method for a practitioner to determine the correct configuration without expensive trial and error. Specifically:

  • Architecture sensitivity. Section 3 describes significant experimentation to arrive at RAD-RWKV6 and RAD-RWKV7 — removing off-by-one decay, bonus, and (for RWKV-7) tokenshift; testing RoPE; evaluating gating at different ranks; testing state balancing vs. GroupNorm. A practitioner adapting RADLADS to a new target RNN architecture (e.g., Mamba-3, a new delta-rule variant) would need to replicate this architectural ablation process. Each ablation at 7B scale costs roughly 7 GPU-hours on 8× Mi300X (Table 8: ~0.75h step 1 + ~5.5h step 2 = ~6.25h for a partial conversion excluding step 3). A full sweep of even 10 architectural variants would cost ~500 GPU-hours — roughly $400–600 at cloud pricing, and requiring the same 8-GPU hardware — before arriving at a viable configuration.

  • Hyperparameter sensitivity. The comparison with ARWKV (Section 2) demonstrates that "specific choices for weight transfer, hyperparameters, dataset, and architecture matter significantly" — ARWKV, using the same general approach (Qwen2.5-7B-Instruct → RWKV-7) but different details, achieves MMLU relative score 0.801 vs. RADLADS’s 0.924. This 12.3 percentage-point gap from implementation details implies that a practitioner who makes slightly suboptimal hyperparameter choices (wrong LR schedule, wrong dataset, wrong token count per step) may obtain substantially worse results than the paper reports. The paper’s negative results (Section 8) describe failure modes qualitatively but provide no quantitative calibration — we do not know whether "much lower performance" from skipping step 1 means MMLU 0.60 or 0.40 or 0.20 absolute.

  • Dataset selection cost. The authors tried FineWeb, FineWeb-Edu, and custom datasets before settling on DCLM. Running even a 1B-token trial conversion on each candidate dataset to evaluate quality would cost several times the headline $2,000 figure. The paper’s hypothesis that dataset choice depends on the teacher’s pretraining distribution means that a practitioner converting a non-Qwen model cannot simply adopt DCLM with confidence — they must budget for their own dataset ablation.

What evidence exists in the paper: The negative results (Section 8) and the ARWKV comparison (Section 2, Tables 1 and 3) implicitly demonstrate the sensitivity of conversion quality to design choices. Table 8 provides timing estimates for one successful run per scale but does not account for development cost. The paper acknowledges (Section 9) that "each new architecture design requires meticulous testing to improve its compatibility with the RADLADS protocol" and that "the details of how the conversion process impacts reasoning models like our converted QwQ model is still unknown and requires further testing."

Mitigation status: Partially acknowledged. The paper is transparent that architecture and hyperparameter choices matter and shares negative results to guide practitioners. However, no principled method is provided for selecting these choices a priori — no proxy metric (beyond step 1 alignment loss, which requires running step 1) that can predict final downstream quality without a full conversion. The open-source code reduces the implementation burden but not the experimentation burden for new model families or architectures. Section 9 lists future work on "converting to and from different architectures and testing of more varieties of datasets against each of these" to "discover the principles for optimal conversion dataset design," acknowledging that these principles do not yet exist.

Limitation 5: No Evaluation of Conversion Impact on Instruction-Following, Safety, or Reasoning Behavior

The assumption: Converting a model through hidden-state alignment and logit distillation should preserve the teacher’s behavior broadly, since the objective functions (L2 on hidden states, KL on logits) penalize any deviation from the teacher’s output distribution. The paper uses instruction-tuned teachers (Qwen2.5-Instruct variants) and reports standard knowledge and reasoning benchmarks.

The consequence: The paper provides no evidence about whether conversion preserves or degrades capabilities that are critical for deployed language models but are not captured by the reported benchmarks. This includes:

  • Instruction-following and chat capability. The teachers are instruct-tuned models designed for conversational use. The conversion training uses DCLM — a web-text corpus — for all three steps, not instruction-formatted data or chat conversations. Step 2 minimizes KL divergence to the teacher’s logits on DCLM text, which may cause the student to regress toward a base-model-like next-token prediction distribution for web text, at the expense of the instruction-following behavior the teacher acquired during instruct tuning. The paper does not evaluate on instruction-following benchmarks (MT-Bench, AlpacaEval, IFEval) or compare chat quality between student and teacher.

  • Safety alignment. The Qwen2.5-Instruct teachers underwent safety training (RLHF or similar). Whether the distillation process preserves this alignment — or whether the DCLM training data re-exposes the model to harmful content distributions that safety training suppressed — is unknown. The paper includes no safety evaluations (harmfulness benchmarks, refusal rate measurements, red-teaming results).

  • Reasoning behavior for the QwQ variant. The paper includes one reasoning-focused model (QRWKV6-QwQ-32B, converted from Qwen2.5-QwQ-32B) and reports standard benchmarks (MMLU, ARC, etc.) for it. But QwQ’s distinctive capability is chain-of-thought reasoning with self-reflection and verification — a behavioral property not captured by accuracy on knowledge benchmarks. The paper explicitly acknowledges (Section 9) that "the details of how the conversion process impacts reasoning models like our converted QwQ model is still unknown and requires further testing." This is an honest disclosure but leaves a practitioner uncertain about whether the converted QwQ model retains the reasoning behaviors that distinguish it from the standard instruct model.

  • Calibration and uncertainty. KL divergence distillation can affect the model’s confidence calibration — the student may learn to match the teacher’s mean predictions while becoming overconfident or underconfident on tail distributions. The paper reports no calibration metrics (expected calibration error, reliability diagrams).

What evidence exists in the paper: None for instruction-following, safety, or calibration. The paper acknowledges the reasoning-model gap (Section 9) but does not evaluate it.

Mitigation status: Not addressed beyond the QwQ acknowledgment. A practitioner considering deploying QRWKV instruct models in a user-facing application has no data on whether they produce coherent conversations, follow system prompts, refuse harmful requests appropriately, or maintain the teacher’s reasoning style. This is a significant barrier to practical deployment, since instruction-following and safety are typically more important than marginal accuracy differences on knowledge benchmarks in production settings. The paper could have included even a small-scale qualitative evaluation (sample conversations comparing QRWKV-Instruct to Qwen2.5-Instruct on a set of prompts) or standard instruction-following benchmarks, but chose not to.

Limitation 6: Scale-Dependent Training Stability Limits Architectural Choice at Larger Sizes

The assumption: The RAD-RWKV7 architecture, which achieves the best 7B results (MMLU relative 0.924 vs. 0.871 for RAD-RWKV6; Table 3), should scale to larger model sizes with the same protocol, since the architectural mechanisms (delta-rule state update, removal/replacement keys) are not inherently size-limited.

The consequence: The paper reports that RAD-RWKV7 exhibits reduced training stability at larger parameter counts, preventing its use at 32B and 72B scales:

"RAD-RWKV7 appears to exhibit reduced training stability at larger (32B+) parameter and layer counts, and we are actively working to design architectural and training methodology changes to remedy this." (Section 9)

As a result, every model at 32B and 72B — including the headline 72B state-of-the-art result — uses RAD-RWKV6, not RAD-RWKV7. This means:

  • The best architecture at 7B does not scale. A practitioner wanting to convert a large model (32B+) cannot use the higher-performing RAD-RWKV7 design and must fall back to RAD-RWKV6, accepting the quality gap observed at 7B (MMLU relative 0.871 vs. 0.924) — and it is unknown whether this gap widens or narrows at larger scales.

  • The source of the instability is undiagnosed (in the paper). The delta-rule state update involves a matrix subtraction $$\mathrm{diag}(w_t) - \hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)$$ applied to the recurrent state $$\mathbf{wkv}_{t-1}$$ (Appendix B.2, Equation 31). This operation can, in principle, produce state matrices whose norms grow or oscillate if the removal term is poorly conditioned relative to the decay term, especially as head dimension $$D/h$$ increases with model size. The LayerNorm on $$p_t = \mathrm{LayerNorm}(r_t \mathbf{wkv}_t^T)$$ may provide insufficient stabilization if the underlying state dynamics become unstable. However, the paper provides no loss curves, gradient norm measurements, or diagnostic analyses of the instability — only the qualitative acknowledgment that it exists.

  • No intermediate-scale RAD-RWKV7 models. There are no RAD-RWKV7 conversions at 14B, 32B, or any scale between 7B and 72B, leaving it unclear at exactly what size the instability emerges and whether it is gradual or abrupt.

What evidence exists in the paper: Section 9 contains the single sentence quoted above. Table 3 shows RAD-RWKV7 only at 7B (QRWKV7-7B-Instruct and QRWKV7-7B-Instruct-from72B); all 32B and 72B models use RAD-RWKV6. No training curves, stability diagnostics, or attempted mitigations are reported.

Mitigation status: The paper states that the authors are "actively working to design architectural and training methodology changes to remedy this" (Section 9). This is a forward-looking acknowledgment, not a solution. A practitioner wanting to convert a 32B+ model today must either use RAD-RWKV6 (accepting lower quality) or independently solve a training stability problem that the authors — who have deep expertise with the RWKV architecture — have not yet resolved. The instability is a fundamental limitation of the current RAD-RWKV7 design at scale, and until addressed, it prevents the best-performing architecture from being used for the largest (and most practically valuable) conversions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that conversion from softmax-attention transformers to pure linear-attention decoders is not primarily a data problem but an architecture-compatibility problem. That is a genuine reframing of the conversion landscape, not merely an incremental refinement. Prior work — spanning four orders of magnitude in training token budgets from LOLCats's 40M tokens to SUPRA's 100B tokens — implicitly treated the target recurrent architecture as a fixed substrate that conversion must work around. The field's central question was "how many tokens do we need to distill a transformer into architecture X?" RADLADS inverts this: the question becomes "which architectural components in X actively help or hinder the conversion objective, and can we remove the hindering ones without sacrificing downstream quality?" The evidence for this reframing is the chain of architectural modifications — removing off-by-one decay and bonus from RWKV-6, adopting a GLA kernel, eliminating tokenshift from RWKV-7 — each motivated by step 1 alignment loss as a fast proxy for conversion quality rather than by pretraining considerations. The fact that these removals (which would likely hurt from-scratch pretraining) improve conversion outcomes demonstrates that conversion has its own architectural design principles, distinct from those of pretraining.

This reframing reconciles a tension that had implicitly structured the conversion literature: the apparent tradeoff between data efficiency and quality preservation. LOLCats achieved extreme data efficiency (40M tokens) but at the cost of pure-RNN quality so low (MMLU below random guessing) that hybrid SWA attention had to be reintroduced. SUPRA and DiJiang used 100B and 40B tokens respectively but still produced MMLU scores near or below random — large data volume did not solve the problem. MOHAWK's two-phase decomposition was a conceptual advance but still required billions of tokens. RADLADS's resolution is that the data-quality tradeoff is an artifact of poor architecture-protocol matching, not a fundamental constraint. When the target architecture is explicitly designed for conversion fidelity (RAD-RWKV7 with delta-rule selective forgetting, RAD-RWKV6 with GLA and balanced state), 350–700M tokens achieve quality that prior methods could not reach with 10–300× more data. The fact that ARWKV — using the same general approach (Qwen2.5-7B-Instruct → RWKV-7) but with standard architecture and different hyperparameters — achieves only 0.801 MMLU relative score vs. RADLADS's 0.924 (Table 3) directly demonstrates that protocol details, not just the overall approach, account for a large fraction of the quality. This is a concrete methodological lesson: conversion quality is brittle to seemingly small choices (learning rate schedule, dataset, which specific RWKV variant is used), and this brittleness explains why prior work with similar high-level strategies produced such variable results.

The paper also shifts the economic calculus for linear-attention model development. Prior to RADLADS, the path to a high-quality 70B-class linear-attention model was either (a) pretrain from scratch on multi-trillion-token corpora — accessible only to organizations with eight-figure training budgets — or (b) use a hybrid architecture that retains some softmax attention and therefore does not achieve O(1) inference. RADLADS demonstrates a third path: convert an existing open-weight 70B-class transformer for under $2,000. This decouples model scale from training budget, making large linear-attention models accessible to academic labs, startups, and individual researchers. The economic implication is not just cheaper models — it is that the bottleneck for linear-attention adoption shifts from pretraining cost to architectural innovation in the target RNN. If conversion is cheap, the limiting factor becomes how well an RNN architecture can approximate softmax attention during step 1 alignment. This redirects research investment: improving architectures for conversion fidelity (understanding the role of selective forgetting, gating at different ranks, state normalization strategies) becomes higher-leverage than improving pretraining recipes for from-scratch linear-attention models.

A secondary but practically significant reframing is the paper's implicit hypothesis about where knowledge resides in transformer models. The observation that de-novo initialization of QKVO weights produces "surprisingly reasonable performance" (Section 8) and that freezing MLPs during step 2 "significantly reduces model performance" together suggest a specific division of labor: factual knowledge is concentrated in MLPs and embeddings (which transfer directly and need only minor adaptation), while the sequence mixer's role is primarily computational — determining how stored knowledge is accessed and composed. This hypothesis, if it holds across model families, has implications beyond conversion: it suggests that model merging, architecture hybridization, and targeted finetuning can treat the sequence mixer and the MLP stack as semi-independent components, with the former being more amenable to architectural substitution than the latter. The paper does not prove this hypothesis — the evidence is correlational — but it provides a concrete empirical basis for further investigation.

The paper also establishes the three-step decomposition as a diagnostic framework. The negative results in Section 8 — skipping step 1 causes loss to plateau higher, freezing MLPs in step 2 degrades performance, switching datasets during step 3 produces qualitatively degraded generations — demonstrate that each step addresses a distinct failure mode that end-to-end distillation cannot compensate for. This modularity means future work can target specific bottlenecks (better long-context datasets for step 3, more expressive architectures for step 1, more effective KL distillation variants for step 2) without redesigning the entire pipeline. This is more than a recipe — it is a research infrastructure that lowers the barrier to entry for testing new RNN designs at scale, since step 1 alignment loss provides a rapid, cheap proxy for downstream conversion quality without requiring full end-to-end distillation for every candidate architecture.

Follow-Up Research This Work Enables

Characterize the step 1 alignment loss as a predictor of downstream conversion quality across architectures. The paper uses step 1 L2 loss qualitatively — RAD-RWKV7 "fits even closer and more rapidly" than RAD-RWKV6 — but never quantifies the relationship between step 1 convergence and final benchmark performance. A systematic study would convert the same teacher (e.g., Qwen2.5-7B) using a diverse set of recurrent architectures (vanilla linear attention, Mamba-2, RWKV-6, RWKV-7, Gated DeltaNet, xLSTM, and several ablated variants of each), record both the step 1 alignment loss curve and the final downstream benchmark scores after full conversion, and compute the correlation. This would establish whether step 1 loss is a reliable proxy for architecture selection, enabling future researchers to screen candidate architectures without running the full $2,000+ conversion pipeline. The paper's existing ablation data (Table 6) provides a starting point — the GroupNorm variant had worse MMLU but we don't know whether this was already visible in step 1 loss — but a controlled study with a dozen architectures and statistical rigor would transform the qualitative observation into a quantitative design tool.

Measure the actual inference throughput and memory savings of RAD-RWKV models vs. their softmax-attention teachers at practical sequence lengths. The paper's central motivation — O(1) per-token inference and elimination of the KV cache — is entirely unmeasured. A rigorous benchmarking study would compare QRWKV6-7B-Instruct, QRWKV7-7B-Instruct, QRWKV6-32B-Instruct, and QRWKV6-72B-Instruct against their Qwen2.5 teacher counterparts on identical hardware (e.g., a single A100-80GB or H100-80GB node), measuring tokens/second and peak GPU memory across sequence lengths from 512 to 32K tokens, at batch sizes of 1 (interactive latency) and 32 (throughput-optimized serving). The key question is not whether linear attention is asymptotically faster — it is — but whether the constant-factor overhead of ddlerp operations (RAD-RWKV6) and delta-rule state updates (RAD-RWKV7) erodes the advantage at the 512–16K sequence lengths where most practical deployments operate. The RAD-RWKV7 ablation removing tokenshift is claimed to "speed up training and inference" but no numbers are given. A proper benchmark would also compare against a well-optimized FlashAttention-3 implementation with KV caching to ensure the softmax baseline is state-of-the-art. The outcome determines whether RADLADS models are genuinely deployable as drop-in efficiency improvements or whether further kernel optimization is needed before the theoretical advantage materializes.

Stress-test the cross-family generality claim by converting Llama-3.1-8B-Instruct using the identical RADLADS protocol. The paper's results are exclusively on Qwen2.5 teachers. Converting Llama-3.1-8B-Instruct — a model with different pretraining data, different GQA head ratios, different RoPE frequencies, and a different training recipe — using the exact same RAD-RWKV7 architecture, DCLM dataset, and hyperparameters from Table 7, would directly test the protocol's robustness. If the conversion achieves comparable relative scores (MMLU relative above ~0.85), the generality claim is substantially strengthened. If benchmark scores are meaningfully worse than the Qwen conversion, the follow-up work would systematically ablate the differences: (a) use DCLM vs. FineWeb-Edu vs. a Llama-pretraining-matched dataset for steps 1–2; (b) adjust the final learning rate in step 1 to match Llama's reported pretraining final LR instead of Qwen's; (c) test whether Llama's specific GQA configuration (8 KV heads for 32 Q heads vs. Qwen2.5-7B's 4 KV heads for 28 Q heads) affects alignment quality. This would produce a principled guide for adapting RADLADS to new teacher families, rather than leaving practitioners to discover the necessary modifications through trial and error.

Diagnose and resolve the RAD-RWKV7 training instability at 32B+ scale. The paper acknowledges that RAD-RWKV7 — which achieves the best 7B results — "exhibits reduced training stability at larger (32B+) parameter and layer counts" (Section 9), preventing its use at the scales where it would be most valuable. A diagnostic study would instrument the RAD-RWKV7 conversion at, say, 14B (if stable) and 32B (where it becomes unstable), tracking per-layer metrics during step 1 and step 2: the norm of the recurrent state matrix $\mathbf{wkv}_t$, the eigenvalues of the removal operator $\hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)$, the gradient norms with respect to the delta-rule parameters $\xi$, $\alpha$, and the decay precursor network, and the evolution of the in-context learning rate $a_t$ distribution across sequence positions. The specific hypothesis to test is whether the removal term $\mathrm{diag}(w_t) - \hat{\kappa}^T_t (a_t \odot \hat{\kappa}_t)$ produces state matrices with growing or oscillating spectral norms at larger head dimensions — since $D/h$ increases with model size for typical head-count configurations. Potential mitigations to test: (a) scaling the removal term by $1/\sqrt{D/h}$, (b) adding a small identity regularization to the removal operator to ensure the effective decay stays sufficiently close to the diagonal, (c) using a learned temperature on the in-context learning rate $a_t$ that is initialized to produce weak removal and gradually increased during training, and (d) pretraining the RAD-RWKV7 blocks from scratch on the DCLM data before step 1 alignment so the state dynamics are in a stable regime when the teacher signal is introduced.

Evaluate whether RADLADS preserves instruction-following, safety alignment, and chain-of-thought reasoning behavior. The paper converts instruct-tuned models (Qwen2.5-Instruct) and a reasoning model (QwQ-32B) but evaluates only on knowledge and commonsense benchmarks that do not measure the behaviors that distinguish instruct models from base models. A behavioral evaluation would compare QRWKV7-7B-Instruct to Qwen2.5-7B-Instruct on: (a) MT-Bench and AlpacaEval for multi-turn instruction-following quality; (b) a standard safety benchmark (e.g., HarmBench or a set of refusal prompts) to measure whether conversion degrades refusal rates or introduces harmful completions — particularly relevant since steps 1–3 train on DCLM, a web corpus that may contain content the teacher's safety training suppressed; (c) for QRWKV6-QwQ-32B, a set of MATH or GSM8K problems evaluated with chain-of-thought prompting, comparing not just final accuracy but the presence and quality of self-verification, backtracking, and reflection behaviors that characterize the QwQ reasoning style. The paper explicitly acknowledges (Section 9) that the reasoning behavior of the converted QwQ model "is still unknown and requires further testing" — this is a direct invitation for such an evaluation. If safety alignment degrades, the follow-up would test whether including a small fraction of the teacher's instruct-tuning data (or synthetic teacher-generated safe responses) during step 2 recovers alignment without requiring a full post-conversion RLHF stage.

Test whether RADLADS can serve as an iterative architecture-improvement loop. The paper envisions enabling researchers to "test, train, and release models containing their new designs at scale" (Section 1), but this promise is only realized if the conversion protocol is fast and cheap enough to serve as an inner loop for architectural iteration. A concrete test: start with a candidate new RNN time-mixing block (e.g., a variant of Gated DeltaNet, a simplified Mamba-3, or an RWKV-7 with an alternative removal mechanism), convert Qwen2.5-7B-Instruct using the RADLADS protocol with this new block, evaluate on the standard benchmarks, and measure the total wall-clock time and compute cost. If the end-to-end cycle (architectural modification → code implementation → conversion → evaluation) can be completed in under 24 hours and under $200 (the approximate cost of one 7B conversion from Table 8), then RADLADS genuinely enables rapid architectural prototyping. If overhead from implementation, debugging, and hyperparameter tuning pushes the cycle to weeks, then the protocol is better suited for production conversions than for research iteration. The study would also measure how well step 1 alignment loss correlates with final benchmark performance across these new architectures, validating (or refuting) step 1 loss as a cheap proxy for architecture screening.

Practical Applications and Downstream Use Cases

Cost-efficient long-context document processing. An organization that currently uses Qwen2.5-72B-Instruct via an API for document summarization, legal contract review, or scientific literature synthesis — workloads that routinely process 4K–32K token contexts — can convert the model to QRWKV6-72B-Instruct for under 2,000anddeployitontheirowninfrastructure.At4Kcontext,theO(1)per−tokencomputationeliminatesthequadraticscalingofattention,andat32KcontextthememorysavingsfromremovingtheKVcache(whichfora72Bmodelat32Ktokenswith8−bitKVcachequantizationwouldrequireroughly2–4GBofGPUmemory)becomesubstantial.Thepaper′sbenchmarkresults(Table3)showthatQRWKV6−72B−InstructachievesrelativeMMLU0.899andLAMBADA1.004—within102,000 and deploy it on their own infrastructure. At 4K context, the O(1) per-token computation eliminates the quadratic scaling of attention, and at 32K context the memory savings from removing the KV cache (which for a 72B model at 32K tokens with 8-bit KV cache quantization would require roughly 2–4 GB of GPU memory) become substantial. The paper's benchmark results (Table 3) show that QRWKV6-72B-Instruct achieves relative MMLU 0.899 and LAMBADA 1.004 — within 10% of the teacher on factual knowledge and matching on language modeling. A deployment could route queries below a difficulty threshold to the converted model and escalate challenging queries to the full teacher, achieving a blended cost-latency profile without a single point of quality failure. The critical missing piece — as discussed in Limitations — is the unevaluated inference throughput and long-context quality; a practitioner would need to benchmark these before committing, but the conversion cost (2,000) is low enough to make the evaluation itself feasible.

On-device or edge deployment of 7B-class instruct models. A mobile or edge application that currently cannot run a 7B transformer due to KV cache memory constraints (growing with context length) can deploy QRWKV7-7B-Instruct, which has constant memory per token regardless of conversation history length. For a multi-turn dialogue application where context grows to 4K–8K tokens over the course of a conversation, the softmax-attention model's KV cache might require several hundred MB of GPU or system memory that exceeds edge device budgets, while the recurrent model's fixed-size state (approximately 524K scalars per layer at 7B with typical heads, totaling a few hundred MB for the entire model state across all layers, independent of conversation length) fits within a tight memory envelope. The paper's benchmark results show QRWKV7-7B-Instruct at MMLU 0.682 absolute vs. the teacher's 0.717 — a drop of only 3.5 percentage points — suggesting the quality loss is acceptable for many consumer applications. The unevaluated inference latency is the main risk: if the ddlerp and delta-rule operations in RAD-RWKV7 are slower per-token than a well-optimized FlashAttention kernel at the 512–2K token lengths typical of early dialogue turns, the constant-memory advantage may be offset by higher latency for short contexts.

Training data generation for self-improvement pipelines at scale. A research group developing a self-improvement loop (e.g., STaR, ReST^EM, or rejection-sampling fine-tuning) that generates training data from a 70B-class teacher model can convert the teacher to a linear-attention variant and use it as a cheaper data generator. In a typical pipeline, the teacher model generates multiple candidate solutions per training example, which are then scored and filtered. The generation phase dominates the compute cost. If QRWKV6-72B-Instruct achieves, say, 2× higher throughput than Qwen2.5-72B-Instruct at the 2K–4K generation lengths common in math and coding tasks, the cost of generating millions of training examples is halved. The benchmark results indicate that the converted model's output quality is very close to the teacher's (relative scores of 0.90–1.00 across most benchmarks), suggesting the generated data would be of comparable utility for downstream training. The risk — again, the unevaluated inference throughput — is the key unknown; a practitioner would run a small pilot comparing generation throughput, output diversity, and downstream student model quality between data generated by Qwen2.5-72B and QRWKV6-72B before committing to the full pipeline.

Architecture research for compressive-state models at meaningful scale. An academic lab with a budget of a few thousand dollars and access to 8× high-memory GPUs can now test a novel recurrent time-mixing block at the 7B scale — converting from Qwen2.5-7B — and evaluate it on standard benchmarks within a week. Before RADLADS, testing a new RNN design at 7B required either a full pretraining run (millions of dollars, infeasible) or a smaller-scale proxy evaluation (e.g., training a 300M-parameter model from scratch) whose findings might not transfer to the 7B regime. The conversion approach provides a direct evaluation: does the new architecture achieve lower step 1 alignment loss than RAD-RWKV7? Does it converge faster? Do the resulting downstream benchmark scores match or exceed the RADLADS baselines? The open-source code release and the detailed hyperparameters in Table 7 lower the implementation barrier so that a researcher can focus on the architectural innovation — designing the time-mixing equations — rather than on building a full training pipeline. The main practical constraint is GPU access: 8× high-memory GPUs (Mi300X, A100-80GB, or H100-80GB) are still a significant resource, but cloud rental for a week (~$500–1,500) is within the budget of many academic grants, in contrast to the months-long, million-dollar training runs that from-scratch pretraining requires.