Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

ArXiv: 2510.19338

🎯 Pitch

A hybrid linear-plus-softmax attention model cuts long-context inference cost to just one-tenth that of a comparable dense Transformer, while achieving state-of-the-art reasoning scores. The key is a systematic search for the optimal block ratio of linear to standard attention layers, combined with meticulously aligning training and inference operators to enable stable reinforcement learning without probability recomputation.

---

1. Executive Summary

This technical report introduces the Ring-linear model series — Ring-mini-linear-2.0 (16B total, 957M non-embedding active parameters) and Ring-flash-linear-2.0 (104B total, 6.1B non-embedding active parameters) — which adopt a hybrid architecture integrating linear attention and softmax attention in a grouped layer pattern (M linear attention blocks followed by one Grouped Query Attention block per layer group) to drastically reduce inference I/O and computational overhead in long-context scenarios. Through systematic scaling-law experiments identifying an optimal hybrid attention ratio and layer group size, combined with a self-developed FP8 kernel library (linghe) delivering 50–77% training throughput improvements and fused linear attention kernels enabling >2.5× prefill throughput and >10× decode throughput over softmax-attention baselines at sequence lengths beyond 64K, the series reduces inference cost to roughly 1/10 that of a 32B dense model and over 50% compared to the prior Ring series. The paper further establishes that systematic training-inference alignment — correcting numerical precision, nondeterminism, and implementation discrepancies across RMSNorm, RoPE, KV cache, attention backends, and MoE routing between training and inference engines — eliminates the need for recomputing training probabilities during reinforcement learning, enabling long-horizon stable RL training that directly uses rollout probabilities in PPO clipping and achieves consistent reward growth on challenging reasoning benchmarks, establishing that training stability for long-output MoE models depends on operator-level alignment rather than algorithmic mitigation alone.

2. Context and Motivation

The Core Problem: Attention Is the Bottleneck for Long-Context Reasoning at Scale

The paper addresses a fundamental tension in modern LLM deployment: test-time scaling and long-context support are both essential for reasoning models, but the attention mechanism that enables contextual understanding simultaneously creates resource bottlenecks that become severe as sequence length grows.

To understand why this matters, consider what happens when a reasoning model — like those powering an AI assistant or a coding agent — processes a query. During the "thinking" phase, the model may generate thousands of tokens of internal reasoning (chain-of-thought) before producing a final answer. Each token the model generates must attend to all preceding tokens through the attention mechanism. In traditional softmax attention (the standard Transformer attention from Vaswani et al., 2017), this means:

• Computational complexity scales quadratically with sequence length: $O(n^2 d)$, where $n$ is the number of tokens and $d$ is the attention head dimension. Processing an 8K-token context is not just twice as expensive as 4K — it is four times as expensive, because every token computes attention weights over every other token.

• KV cache storage scales linearly with sequence length. During autoregressive decoding, every previously generated key and value vector must be stored in memory and read back at each generation step. At 128K context with a large model, the KV cache alone can consume tens of gigabytes of GPU memory.

• I/O overhead scales linearly with output length. Each decoding step requires reading the entire KV cache from GPU memory to compute attention — an I/O-bound operation on modern accelerators whose memory bandwidth limits throughput far more than raw compute.

These constraints are not theoretical: they directly limit how long a context a model can handle before inference becomes impractically slow or memory-prohibitive, and they impose a hard ceiling on the length of chain-of-thought reasoning that is feasible during test-time scaling.

The problem is exacerbated by two concurrent trends that the paper identifies:

Trend 1: Test-time scaling demands longer outputs. Models in the OpenAI O-series, DeepSeek R1 (Guo et al., 2025), Gemini 2.5 (Comanici et al., 2025), and Qwen Thinking series achieve stronger reasoning by generating longer chain-of-thought sequences — sometimes thousands or tens of thousands of tokens per query. As Section 1 notes, "breakthroughs in reasoning models have also contributed to improvements in non-reasoning tasks." But each additional reasoning token compounds the quadratic attention cost and linear KV-cache growth. The very mechanism that makes these models better (longer reasoning chains) makes them increasingly expensive to run.

Trend 2: Long-context support is a deployment requirement. Applications like agent systems (where the model must maintain state over many interaction steps), code generation (where large codebases must fit in context), and document analysis demand context lengths of 32K, 128K, or more. The paper states plainly: "growing demands for long-context support in core applications such as Agent systems and code generation have made extended context capability a critical requirement for real-world model deployment."

The collision of these two trends — longer outputs from reasoning models, longer inputs from real-world applications — means that attention mechanism efficiency is no longer a nice-to-have optimization but a blocking constraint on deploying capable reasoning models.

Prior Approaches and Where They Fall Short

The paper identifies three categories of prior work that address the attention bottleneck, each with specific limitations:

Softmax Attention Variants (GQA, MQA, MLA)

The dominant approach to improving attention efficiency has been to reduce the KV cache footprint by sharing key-value heads across query heads:

• Multi-Query Attention (MQA) (Shazeer et al., 2017) collapses all KV heads to a single head, dramatically reducing KV cache size but at the cost of model quality — the shared KV representation is too constrained for many tasks.

• Grouped Query Attention (GQA) (Ainslie et al., 2023) is the current standard compromise: it groups query heads into clusters, each sharing a single KV head. This balances cache reduction with quality, and is used in models like Llama 3 and the paper's own softmax attention blocks.

• Multi-head Latent Attention (MLA) (Liu et al., 2024a) compresses the KV cache into a low-rank latent space, further reducing cache size while attempting to preserve representational capacity. It is used in DeepSeek-V2/V3 and has shown strong results.

However, the paper notes a fundamental limitation: all of these variants still have quadratic computational complexity and linear KV cache growth with sequence length. GQA might reduce the constant factor — fewer KV heads means smaller cache — but the scaling behavior with sequence length remains fundamentally $O(n^2)$ for computation and $O(n)$ for storage. As Figure 4 demonstrates, at sequence lengths beyond 8K, even optimized softmax attention variants see KV cache memory access grow linearly and unboundedly. These approaches buy time but do not solve the asymptotic scaling problem. For context lengths of 64K or 128K — which are increasingly common in deployment — the overhead remains substantial.

Pure Linear Attention Models

Linear attention mechanisms (Retnet from Sun et al., 2023; Lightning Attention from Qin et al., 2023; Mamba from Gu and Dao, 2024; Gated Linear Attention from Yang et al., 2023; DeltaNet from Yang et al., 2024) take a fundamentally different approach. Instead of computing pairwise attention weights between all tokens (the softmax $QK^T$), they reformulate attention as:

$O = Q(K^T V)$

Where $Q$ and $K$ now interact through a matrix product that can be computed sequentially, yielding:

• Computational complexity of $O(n d^2)$ instead of $O(n^2 d)$. When sequence length $n$ is much larger than head dimension $d$, this is a dramatic improvement — it scales linearly with $n$ rather than quadratically. The theoretical efficiency gain at long contexts is enormous.

• Constant KV state memory. Instead of storing a growing cache of key-value vectors, linear attention maintains a fixed-size recurrent state matrix $kv_t \in \mathbb{R}^{d \times d}$ that is updated at each step (Equation 3–4). The storage requirement stays constant regardless of how many tokens are processed.

This solves the asymptotic scaling problem, but the paper identifies critical practical limitations:

"Pure linear language models often underperform in industrial-scale scenarios, particularly as model parameter counts and sequence lengths increase."

Specifically, the paper notes that linear attention underperforms on retrieval tasks compared to softmax attention. This is a significant weakness because retrieval — finding specific information from earlier in the context — is a core capability for long-context applications. If a model cannot reliably locate a function definition in a codebase or a key fact in a document, the computational savings are moot.

Additionally, the paper makes an important economic observation about pre-training:

"Although pure Linear Attention offers lower theoretical computational cost, its advantages only become pronounced at sequence lengths beyond 8K. However, the mainstream context length during pre-training typically remains in the 4K–8K range."

This means that during the pre-training phase — which consumes the majority of total compute in an LLM's lifecycle — the theoretical efficiency advantage of linear attention is not fully realized because typical training sequences are too short for the asymptotic benefits to dominate. The quadratic softmax attention at 4K–8K may actually be faster in practice due to highly optimized FlashAttention kernels (Dao et al., 2022) that make hardware-efficient use of GPU compute for moderate-length sequences.

Mixture-of-Experts (MoE) Architectures Compound the Challenge

The paper further notes a practical complication: the growing adoption of MoE architectures (which Ring-linear itself uses). MoE models route tokens to a subset of expert FFN layers rather than processing all tokens through all FFN parameters. This makes the FFN computation more efficient — particularly for large models — but:

"with the growing adoption of Mixture-of-Experts (MoE) architectures—which account for a high proportion of computations at lengths below 8K—the efficiency gains from Linear Attention during pre-training are further constrained."

In other words, in an MoE model at typical pre-training sequence lengths, the FFN layers (fed through MoE) may dominate total compute, meaning that even if attention were free, the overall training speedup from switching to linear attention would be limited. The benefits of linear attention become proportionally larger as attention starts to dominate — which happens at longer sequence lengths that are not typical in pre-training but are common in inference and RL.

The Hybrid Architecture Gap

A natural solution has emerged in the field: hybrid architectures that combine linear attention for efficiency with softmax attention for retrieval and expressive power. The paper cites several examples:

• Minimax M1 (Chen et al., 2025)
• GPT-OSS (Agarwal et al., 2025)
• Qwen3-Next

These models interleave softmax attention layers with linear attention layers, or use linear attention in most layers while reserving a few for softmax. The intuition is that softmax attention provides global retrieval capability — the ability to pull information from arbitrary positions in context — while linear attention handles the bulk of local and sequential processing at dramatically lower cost.

The paper positions itself within this hybrid paradigm, but identifies a specific gap: there is no established understanding of the optimal ratio and arrangement of softmax to linear attention layers. How many linear attention layers should be grouped before one softmax attention layer is inserted? How does this ratio affect scaling behavior at different model sizes and compute budgets? Prior hybrid architectures made design choices ad-hoc, without systematic study.

Furthermore, the paper identifies that hybrid linear models face unique systems challenges that are not addressed by existing infrastructure:

• Fragmented kernel implementations for linear attention during inference. As Section 3.4 notes, "the decoding kernel is often fragmented into multiple operations, which further diminishes overall efficiency." Existing linear attention implementations use 2–4 separate kernels for the prefill stage, incurring high launch overhead and redundant memory traffic.

• Lack of support in standard inference frameworks. The paper states that "practical implementations have been limited by the absence of support for advanced inference solutions such as SGLang and the vLLM V1 framework." Without native support, the theoretical efficiency of linear attention cannot be realized in production.

• No speculative decoding for hybrid linear models. Standard speculative decoding uses tree-structured attention masks that existing linear attention kernels cannot handle, preventing a key inference optimization from being applied to these models.

The Reinforcement Learning Stability Gap

Beyond architecture efficiency, the paper identifies a deeply important problem that has been poorly understood: training-inference disparity in RL for long-output MoE models causes training instability.

The standard RL training pipeline for language models involves two phases:

1. Rollout (inference): The model generates completions using an inference engine (e.g., vLLM, SGLang), producing token sequences and their associated probabilities.
2. Update (training): The training engine (e.g., Megatron, FSDP) recomputes the probabilities for those same sequences and applies policy gradient updates (typically PPO).

PPO's clipped objective relies on the importance sampling ratio between the current policy and the behavior policy that generated the rollout (Equation 5 in the paper). Ideally:

$\nabla_\theta J(\theta) = \mathbb{E}_{x \sim \pi_{\text{rollout}}} \left[ \nabla_\theta \min\left(\frac{\pi_{\text{training}}(x, \theta)}{\pi_{\text{rollout}}(x, \theta_{\text{old}})}\hat{A}, \text{clip}\left(\dots\right)\hat{A}\right) \right]$

The key requirement is that $\pi_{\text{rollout}}$ — the probability distribution from the inference engine — is the same as what the training engine would compute for the same tokens. If they differ, the importance sampling ratio is incorrect, and the gradient update is biased toward a policy different from the one actually generating data. This violates the on-policy assumption and can lead to rapid degradation, often called "training collapse."

The paper reveals that this identity is violated in practice, and for deep reasons that compound in long-output MoE models:

"Even standard components in large language models, such as RMSNorm and RoPE, exhibit non-negligible implementation discrepancies across common training (e.g., Megatron, FSDP) and inference (e.g., vLLM, SGLang) frameworks."

These are not bugs in the traditional sense — both implementations may be "correct" — but small numerical differences accumulate layer by layer through the deep network. The paper reports that:

"In extreme cases, the output probability for the same token can be 0 during training and 1 during inference."

When a training framework computes $\pi_{\text{training}}(\text{token}) = 0$ but the rollout engine recorded $\pi_{\text{rollout}}(\text{token}) = 1$, the importance sampling ratio becomes infinite, the PPO clip provides no protection (since the ratio is outside the clipping range), and the gradient becomes nonsensical. This is catastrophic for learning.

Two architectural features make this problem especially acute:

• MoE routing: The expert assignment in MoE layers uses a softmax over router logits followed by top-k selection. Small numerical differences in the logits (from RMSNorm precision, RoPE computation, etc.) can cause tokens to be routed to different experts in training vs. inference. Once experts diverge, the entire computation path diverges, and the output distribution becomes completely incomparable.

• Long chain-of-thought: The error compounds with sequence length. For short generations, the probability discrepancy might be tolerable. For reasoning models generating thousands of tokens, the cumulative error ensures that $\pi_{\text{training}}$ and $\pi_{\text{rollout}}$ become effectively decorrelated.

Prior work (Zheng et al., 2025; Yao et al., 2025) had recognized this problem but addressed it through algorithmic mitigation — modifying the PPO objective or recomputation strategies to be more tolerant of discrepancy. The standard practice in frameworks like verl (Sheng et al., 2024) and OpenRLHF (Hu et al., 2024a) is to simply re-forward the rollout data through the training engine and use the recomputed probabilities for importance sampling (Equation 6 in the paper), effectively ignoring the rollout probabilities entirely:

$\nabla_\theta J'(\theta) = \mathbb{E}_{x \sim \pi_{\text{rollout}}} \left[ \nabla_\theta \min\left(\frac{\pi_{\text{training}}(x, \theta)}{\pi_{\text{training}}(x, \theta_{\text{old}})}\hat{A}, \dots\right) \right]$

The paper argues this is inherently biased because it replaces the true behavior distribution $\pi_{\text{rollout}}$ with an approximation $\pi_{\text{training}}(x, \theta_{\text{old}})$ that is computed under different numerical conditions. The approximation may be close when discrepancies are small, but it fundamentally cannot track by how much the rollout distribution actually differed from the current policy.

Moreover, recomputing training probabilities from scratch doubles the forward-pass cost of RL training, since every rollout token must be processed once by the inference engine and once by the training engine. For long reasoning chains, this is a substantial computational overhead.

How This Paper Positions Itself

The paper's positioning is multi-faceted, addressing the architecture gap, the systems gap, and the training stability gap:

On architecture: Rather than proposing a fundamentally new attention mechanism, the paper conducts a systematic empirical study of hybrid linear-softmax architecture ratios using scaling law methodology (Chinchilla-style power-law fitting from Hoffmann et al., 2022). By training multiple model variants with different layer group sizes (the number of linear attention layers $M$ before each softmax attention layer) and fitting loss-vs-FLOPs curves (Figure 3), the paper establishes evidence-based guidelines: $M=7$ for the 104B model, $M=4$ for the 16B model. This is positioned as filling the gap left by prior hybrid models that made these choices heuristically.

The paper also reports extensive ablation experiments on the internal design of linear attention blocks — grouped RMSNorm to avoid all-reduce communication, partial RoPE application (half dimensions), and head-wise decay coefficients — establishing that seemingly minor implementation choices (e.g., power-law decay vs. linear decay for the hidden state coefficient) can shift training loss by ~0.04, which is a meaningful difference in the Chinchilla scaling regime.

On systems: The paper develops what it calls "the first linear attention kernel that supports tree masks" for speculative decoding, and integrates optimized fused linear attention kernels into both SGLang and vLLM frameworks (through the linghe kernel library and Flood inference framework). This directly addresses the prior gap where "practical implementations have been limited by the absence of support for advanced inference solutions." The systematic comparison of throughput across sequence lengths from 4K to 512K (Figures 7–8) provides concrete evidence that the theoretical efficiency of linear attention can be realized in practice once appropriate engineering is done — but only for context lengths beyond 8K, consistent with the pre-training limitation observation.

On training stability: The paper takes a fundamentally different approach from prior work. Rather than accepting training-inference discrepancy as inevitable and mitigating it algorithmically, the paper argues for systematically eliminating the discrepancy at the operator level:

"We not only improve RL stability from an algorithmic perspective but also devote substantial effort to systematically addressing the training-inference disparity, aiming to fundamentally solve the problem."

This involves a three-stage alignment process (prefill-prefill, prefill-decode, different parallelism configurations) that identifies and fixes precision mismatches, nondeterministic operations, and implementation inconsistencies across every module: KV cache precision (FP32 accumulation required for linear attention's recurrent state), LM Head softmax precision, RMSNorm epsilon values and residual computation, RoPE minor implementation differences, FlashAttention backend consistency between prefill and decode, and MoE routing determinism (replacing non-stable torch.topk with a stable implementation, fixing token permutation order). Figure 10 shows an ablation experiment where each aligned module contributes incrementally to RL training stability, and Figure 11 visualizes the dramatic correction in token probability distributions after KV cache precision is fixed.

Once alignment is achieved, the paper shows that directly using rollout probabilities for PPO clipping (Equation 5) — the theoretically correct formulation — not only works but outperforms the recomputed-training-probability approach (Equation 6) in later training stages, with the training-inference probability disparity remaining in a stable low range (Figure 12). This eliminates the need for costly re-forwarding and establishes that training stability in long-horizon RL is fundamentally an engineering discipline problem rather than one requiring algorithmic innovation.

Overall framing: The paper positions the Ring-linear series not as pushing the theoretical frontier of attention mechanisms or RL algorithms, but as demonstrating that systematic engineering across the full stack — architecture design guided by scaling laws, optimized kernels for both training and inference, and operator-level training-inference alignment — can produce models that are simultaneously more efficient at long contexts and more stable during RL training than prior approaches, achieving state-of-the-art reasoning performance (Section 6, Tables 2–3) while reducing inference costs to a fraction of comparable dense models.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

The paper presents the Ring-linear model series, which is a family of large language models built on a hybrid attention architecture — most layers use efficient linear attention (constant memory, linear compute), but every few layers a standard softmax attention block is inserted to preserve retrieval capability — all wrapped in a highly sparse Mixture-of-Experts framework and supported by custom FP8 kernels and a systematic training-inference alignment methodology for stable reinforcement learning. The system solves the problem that softmax attention becomes prohibitively expensive at long sequence lengths (quadratic compute, linear KV cache growth), while pure linear attention underperforms on retrieval tasks, by empirically determining the optimal ratio of linear-to-softmax attention layers through scaling law experiments, then engineering the full infrastructure stack — from fused GPU kernels to RL training stability — to realize those efficiency gains in practice.

3.2 Big-picture architecture (diagram in words)

The Ring-linear model can be understood as five interacting components:

1. Token embedding layer — converts input text tokens into continuous vector representations. Standard embedding lookup table with vocabulary size of 157K.

2. Layer groups (the hybrid attention core) — the model is divided into repeating groups. Each group contains M linear attention blocks followed by one Grouped Query Attention (GQA) block. This is the central architectural innovation: linear attention provides efficiency (constant memory, $O(nd^2)$ compute), while the periodic GQA block provides retrieval capability. Ring-mini-linear-2.0 uses $M = 4$ (group size 5), Ring-flash-linear-2.0 uses $M = 7$ (group size 8).

3. Mixture-of-Experts (MoE) feed-forward layers — each attention block (both linear and GQA) is followed by an MoE layer with 256 experts, routing each token to 8 experts (ntop_k = 8) plus one shared expert via sigmoid routing with auxiliary-loss-free load balancing. The first block uses a dense MLP instead of MoE. The overall activation ratio is approximately 1/32 of total parameters.

4. Output projection and prediction heads — final RMSNorm, then a linear output layer projecting to vocabulary size. Training uses both standard next-token prediction and Multi-Token Prediction (MTP) objectives.

5. Training and inference infrastructure — custom FP8 kernel library (linghe) providing fused quantization, linear attention, MoE routing, and normalization operations, integrated into Megatron for training and SGLang/vLLM for inference. Also includes a systematic module-by-module alignment procedure to eliminate numerical discrepancies between training and inference engines during RL.

Information flows as follows: tokenized text → embedding lookup → repeated layer groups (M × [RMSNorm → Linear Attention → RMSNorm → MoE] then 1 × [RMSNorm → GQA → RMSNorm → MoE]), with first block using dense FFN → final RMSNorm → output projection → next-token prediction (and MTP).

3.3 Roadmap for the deep dive

• First, the hybrid linear attention mechanism — how linear attention reduces compute from $O(n^2 d)$ to $O(nd^2)$ through re-ordering matrix operations, the recurrent formulation that enables constant KV state memory, and the role of the fixed decay coefficient — because this is the core efficiency innovation that makes everything else possible and determines the scaling behavior.
• Second, the hybrid architecture design — how layer groups combine linear and softmax attention, the scaling law experiments that determined the optimal group size $M$, and the key ablation decisions (Grouped RMSNorm, Partial RoPE, head-wise decay type) — because these architectural choices directly determine the efficiency-performance tradeoff.
• Third, the KV cache cost analysis — the quantitative comparison showing how hybrid linear attention's constant-state memory diverges from GQA and MLA's linear cache growth at scale — because this explains why the architecture matters for practical deployment at long context lengths.
• Fourth, the FP8 training optimization pipeline — kernel fusion strategy (linear gates, MoE routing, QK norm, quantization fusion, state-aware recomputation) and the resulting throughput improvements (77% for mini, 57% for flash) — because these are the engineering contributions that make the theoretical efficiency realizable.
• Fifth, the inference optimization pipeline — fused linear attention kernels for prefill and decode, tree-mask-compatible speculative decoding, and integration into SGLang/vLLM — because inference efficiency is where the architecture's advantages are most visible.
• Sixth, the systematic training-inference alignment for RL — the module-by-module precision and determinism fixes that eliminate probability discrepancies, and why this enables directly using rollout probabilities in PPO — because this is the paper's most distinctive methodological contribution beyond architecture design.

3.4 Detailed, sentence-based technical breakdown

This is primarily an empirical systems paper whose core idea is that the optimal hybrid linear-softmax attention architecture can be determined through scaling law experiments, and that realizing its theoretical efficiency requires co-designed kernel optimization and operator-level training-inference alignment.

---

Linear Attention Mechanism

The Ring-linear models use a linear attention mechanism with fixed decay, building on the formulations from RetNet (Sun et al., 2023) and Lightning Attention (Qin et al., 2023). The key insight is to bypass the quadratic $QK^T$ pairwise similarity matrix entirely.

The core reformulation. Standard softmax attention computes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right) V$

The softmax requires computing all $n^2$ pairwise similarities before multiplying by $V$, giving $O(n^2 d)$ complexity. Linear attention instead reorders the multiplication:

$O = Q(K^T V)$

where $Q, K, V \in \mathbb{R}^{n \times d}$ are the query, key, and value matrices respectively, $n$ is the sequence length, and $d$ is the attention head dimension (or feature dimension in the head).

What it computes: Instead of computing an $n \times n$ attention weight matrix and using it to aggregate values, this form first multiplies $K^T$ (shape $d \times n$) with $V$ (shape $n \times d$) to produce an intermediate $d \times d$ matrix — a fixed-size "memory" or "state" that summarizes all key-value pairs. Then $Q$ (shape $n \times d$) multiplies this state to produce the output $O$ (shape $n \times d$). The critical property is that the $K^T V$ multiplication costs $O(nd^2)$ rather than $O(n^2 d)$, because it is performing $d$ outer products of key-value vector pairs rather than $n^2$ pairwise dot products.

Why this form: The reordering exploits the associativity of matrix multiplication: $(QK^T)V = Q(K^T V)$. In the softmax case, this reordering is blocked by the non-linear softmax operation, which must be applied element-wise to the full $QK^T$ matrix. By removing the softmax, the computation can be restructured so that the dependence on $n$ becomes linear rather than quadratic. When $n \gg d$ — which is the case at long context lengths — this produces orders of magnitude fewer FLOPs. The tradeoff is that each token's attention is now computed through this global $d \times d$ state rather than through direct pairwise comparisons, which limits the model's ability to do fine-grained retrieval (finding a specific earlier token) but preserves its ability to aggregate information across the sequence.

The recurrent formulation with decay. In the Lightning Attention implementation, the computation is further structured to include a fixed decay coefficient that gives the linear attention a recency bias. The output for the $t$-th token is:

$o_t = q_t \sum_{s \leq t} \lambda^{t-s} k_s^T v_s$

where $q_t \in \mathbb{R}^d$ is the query vector at position $t$, $k_s, v_s \in \mathbb{R}^d$ are the key and value vectors at position $s$, and $\lambda \in (0, 1]$ is a fixed decay factor (closer to 1 means older tokens contribute more, closer to 0 means only recent tokens matter).

What it computes: For each new token $t$, this equation computes a weighted sum of all previous key-value outer products $k_s^T v_s$ (each a $d \times d$ matrix), where the weight decays exponentially with temporal distance: a token $s$ receives weight $\lambda^{t-s}$. The query $q_t$ then reads from this weighted sum to produce $o_t$. The decay factor $\lambda$ controls the effective memory horizon — with $\lambda = 1$, all tokens are weighted equally (truly uniform attention); with $\lambda < 1$, the model has a bias toward recent context, which is often beneficial for language modeling since nearby tokens are typically more relevant.

Why this form: The decay mechanism is the simplest way to introduce positional information into the otherwise position-agnostic $Q(K^T V)$ formulation. Without it, the model treats all tokens in the sequence identically — the same $K^T V$ summary would be computed regardless of token order. The exponential decay provides an inductive bias that recent context matters more, which aligns with the statistical structure of natural language. It is "fixed" (not learned) for efficiency — a learned decay would require materializing per-position coefficients or adding parameters, undermining the simplicity of the linear formulation.

The recurrent state update. This can be rewritten in a recurrent form that makes the constant memory property explicit:

$kv_0 = 0 \in \mathbb{R}^{d \times d}$

$kv_t = \lambda \cdot kv_{t-1} + k_t^T v_t$

$o_t = q_t(kv_t)$

where $kv_t \in \mathbb{R}^{d \times d}$ is the accumulated key-value state at position $t$, defined as:

$kv_t = \sum_{s \leq t} \lambda^{t-s} k_s^T v_s$

What it computes: The state matrix $kv_t$ is updated at each step by: (1) decaying the previous state by $\lambda$ (down-weighting all prior tokens), (2) adding the new token's contribution $k_t^T v_t$ (an outer product yielding a $d \times d$ matrix). The output $o_t$ is then simply $q_t$ multiplied by this state. The entire history of the sequence is compressed into a single $d \times d$ matrix — its size does not grow with sequence length.

Why this form: This is the key efficiency property for inference. In standard softmax attention, the KV cache grows as $n \times d$ (storing every key and value vector). At 128K context with $d = 128$ per head and 32 heads, that is $128\text{K} \times 128 \times 32 \times 2 = 1.05$ billion floats (over 2 GB in FP16). In linear attention, the state is $d \times d = 16,384$ floats per head, or $32 \times 16,384 = 524,288$ floats total — roughly 1 MB. This is a 2000× reduction and, crucially, it is constant: generating the 128,001st token costs the same memory access as generating the 129th token. The tradeoff is that the $d \times d$ state is a lossy compression — it can represent only $d^2$ degrees of freedom, whereas the full KV cache can represent $nd$ — but for many language modeling purposes, this compressed representation is sufficient.

---

Hybrid Architecture Design: Layer Groups and Optimal Ratio

The paper does not use pure linear attention. Instead, it adopts a hybrid architecture that interleaves softmax attention blocks at regular intervals among linear attention blocks. The rationale is explicit: "linear attention underperforms in retrieval" while "the hybrid linear attention architecture not only matches but also surpasses the retrieval and extrapolation capabilities of the pure softmax attention architecture" (Section 2.2.2, citing Li et al., 2025).

The layer group structure. The model's $N$ transformer layers are evenly divided into layer groups, each containing $M + 1$ layers: $M$ consecutive linear attention layers followed by one Grouped Query Attention (GQA) layer. The paper defines the "hybrid ratio" as the ratio of linear attention to softmax attention blocks within a group. From Table 1:

• Ring-mini-linear-2.0: 20 layers total, $M = 4$, meaning each group has 5 layers (4 linear + 1 GQA). The total ratio of linear-to-softmax is 4:1 across 4 groups. Context length: 128K.

• Ring-flash-linear-2.0: 32 layers total, $M = 7$, meaning each group has 8 layers (7 linear + 1 GQA). The total ratio of linear-to-softmax is 7:1 across 4 groups. Context length: 128K.

When $M = 0$, every layer would be softmax attention — this recovers the standard Transformer. The larger the $M$, the more linear attention layers separate the softmax blocks, and the more overall efficiency is gained, but potentially at the cost of retrieval capability.

Determining the optimal $M$ through scaling laws. The paper uses Chinchilla-style scaling law methodology (Hoffmann et al., 2022) to determine the optimal layer group size empirically rather than heuristically. The procedure:

1. Train multiple model variants with different values of $M$ (the paper explores at least $M = 1$ and higher values, as shown in Figure 3) using the hybrid linear architecture.
2. For each variant, record the training loss at multiple points during training, along with the cumulative training FLOPs spent.
3. Fit power-law relationships between training FLOP budget and training loss for each configuration, following the standard form $\text{Loss} = a \cdot (\text{FLOPs})^b + c$ where lower loss is better.
4. Compare the fitted curves to determine which architecture achieves lower loss at equivalent compute budgets.

The results are shown in Figure 3. The left panel shows that the hybrid linear architecture ($M = 1$, meaning groups of 2 layers: 1 linear + 1 softmax) consistently achieves lower training loss than the pure softmax attention architecture across all FLOP budgets. This establishes that even a modest hybrid ratio (1:1) provides a meaningful improvement.

The right panel compares different layer group size configurations. The key finding: "a large layer group size (e.g., $M = 7$) performs well under high FLOP budgets." This means that at very large compute scales, more aggressive use of linear attention (more linear layers between softmax blocks) does not hurt and may help loss scaling. However, the paper notes a tradeoff: the optimal $M$ depends on total model size and training budget.

The final design choices. Based on the scaling law analysis, the paper selects:

• $M = 7$ (group size 8) for Ring-flash-linear-2.0 (104B total parameters, larger training budget)
• $M = 4$ (group size 5) for Ring-mini-linear-2.0 (16B total parameters, smaller training budget)

The paper's stated principle is "to strike a balance between efficiency and effectiveness." The larger model can afford more aggressive linear-to-softmax ratios because (1) its larger hidden dimension ($d_{\text{model}} = 4096$ vs. 2048) gives the linear attention state ($d \times d$) more representational capacity, and (2) the larger total layer count means there are still 4 softmax attention layers in total (32 layers / 8 per group = 4 groups) — enough for retrieval. The smaller model, with only 20 layers and $d_{\text{model}} = 2048$, keeps more frequent softmax blocks (4 per group × 5 layers = 4 softmax blocks total as well, but at a 1:4 ratio rather than 1:7).

---

Key Architectural Design Choices Within Linear Attention Blocks

The paper reports extensive ablation experiments on the internal design of linear attention blocks, guided by the principle of "achieving the best model performance while ensuring efficient distributed training and decoding" (Section 2.2.3).

Grouped RMSNorm. Standard RMSNorm normalizes activations across the full hidden dimension before feeding into the linear attention kernel and output projection. Under tensor parallelism (TP > 1), this requires an all-reduce communication to compute the global mean and variance across all ranks, which adds latency in both forward and backward passes.

What the paper does instead: Each rank computes RMSNorm locally on its own shard of the hidden dimension using a grouped normalization strategy. This eliminates the all-reduce entirely — no communication across ranks for normalization. The paper justifies this as necessary "to avoid the all-reduce operations required by a standard RMSNorm layer between the linear attention kernel and output projection under tensor parallelism." The cost is that each rank's normalization is computed on a subset of the full feature vector, which may be slightly less statistically stable than global normalization. The paper's ablation results (not quantified explicitly but implied by adoption) show that this tradeoff is favorable for training throughput.

Partial Rotary Position Embedding (RoPE). Rotary Position Embedding (RoPE) encodes position information by rotating query and key vectors in 2D subspaces. The standard approach applies RoPE to all dimensions of Q and K before attention computation. The paper's experiments found that applying RoPE to only half of the Q and K dimensions produced a measurable improvement.

What they report: "incorporating RoPE resulted in a reduction of approximately 0.004 in the training language model (LM) loss." This is a small but meaningful difference in the Chinchilla scaling regime, where even 0.001 loss improvements at scale correspond to significant effective compute savings.

Why partial RoPE: The paper does not provide a specific theoretical justification, but the likely rationale (consistent with prior work on partial RoPE) is that applying RoPE to only part of the representation allows the model to maintain some position-invariant features alongside position-sensitive ones. In linear attention specifically, where the decay mechanism already encodes coarse positional information, full-dimensional RoPE may create redundant or conflicting positional signals. Partial application provides the benefits of relative position encoding without distorting the representation in dimensions that the linear attention's recurrent state depends on for content-based retrieval.

Head-wise decay with power-law scheduling. The decay coefficient $\lambda$ in Equation 3–4 determines how quickly older tokens are "forgotten" from the recurrent state. The paper discovers that the choice of decay rate and its scheduling across attention heads has a substantial impact on model quality.

What they compared: Two methods for assigning decay rates to different attention heads:

1. Linear decay schedule: Assign each head a decay rate uniformly spaced in some range (e.g., head 1 gets $\lambda = 0.99$, head 2 gets $\lambda = 0.98$, etc.). The spacing between adjacent heads' decay rates is constant.

2. Power-law decay schedule: Assign decay rates according to a power-law distribution, where some heads have very long memory (high $\lambda$) and others have very short memory (low $\lambda$), with the distribution skewed to provide more granularity at the high-$\lambda$ end.

What they report: "Using a power-law decay rate for the head-wise decay, as opposed to a linear decay rate, resulted in a reduction of approximately 0.04 in the training LM loss." This is a much larger effect than the RoPE ablation (0.004 vs. 0.04) — an order of magnitude more impactful — and the paper notes it "had a significant impact on the performance of downstream tasks" as well.

Why power-law decay is better: In natural language, the relevance of past tokens does not decay uniformly. Some information (e.g., the topic of a paragraph, a key entity mentioned early) remains relevant for a long time, while other information (e.g., local syntactic constraints) becomes irrelevant quickly. A linear spacing of decay rates does not provide enough resolution at the high-$\lambda$ end — multiple heads end up with nearly identical long-term memory horizons, wasting representational capacity. A power-law distribution places more heads in the high-$\lambda$ regime (long memory), allowing the model to learn fine-grained distinctions between different levels of long-range dependence, while still providing some heads with short horizons for local processing.

---

KV Cache Cost Analysis Across Architectures

Section 2.2.4 provides a quantitative justification for why hybrid linear attention matters for decoding throughput. The key metric is KV cache/State memory access size — the amount of data that must be read from GPU memory at each decoding step to compute attention over the full context.

The comparison (Figure 4). The paper plots how memory access size grows with sequence length from 1K to 128K for three architectures:

• Hybrid Linear (Ring-linear's approach): The state memory is a fixed-size $d \times d$ matrix per head. Total memory access is constant — it does not grow with sequence length. At 128K context, the memory access is the same as at 1K.

• GQA (Grouped Query Attention, used in Llama 3 and the paper's own softmax blocks): KV cache grows linearly with sequence length as $n \times (n_{\text{kv\_heads}} \times d_{\text{head}})$. Memory access scales proportionally. At 128K, access is ~128× larger than at 1K (minus constant overheads).

• MLA (Multi-head Latent Attention, used in DeepSeek-V2/V3): KV cache is compressed into a low-rank latent space, reducing the constant factor but still growing linearly with sequence length. Memory access at 128K is substantially lower than GQA but still grows unboundedly.

The quantitative takeaway from the figure: At short sequences (1K–4K), the three architectures have comparable memory access sizes. GQA may even be slightly more efficient due to highly optimized FlashAttention kernels. But the curves diverge dramatically at longer lengths: GQA and MLA both show linear growth (GQA steeper than MLA), while Hybrid Linear stays flat. By 64K–128K, Hybrid Linear's memory access is an order of magnitude or more below the alternatives.

Why this matters: In autoregressive decoding, each token generation step requires:

1. Loading the full KV cache / state from GPU memory into compute units.
2. Computing attention using that state.
3. Updating the state with the new token's key and value.
4. Storing the updated state back to memory.

On modern GPUs, step 1 is typically the bottleneck — memory bandwidth (HBM throughput) limits how fast data can be moved, while compute units often sit idle waiting for data. If the state size is 1 MB (linear attention) vs. 2 GB (GQA at 128K), the memory transfer takes 2000× less time, and the decode step completes proportionally faster. This is why Figures 7–8 show >10× decode throughput improvements at long generation lengths — the constant-state property of linear attention directly eliminates the memory bandwidth bottleneck that dominates in softmax attention decoding.

---

FP8 Training Optimization Pipeline

The paper develops a comprehensive FP8 (8-bit floating point) training infrastructure called linghe to realize the computational efficiency of the hybrid architecture during pre-training and continued pre-training. The baseline for comparison is "the native blockwise FP8 mixed-precision training method provided by Megatron" (Section 1 contribution bullet). The core insight is that while FP8 GEMM (matrix multiplication) is faster than BF16 GEMM, the quantization overhead — converting BF16 activations to FP8 before each GEMM — can consume nearly 20% of the GEMM time on high-performance GPUs like the H800. Therefore, the optimization strategy is to fuse quantization operations with adjacent kernels to eliminate redundant memory traffic.

Kernel fusion strategy (Figure 5). The paper's optimization reorganizes the computation graph so that quantization, normalization, activation functions, and the linear operations themselves are combined into single GPU kernels, reducing the number of reads and writes to GPU global memory (HBM). Each fusion is described in Section 3.1:

Linear Gate fusion. In linear attention layers, the gate mechanism involves several operations: the attention output goes through a transpose, then a grouped RMSNorm, then a sigmoid activation, and finally element-wise multiplication with the normalized output. In a naive implementation, each of these is a separate kernel — the output of one is written to memory, the next kernel reads it, processes it, and writes it back.

What the fused kernel does: All operations related to the gating mechanism — "attention output transpose, group RMS norm, gate sigmoid, and multiplication" — are combined into a single kernel. The intermediate values stay in GPU registers or shared memory rather than being written to and read from HBM. The paper states this "reduces multiple memory accesses to the GPU memory and lowers activation memory consumption during training." Lower activation memory is critical because it enables larger micro-batch sizes, which the paper found yield "over 20% improvement in training efficiency" when going from MBS=1 to MBS=2 or MBS=2 to MBS=4 in MoE training.

Permute/Unpermute optimization. MoE layers require routing tokens to their assigned experts. The standard Megatron implementation involves separate padding/unpadding (to make token counts uniform across GPUs) and permute/unpermute (to gather tokens for each expert) operations. The paper's optimization modifies the routing map directly: "we adopted a more efficient strategy by modifying the routing map directly. This allows us to integrate padding/unpadding into the permute/unpermute operations." In extreme cases where a GPU receives fewer tokens than the padding size, this approach avoids the overhead of allocating and processing dummy tokens.

QK Norm + Partial RoPE fusion. The linear attention module includes several sequential operations on query and key tensors: splitting them from the combined QKV projection output, applying QK normalization (RMSNorm on Q and K separately), applying partial RoPE (rotary embeddings to half the dimensions), and transposing for the attention computation.

What the fused kernel does: All of these — "split, RoPE, and transpose operations" — are fused with the preceding QKV projection kernel. Instead of writing the QKV projection output to memory, then reading it for split, writing split outputs, reading for norm, etc., the entire sequence happens in registers and shared memory, with only the final attention-ready Q and K tensors written to HBM.

MoE Router fusion. Computing expert routing involves taking hidden states, casting them to FP32 (higher precision needed for the softmax over expert logits), computing router logits, and selecting top-k experts. The baseline approach "requires casting hidden states to FP32 before computing the router, which significantly increases both I/O and activation memory."

What the fused kernel does: The cast to FP32 and the router computation are fused into a single kernel. The kernel takes BF16 hidden states as input, performs the cast internally, computes the router logits and top-k, and outputs results in FP32 (during forward pass, needed for the subsequent dispatch) or BF16 (during backward pass, where full FP32 precision is unnecessary for gradient computation). This eliminates the separate BF16→FP32 casting kernel and the associated memory traffic for storing intermediate FP32 tensors.

Linear Attention kernel redesign. Existing linear attention implementations (the paper cites fla.chunk_simple_gla_fwd from the flash-linear-attention library) use "2–4 kernels for the prefill stage, leading to high kernel launch overhead and memory traffic." Each kernel launch has fixed overhead (microseconds) regardless of how much work it does, and each kernel boundary means writing outputs to memory and reading them back.

What the paper does: A redesigned partitioning strategy using "partitioned Q/K and V" that enables the entire prefill computation in a single Triton kernel (optionally two if Q and K need to be split for further processing). The key insight is to partition the computation along a different dimension that allows all necessary operations — the recurrent state accumulation in Equation 3 — to happen within one kernel without inter-kernel data dependencies. The paper states this "improves performance without sacrificing parallelism."

Quantization fusion. The most impactful fusion is between activation/quantization operations. In FP8 training, every GEMM must receive FP8 inputs. The standard pattern is: activation function (e.g., SiLU) produces BF16 output → quantization kernel reads BF16, writes FP8 → GEMM kernel reads FP8. This involves writing and reading an intermediate tensor whose size equals the activation dimensions.

What the fused kernel does: The activation function kernel is modified to "directly output the quantized tensor, eliminating the need to write and read the BF16 output." For example, the SiLU kernel takes BF16 input, computes the SiLU activation in registers, quantizes the result to FP8, and writes only the FP8 tensor. The I/O volume drops from approximately $8MN$ bytes (write BF16 at $2MN$, read BF16 at $2MN$, write FP8 at $MN$, read FP8 at $MN$) to approximately $4MN$ bytes (write FP8 at $MN$, read FP8 at $MN$). Similarly, in the backward pass, the SiLU backward kernel is modified to directly output quantized gradients dy and dyT, reducing kernel time by "nearly half."

State-aware recomputation. Standard gradient checkpointing (recomputation) trades compute for memory by not storing activations during the forward pass and recomputing them during the backward pass. In FP8 training, the forward pass during recomputation has different requirements than the regular forward pass.

What the paper observes: In the regular forward pass, the quantized input $x$ (FP8) is needed to compute the forward GEMM output $y = xW$. In the backward pass, the transposed quantized input $x^T$ (FP8) is needed to compute the weight gradient $dW = x^T dy$. When activations are recomputed during the backward pass, only the backward-relevant form ($x^T$ in FP8) is needed — the forward-relevant form ($x$ in FP8) is not.

What the paper implements: The recomputation forward pass is specialized: "the forward pass in recomputation can have different computation and quantization logic compared to the regular forward pass." Specifically, when not in recomputation mode, the kernel computes and outputs only the quantized $x$ (needed to compute $y$). When in recomputation mode, the kernel computes and outputs only the quantized $x^T$ (needed to compute $dW$). This avoids redundant quantization operations and reduces the memory traffic during recomputation.

Training throughput results (Figure 6). The paper reports speedups for two configurations:

Ring-mini-linear-2.0 training:

• Baseline: Megatron FP8 blockwise, GBS=4416, MBS=4, TP=2, 32 H800 GPUs.
• Fused kernels: GBS=4352, MBS=4, TP=1, 32 H800 GPUs. Speedup: +21%.
• Fused kernels + TP=1: Same as above. Total speedup vs. baseline: +77%.

The critical enabler for the 77% improvement is that the memory savings from kernel fusion allow tensor parallelism to be eliminated (TP=1 instead of TP=2). TP=2 means each GPU holds half the model parameters and communicates with its pair on every forward/backward pass. TP=1 eliminates this communication entirely, and the saved memory from fused kernels means the model still fits on a single GPU at MBS=4.

Ring-flash-linear-2.0 training:

• Baseline: Megatron FP8 blockwise, GBS=8352, MBS=1, PP=6, VPP=2, 288 H800 GPUs.
• Fused kernels: Same configuration. Speedup: +25%.
• Fused kernels + MBS=2: GBS=8352, MBS=2, PP=8, VPP=1. Total speedup: +57%.

Here, the memory savings enable increasing micro-batch size from 1 to 2 and simplifying the pipeline parallelism configuration (PP=8, VPP=1 instead of PP=6, VPP=2). Larger MBS improves GPU utilization because GEMM operations are more efficient with larger matrices. Simpler pipeline parallelism reduces the number of pipeline bubbles (idle time waiting for dependencies between stages).

---

Inference Optimization Pipeline

The paper develops inference optimizations to ensure that the theoretical efficiency of linear attention translates to actual throughput improvements during deployment. The two key contexts are prefill (processing the input prompt, which is compute-bound) and decode (generating output tokens one by one, which is memory-bandwidth-bound).

Fused linear attention kernels for inference frameworks. Prior to this work, "practical implementations have been limited by the absence of support for advanced inference solutions such as SGLang and the vLLM V1 framework" and "the decoding kernel is often fragmented into multiple operations, which further diminishes overall efficiency" (Section 3.4).

What the paper does: Develops a set of optimized fused linear attention kernels and integrates them into both SGLang and vLLM. These kernels combine the recurrent state update (Equation 3) — decay previous state, add new key-value outer product, multiply by query — into a single GPU kernel rather than separate operations. For the prefill phase, a single fused kernel handles the entire sequence-to-state accumulation, eliminating the "2–4 kernels" that existing libraries used and the associated launch overhead.

Prefill throughput measurement (Figures 7a and 8a). Prefill throughput is measured as the number of input tokens processed per second at a batch size of 1, normalized relative to a baseline model (Qwen3-8B for the mini comparison, Qwen3-32B for the flash comparison).

Ring-mini-linear-2.0 prefill (Figure 7a):

• At context length 4K: Ring-mini-linear-2.0 has slightly lower throughput than Ring-mini-2.0 (the softmax-attention counterpart) and the baseline — consistent with the paper's statement that linear attention's advantages "only become pronounced at sequence lengths beyond 8K."
• At context length 8K: Throughput begins to pull ahead, roughly matching Ring-mini-2.0 and exceeding the baseline.
• At context length 32K: Ring-mini-linear-2.0 shows approximately 1.5–2× the throughput of Ring-mini-2.0 and approximately 4–5× the baseline.
• At context length 128K–512K: Ring-mini-linear-2.0 is at roughly 13–14× normalized throughput, compared to Ring-mini-2.0 at roughly 5–6× and the baseline at 1×. The paper states it "achieves more than 2.5 times the throughput of Ring-2.0 and over 8 times that of the baseline models for context lengths beyond 128K."

Ring-flash-linear-2.0 prefill (Figure 8a):

• The pattern is similar but compressed: at 4K–8K, all models are comparable.
• At 32K: Ring-flash-linear-2.0 shows approximately 1.5–2× over Ring-flash-2.0 and approximately 3× over the Qwen3-32B baseline.
• At 128K: Ring-flash-linear-2.0 achieves approximately 8× normalized throughput, Ring-flash-2.0 around 4×, and Qwen3-Next-80BA3B (also hybrid linear) around 3–4×.

Decode throughput measurement (Figures 7b and 8b). Decode throughput is measured as the number of output tokens generated per second at a batch size of 64, normalized relative to the same baseline. This is the more important metric for reasoning models that generate long chain-of-thought responses.

Ring-mini-linear-2.0 decode (Figure 7b):

• At generation length 4K: Ring-mini-linear-2.0 slightly outperforms Ring-mini-2.0 and is substantially above the baseline (approximately 4–5×).
• At generation length 16K–64K: Ring-mini-linear-2.0 delivers approximately 14–16× the normalized throughput of the baseline, and approximately 2× Ring-mini-2.0.
• The paper states: "At 64K context length, they deliver more than twice the throughput of Ring-2.0 and exceed baseline performance by over tenfold."

Ring-flash-linear-2.0 decode (Figure 8b):

• At generation length 4K: Ring-flash-linear-2.0 is slightly above Ring-flash-2.0 and Qwen3-Next (all around 2–3× baseline).
• At 32K–64K: Ring-flash-linear-2.0 reaches approximately 9–10× normalized throughput. Ring-flash-2.0 is at approximately 4–5×, and Qwen3-Next trails at around 3–4×.

The key insight from these measurements: the throughput advantage grows with sequence length, consistent with the constant-state property of linear attention. At short sequences, the overhead of the linear attention kernel (which computes the $d \times d$ outer product at each step) may be comparable to or even slightly higher than FlashAttention (which is extremely well-optimized for moderate-length softmax attention). But as sequence length grows, the linear attention's constant memory access becomes the dominant factor, while softmax attention's linear KV cache growth increasingly bottlenecks on memory bandwidth.

Speculative decoding with tree masks. Speculative decoding accelerates generation by using a small "draft" model to propose multiple candidate next tokens, then having the large model verify them in parallel. To verify multiple candidates simultaneously, the attention computation must handle a tree-structured attention mask: each candidate token attends to the shared prefix and its own branch, but not to tokens on other branches (which it cannot see in the autoregressive sequence).

The problem: "Existing linear attention kernels in the community do not support custom attention masks, making tree-based speculative decoding infeasible." Standard linear attention accumulates a single recurrent state — there is no mechanism to branch the state and maintain separate states for different speculative paths.

What the paper developed: "the first linear attention kernel that supports tree masks." The implementation details are not fully described, but the core requirement is that the kernel must be able to maintain multiple $kv_t$ state matrices (one per branch in the speculation tree), update each independently during the verification pass, and compute attention outputs that respect the tree mask (each candidate token only sees its own branch's accumulated state). This functionality is "already available in our offline inference framework Flood," and the paper notes work is ongoing to port it to SGLang for online serving.

---

Systematic Training-Inference Alignment for Reinforcement Learning

This is the paper's most distinctive methodological contribution. The problem: RL training for language models involves two separate software stacks — an inference engine (vLLM, SGLang) that generates rollouts, and a training engine (Megatron, FSDP) that computes policy gradients. Even when both are "correct" implementations of the same model architecture, small numerical differences accumulate to produce substantially different output probability distributions, violating the on-policy assumption of PPO and causing training instability.

The three-stage alignment process. The paper describes a systematic methodology (Section 5.2.1):

1. Alignment of prefill in training with prefill in inference. The training engine's forward pass on a prompt is compared token-by-token and layer-by-layer with the inference engine's forward pass on the same prompt, with all activations compared. Discrepancies are traced to their source.

2. Alignment of prefill in training with decode in inference. Even after prefill-prefill matches, the inference engine's decode step (processing one new token at a time, reading from KV cache) may produce different outputs than the training engine's prefill (processing the full sequence at once), due to how attention backends handle these modes differently.

3. Alignment under different parallelization configurations. The same model may be run with different tensor parallelism, pipeline parallelism, or data parallelism settings during training vs. inference. Different parallelism strategies change the order of operations and the aggregation of partial results, which can introduce numerical differences.

The guiding principles for each fix: "ensure identical implementation, maintaining appropriate precision, and eliminating non-determinism."

KV Cache precision (Figure 11). The most impactful fix. Linear attention's recurrent state (Equation 4) requires accumulation: $kv_t = \lambda \cdot kv_{t-1} + k_t^T v_t$. Over many steps, the state accumulates contributions from thousands of tokens. If the state is stored in BF16 (16-bit floating point), rounding errors compound — each addition loses precision relative to the accumulated value. Over 128K tokens, the state can drift substantially from what a high-precision accumulation would produce.

What the paper found: "If the KVCache is initialized as BF16 in the inference engine, errors will accumulate progressively during the recurrent process, leading to significant precision divergence." Figure 11 visualizes this: before correction (Figure 11a), the token output probability distributions from training and inference are visually dissimilar — peaks are in different locations, probabilities differ dramatically. After correction (Figure 11b), the distributions are virtually identical.

The fix: Use FP32 (32-bit floating point) for the KV state storage and accumulation in both training and inference. FP32 provides approximately 7 decimal digits of precision versus BF16's approximately 3 digits, which is sufficient for stable accumulation over 128K+ steps.

LM Head (softmax layer) precision. The final layer that produces token probabilities uses a softmax over logits, which is highly sensitive to numerical precision because softmax exponentiates its inputs — small differences in logits become large differences in probabilities. "Necessitating FP32 for the lm_head layer."

The problem: Computing the LM head in FP32 during training is expensive — it increases memory usage (FP32 tensors are 2× the size of BF16) and computational cost (FP32 GEMM is slower than BF16/FP8 GEMM).

What the paper implements: "a custom GEMM operator that accepts BF16 inputs and performs conversion and computation within registers. This approach maintains sufficient precision while significantly reducing computational and memory costs." The hidden states are kept in BF16 in memory, loaded into GPU registers, converted to FP32, the matrix multiplication is performed in FP32 (or with FP32 accumulation), and the output is written. This avoids storing FP32 intermediates in global memory.

RMSNorm alignment. The paper identifies several specific points where RMSNorm implementations diverge:

• Computation precision: RMSNorm's internal mean-square computation and division must use FP32, not BF16, to maintain consistency. BF16's limited mantissa causes rounding errors in the normalization denominator, which shifts the output distribution.

• Epsilon value: The small constant added to the variance for numerical stability (epsilon) must be exactly the same value in both implementations. Different frameworks may use slightly different defaults (e.g., $10^{-5}$ vs. $10^{-6}$).

• Residual connection handling: In many implementations, RMSNorm is "fused" with the residual addition: the input to layer $L$ is normalized and added to the output of layer $L-1$ in a single kernel. The paper recommends "un-fuse the RMSNorm and residual in both training and inference" — compute them as separate operations — to ensure the residual is kept in FP32 and the normalization is applied identically.

RoPE implementation consistency. The paper warns that "subtle implementation differences between training and inference should be carefully checked. For example, minor discrepancies often exist between a common PyTorch implementation and a RoPE operator in an inference engine, which can lead to slightly different outputs even with identical inputs." Potential discrepancies include: the order of rotation and concatenation of real/imaginary parts, the handling of odd-dimensional splits (when half the dimension is not an integer), the numerical method for computing $\cos$ and $\sin$ of position indices, and whether the rotation is applied in-place or creates a new tensor.

Attention backend consistency. The paper requires that "the used backend must be consistent between training and inference, e.g., FlashAttention (Dao et al., 2022)." Different attention implementations (FlashAttention vs. native PyTorch attention vs. xformers memory-efficient attention) may produce slightly different numerical outputs due to different loop ordering, tiling strategies, and floating-point accumulation orders.

More subtly, "watch out for the misalignment between prefill (used during training) and decode (used during inference)." In FlashAttention, the prefill kernel (processing all query tokens at once) uses a different tiling and parallelization strategy than the decode kernel (processing one query token at a time against a growing KV cache). Even with the same backend, these modes can produce slightly different outputs for the same token. Why this matters for RL: During training, the training engine processes the full rollout sequence in prefill mode (all tokens at once). During rollout generation, the inference engine generated those tokens one at a time in decode mode. The probability computed during re-forwarding (training engine, prefill mode) for token $t$ may differ from the probability that was actually used to sample token $t$ (inference engine, decode mode), because the attention output for token $t$ differs between the two modes. This discrepancy "worsens with longer outputs, making RL training more prone to collapse."

MoE routing determinism. Three specific fixes are required:

• High precision in router computation: The router logits must be computed in FP32 (or with FP32 accumulation) to avoid BF16 rounding causing tokens to be assigned to different experts in training vs. inference.

• Stable top-k implementation: The standard torch.topk function is non-deterministic when multiple values tie for the k-th position — which expert among the tied ones is selected can vary between runs or environments. The paper replaces this with "a stable implementation" that uses a deterministic tie-breaking rule (e.g., always selecting the expert with the lower index).

• Deterministic token permutation and summation order: In MoE layers, tokens assigned to different experts are permuted (reordered) so that all tokens for expert 1 are contiguous, then expert 2, etc. After expert computation, they are un-permuted back to the original order and combined. The paper requires "a deterministic order for token permutation and summation" — the permutation indices must be computed in a reproducible way, and the summation of expert outputs for tokens assigned to multiple experts (ntop_k = 8, so each token goes to 8 experts) must be done in a fixed order rather than allowing parallel reduction with non-deterministic summation order.

The ablation experiment (Figure 10). The paper demonstrates that fixing each module incrementally improves RL training stability. The experiment tracks training reward over steps, starting from an "Original" (unaligned) baseline:

• Original: The reward curve shows high variance and limited growth — training is unstable.
• Fix KV Cache & LM Head: Some improvement in stability, but still significant variance.
• Fix KV Cache & LM Head & RMSNorm: Further improvement in stability and modest reward growth.
• Fix KV Cache & LM Head & RMSNorm & Attention: Significant improvement — the reward curve rises more smoothly.
• Fix KV Cache & LM Head & RMSNorm & Attention & RoPE: The most stable curve with the highest final reward — "each aligned module contributes to improved training efficiency and stability in RL."

The cumulative nature of the improvement confirms that the discrepancies are additive — each unaligned module contributes some noise, and fixing them all is necessary for fully stable training.

Using rollout probabilities directly (Figure 12). Once alignment is achieved, the paper compares two approaches for PPO training:

• Clipping with training probabilities (Equation 6, the standard practice): Re-forward the rollout data through the training engine to recompute probabilities, and use these recomputed probabilities for both the current and old policy in the importance sampling ratio.

• Clipping with rollout probabilities (Equation 5, the theoretically correct approach): Use the probabilities recorded by the inference engine during rollout as $\pi_{\text{rollout}}$, and use the training engine's forward pass only for the current policy $\pi_{\text{training}}(x, \theta)$.

The results (Figure 12, left panel) show that "using rollout probabilities instead of recomputed training probabilities yields higher rewards in the later stages of training." The right panel of Figure 12 tracks the proportion of tokens where the absolute difference between training and inference probabilities exceeds 0.8 — a measure of training-inference disparity. The rollout-probability approach "maintains the training-inference disparity within a more stable range."

Why this works: When alignment is achieved, $\pi_{\text{rollout}} \approx \pi_{\text{training}}(x, \theta_{\text{old}})$ — the inference engine's probabilities are essentially the same as what the training engine would have produced for the behavior policy. Therefore, using the inference probabilities directly is not an approximation; it is the ground truth of what distribution actually generated the data. The recomputed training probabilities, by contrast, are an unnecessary approximation that may introduce small errors due to the prefill-vs-decode attention discrepancy (which is hard to eliminate completely). Using the rollout probabilities also eliminates the computational cost of re-forwarding through the training engine — "not only saves the time required for recomputing training probabilities but also further enhances the efficiency and stability of RL training."

---

Continued Pre-Training Procedure

The Ring-linear models are not trained from scratch. Instead, they are initialized from existing Ling-base-2.0 models (dense softmax-attention models trained on 20T tokens), and the linear attention parameters are added through a two-stage continued pre-training process (Section 4).

Initialization. Starting from the Ling-base-2.0 checkpoints:

• The QKV projection in each new linear attention layer is initialized by converting the corresponding MHA (Multi-Head Attention) parameters: "expanding parameters along the head dimension." This preserves the learned projection weights from the original softmax attention, giving the linear attention a strong initialization.
• Additional parameters introduced by the architecture change — specifically, the gate projection (for the gating mechanism in linear attention) and the new RMSNorm layers — are randomly initialized.

Stage 1: Continued training (capability restoration). The model is trained on data sampled from the same corpus used for Ling-base-2.0, with a 4K context length. The goal is to adapt the new linear attention layers and the modified architecture to perform the language modeling task at a level comparable to the original dense model.

• Ring-mini-linear-base-2.0: trained on 600B tokens.
• Ring-flash-linear-base-2.0: trained on 1T tokens.

The paper uses the Warmup-Stable-Merge (WSM) learning rate scheduler (Tian et al., 2025b) instead of the standard Warmup-Stable-Decay (WSD; Hu et al., 2024b). WSM merges checkpoints from the stable phase to simulate a learning rate decay effect without actually decaying the learning rate — this avoids the loss-of-momentum problem that can occur when decaying from a high stable learning rate.

Stage 2: Mid-training (context extension and quality improvement). Following the same strategy as Ling-base-2.0, the context window is progressively extended:

• From 4K to 32K.
• Then from 32K to 128K.

Simultaneously, "the proportion of high-quality reasoning data" in the training mixture is increased. This prepares the model for the post-training phase (SFT and RL), where it will encounter long-context reasoning problems and must generate long chain-of-thought responses.

Effectiveness of continued pre-training (Figure 9). The paper evaluates how much of the original Ling-base model's capabilities are preserved after the architecture conversion and continued pre-training. Performance is normalized relative to Ling-base-2.0 across different capability categories:

• Ring-mini-linear-base-2.0: Restores more than 98% of Ling-mini-base-2.0's performance in most categories (NLU, Math, Code, Basic Knowledge), but shows "minor deficiencies in reasoning and professional knowledge tasks" — attributed to "the knowledge forgetting problem (Ibrahim et al., 2024) during the continued pre-training process." The 600B tokens of continued training, while a small fraction of the original 20T pretraining corpus, shifts the data distribution enough to partially overwrite some specialized knowledge.

• Ring-flash-linear-base-2.0: Shows a similar pattern — >98% retention in most categories, with slightly larger drops in reasoning and professional knowledge, likely due to the larger model having more parameters to adapt and thus more opportunity for forgetting.

The paper's framing is that this is an acceptable tradeoff: the efficiency gains from the hybrid architecture (10× inference cost reduction vs. a dense 32B model, >50% reduction vs. the prior Ring series) justify a small regression in specialized capabilities, which can then be recovered and extended during the post-training phase (SFT and RL on high-quality reasoning data).

---

Post-Training: Supervised Fine-Tuning and Reinforcement Learning

Supervised Fine-Tuning (SFT). The SFT stage (Section 5.1) prepares the model for RL by training it to follow instructions and produce well-structured reasoning traces, with data emphasizing "comprehensive and balanced reasoning capabilities and generalization ability."

Data composition: The SFT mixture includes:

• High-difficulty data in mathematics, coding, science, and logic reasoning domains.
• General-purpose data covering knowledge, agent tasks, subjective creation, and medical domains — ensuring the reasoning-oriented model can still handle non-reasoning queries.
• Re-synthesized Function Calling data "to better align with more general Function Calling patterns" — an important practical capability for agent applications.

Data quality: "All SFT data has undergone stricter de-noising and de-toxification processes, including n-gram filtering and semantic similarity detection, to ensure model safety and the authenticity of its capabilities." De-duplication (n-gram and semantic) is critical because SFT data repeated verbatim from pre-training can cause the model to memorize rather than generalize.

Training details: SFT is conducted with a context window of 128K. To prevent overfitting (which would reduce the model's flexibility for the subsequent RL stage), the paper selects "a checkpoint of an earlier epoch (not the one with the highest benchmark score) for the downstream RL stage." This is a deliberate choice to leave room for RL to improve the model, rather than converging to a narrow SFT optimum that RL cannot escape.

Reinforcement Learning setup. The RL stage (Section 5.2) builds on the aligned training-inference infrastructure described above. Key configuration choices:

• Domains: RL training spans "mathematics, coding, science, logic, and subjective tasks."
• Data curation: "All of the samples underwent meticulous screening, from which samples with an appropriate difficulty level were selected for RL training." The screening ensures that prompts are within the model's capability range — not trivially easy (no learning signal) but not impossible (no positive rewards).
• Context window during RL: Trained with "a sufficiently long context window (e.g., 64K) to strike an optimal balance between performance and efficiency." The paper argues that "with high-difficulty training data, smaller windows (e.g., 32K) introduce potential limitations, namely a high truncation rate and a lower performance ceiling." If reasoning chains are frequently truncated at 32K, the model never learns to produce longer, more thorough chains. A 64K window allows most reasoning traces to complete, providing full learning signals.

Why the alignment matters for PPO (Equations 5–6). The paper formalizes the impact of training-inference alignment on the PPO objective. The ideal PPO update (Equation 5) is:

\nabla_\theta J(\theta) = \mathbb{E}_{x \sim \pi_{\text{rollout}}} \left[ \nabla_\theta \min\left(\frac{\pi_{\text{training}}(x, \theta)}{\pi_{\text{rollout}}(x, \theta_{\text{old}})} \hat{A}, \text{clip}\left(\frac{\pi_{\text{training}}(x, \theta)}{\pi_{\text{rollout}}(x, \theta_{\text{old})}, 1 - \epsilon, 1 + \epsilon\right) \hat{A}\right) \right]

where $\pi_{\text{rollout}}(x, \theta_{\text{old}})$ is the true behavior policy that generated the rollout data, $\pi_{\text{training}}(x, \theta)$ is the current policy being optimized, $\hat{A}$ is the advantage estimate, and $\epsilon$ is the PPO clipping parameter.

What this computes: The PPO clipped objective. The importance sampling ratio $r = \pi_{\text{training}}(x, \theta) / \pi_{\text{rollout}}(x, \theta_{\text{old}})$ measures how much more likely the current policy makes the observed actions compared to the behavior policy. If $r$ is close to 1, the policies are similar and the update is standard. If $r$ deviates far from 1, the clip function limits the update to prevent destructive large policy changes. The advantage $\hat{A}$ (positive for better-than-expected actions, negative for worse-than-expected) determines the direction of the update.

Why this form requires alignment: The denominator $\pi_{\text{rollout}}(x, \theta_{\text{old}})$ must be the actual probability that the inference engine used when generating the data. If training and inference disagree on what this probability was, the ratio $r$ becomes incorrect, and the clip provides no protection because the error is in the ratio computation itself, not in its magnitude.

The standard workaround (Equation 6) — used in verl, OpenRLHF, and most RL frameworks — recomputes the "old" probabilities through the training engine:

$\nabla_\theta J'(\theta) = \mathbb{E}_{x \sim \pi_{\text{rollout}}} \left[ \nabla_\theta \min\left(\frac{\pi_{\text{training}}(x, \theta)}{\pi_{\text{training}}(x, \theta_{\text{old}})} \hat{A}, \text{clip}\left(\frac{\pi_{\text{training}}(x, \theta)}{\pi_{\text{training}}(x, \theta_{\text{old}})} , 1 - \epsilon, 1 + \epsilon\right) \hat{A}\right) \right]$

This uses $\pi_{\text{training}}(x, \theta_{\text{old}})$ — the training engine's re-forward of the rollout data under the behavior policy parameters — as the denominator. This is "biased because it totally ignores the training-inference disparity" — if the inference engine would have assigned probability 0.01 to a token but the training engine's recomputation assigns 0.05, the importance ratio is off by a factor of 5, and the gradient update is computed as if the data came from a different distribution than it actually did.

The paper's position: Once training-inference alignment is achieved, $\pi_{\text{rollout}}(x, \theta_{\text{old}}) \approx \pi_{\text{training}}(x, \theta_{\text{old}})$ (the probability computed by the inference engine during rollout matches what the training engine would compute for the same checkpoint). Therefore, Equation 5 can be used directly with the rollout probabilities, which is the theoretically correct formulation and avoids the computational cost of re-forwarding. Figure 12 confirms that this approach both achieves higher reward and maintains smaller training-inference probability disparity throughout training.

RL training results (Figure 13). The paper shows training curves for Ring-mini-linear-2.0 demonstrating the effectiveness of the full pipeline:

• Training reward (Figure 13a): Steadily increases from approximately 0.54 to 0.64 over the training steps, with low variance — no training collapse or degradation. This is the direct evidence that the alignment methodology achieves long-horizon RL stability.
• Test score on AIME'25 (Figure 13b): Correlates with training reward, rising from approximately 0.70 to 0.74. The paper notes this as evidence that the RL improvements transfer to held-out evaluation.
• Test score on LiveCodeBench (Figure 13c): Rises from approximately 0.58 to 0.62, showing improvement in code reasoning as well.

The key property is the monotonic improvement across all three metrics without the divergence or collapse that the paper argues is common in long-output MoE model RL training without systematic alignment. This validates the central claim: "training stability for long-output MoE models depends on operator-level alignment rather than algorithmic mitigation alone."

4. Key Insights and Innovations

Innovation 1: Training-Inference Disparity Is a Systems Engineering Problem, Not an Algorithmic One

The dominant assumption in the RLHF community — encoded by default in frameworks like verl (Sheng et al., 2024) and OpenRLHF (Hu et al., 2024a) — has been that training-inference discrepancies are an unavoidable fact of life when working with separate training and inference engines. The standard response is algorithmic mitigation: recompute probabilities through the training engine (Equation 6) to avoid using "untrustworthy" inference-engine probabilities, and apply PPO clipping as a safeguard. Prior work on this specific problem (Zheng et al., 2025; Yao et al., 2025) focused entirely on developing more sophisticated algorithmic workarounds — modified objectives, better clipping schedules, off-policy corrections — that treat the discrepancy as a given.

This paper makes a fundamentally different diagnostic move: it argues that training-inference disparity is primarily a correctable engineering artifact, not an inherent limitation, and that addressing it at the systems level removes the need for algorithmic workarounds entirely. The conceptual shift is from "how do we design RL algorithms robust to numerical drift?" to "why is there numerical drift, and can we eliminate it?"

The evidence for this reframing is the ablation experiment in Figure 10: each module fixed (KV cache precision, LM head, RMSNorm, attention backend, RoPE implementation) produces an incremental improvement in RL training stability and final reward. This is not a case where one bug fix changes everything — the cumulative curve demonstrates that the problem is the sum of many small, individually addressable discrepancies across the stack. Once all five modules are aligned, the training curve becomes monotonically improving without any algorithmic modifications to PPO. The paper's statement that "no additional algorithmic modifications are necessary" (Section 5.2.2) is a direct challenge to the premise of prior algorithmic-mitigation work.

The significance of this innovation extends beyond the specific fixes described. It establishes a new diagnostic framework: when RL training for long-output models is unstable, the first step should be systematic activation-comparison between training and inference engines, module by module, layer by layer — not tuning PPO hyperparameters. This is a practical methodology that any team training reasoning models can adopt. The three-stage alignment protocol (prefill-prefill, prefill-decode, different parallelization configurations) provides a concrete checklist for what to check and in what order.

This is a fundamental reframing, not incremental. The field had implicitly accepted that inference engines and training engines produce different outputs as a cost of using specialized, optimized software for each phase. This paper argues that this cost is not necessary — it is an engineering failure that can be systematically eliminated — and shows that doing so produces smoother RL training than any algorithmic mitigation. The counter-argument (which the paper does not fully address) is whether this level of operator-level alignment is maintainable across framework versions, hardware generations, and model architectures. The paper demonstrates it for one stack (Megatron training, vLLM/SGLang inference, PaLM-derived architecture), but the maintenance burden of ensuring every operator remains bit-identical across independently developed codebases is substantial.

---

Innovation 2: The Optimal Hybrid Architecture Ratio Is Empirically Determined, Not Heuristically Chosen

Prior hybrid linear-softmax attention models — Minimax M1 (Chen et al., 2025), GPT-OSS (Agarwal et al., 2025), Qwen3-Next — made architectural decisions about how many linear attention layers to place between softmax attention blocks without systematic public study. The choice was based on intuition (linear attention is cheaper, use as much as possible without obviously degrading quality) or small-scale ablations. The field lacked evidence for what the optimal ratio actually is, how it scales with model size and compute budget, and whether the answer is uniform or model-specific.

This paper applies Chinchilla-style scaling law methodology (Hoffmann et al., 2022) — fitting power-law relationships between training FLOPs and loss for different architectural variants — to the hybrid attention ratio problem. The conceptual contribution is recognizing that the hybrid ratio is not just an engineering tradeoff but a scaling law variable: the optimal choice depends on total compute budget, not just model size. The right panel of Figure 3 shows that $M=7$ (large layer group size, aggressive linear attention use) performs best at high FLOP budgets, while smaller $M$ values may be preferable at lower budgets. This means the optimal architecture for one training regime may not be optimal for another — a scaling law insight that the prior heuristic approaches could not capture.

The specific findings — $M=4$ for the 16B model, $M=7$ for the 104B model — are empirical guidelines for future hybrid architecture design. But the deeper contribution is the demonstration that these ratios can and should be empirically determined through systematic FLOPs-matched comparison, rather than guessed. This is an incremental but important refinement to the hybrid architecture design space: prior work established that hybrid architectures work; this paper shows how to design them optimally for a given compute budget.

The scaling law fit also provides a negative result of significance: the left panel of Figure 3 shows that even a modest hybrid ratio ($M=1$, meaning 1:1 linear-to-softmax) consistently outperforms pure softmax attention. This means there is no regime, at any FLOP budget studied, where pure softmax attention is preferable. The hybrid architecture is not just an efficiency hack — it is a genuinely better architecture for language modeling at scale, independent of efficiency considerations. This challenges the default assumption that softmax attention is the "safe" choice and linear attention is a compromise.

The paper does not explore the full design space — it fixes the hybrid pattern to uniform groupings of $M$ linear + 1 softmax, rather than exploring non-uniform patterns or different placement strategies — but the scaling law methodology is the key intellectual contribution that future work can extend.

---

Innovation 3: The Constant-State Property of Linear Attention Enables a Qualitative Shift in Decoding Economics at Scale

The theoretical efficiency advantage of linear attention — $O(nd^2)$ instead of $O(n^2 d)$, constant KV state instead of linear KV cache — has been understood since the original RetNet and Lightning Attention papers. What this paper contributes is not the theory but the empirical demonstration that these theoretical advantages manifest as a qualitative, not just quantitative, difference in decoding throughput at sequence lengths relevant to modern deployment.

This is best understood through Figures 7b and 8b: the decode throughput curves for Ring-linear models do not just show a constant factor improvement over softmax attention baselines — they show a fundamentally different scaling behavior with generation length. Softmax attention decode throughput degrades as generation length increases (because the KV cache grows, increasing memory bandwidth pressure at each step). Hybrid linear attention decode throughput either grows or remains flat. At 64K generation length, the throughput difference is more than 10×. It is not that linear attention is 10× faster at every length; it is that linear attention's cost per token is independent of prior context length, while softmax attention's cost per token grows linearly. The curves diverge, not just shift.

This qualitative shift has economic implications that the paper quantifies but does not fully articulate: in softmax attention, generating token 64,001 costs substantially more than generating token 1,001, because the KV cache is 64× larger. This means the cost of long reasoning chains is super-linear in chain length — each additional reasoning step is more expensive than the last. In hybrid linear attention, generating token 64,001 costs the same as generating token 1,001. Reasoning chains scale linearly in cost with length, not super-linearly. For test-time scaling strategies that deliberately generate very long chain-of-thought sequences to improve accuracy (Snell et al., 2024; Muennighoff et al., 2025), this changes the economics fundamentally: doubling the reasoning budget doubles the cost, rather than quadrupling it.

The paper supports this with the specific comparison that inference cost is "1/10" that of a 32B dense model (Section 1) — a figure that only makes sense when considering long-context scenarios where the asymptotic differences dominate. At 4K context, the advantage is modest or nonexistent (Figures 7–8, leftmost points). At 128K+, it is dramatic. The innovation is not the mechanism but the demonstration that the deployment regimes where linear attention matters are exactly the regimes that reasoning models are pushing toward — long outputs, long contexts — meaning the efficiency gains are not just theoretical but immediately relevant to the frontier of LLM deployment.

This is an incremental empirical contribution to the linear attention literature: it confirms, with careful measurements on production-scale models and frameworks (SGLang, vLLM), that the theoretical asymptotic advantage is realizable in practice. The key twist is that it is only realizable with substantial kernel engineering — the fused linear attention kernels that combine recurrent state updates into single GPU operations — which explains why prior linear attention implementations had not demonstrated these dramatic throughput advantages in deployed systems. The paper's contribution is closing the gap between theory and practice.

---

Innovation 4: Knowledge Forgetting During Architecture Conversion Is Quantifiable and Partially Recoverable

When converting a pre-trained softmax attention model to a hybrid linear architecture through continued pre-training, the paper quantifies a specific cost: knowledge forgetting. Figure 9 shows that after 600B–1T tokens of continued pre-training with the new architecture, the Ring-linear base models recover >98% of the original Ling-base model performance in most categories (NLU, Math, Code, Basic Knowledge), but show "minor deficiencies in reasoning and professional knowledge tasks."

This is significant not because it is surprising — catastrophic forgetting during distribution shift is well-documented (Ibrahim et al., 2024) — but because it quantifies the tradeoff for architecture conversion and establishes that forgetting is not uniform across capability types. The model does not lose capabilities evenly; it selectively degrades on tasks requiring specialized reasoning patterns or domain-specific professional knowledge, while general language understanding and standard benchmarks are largely preserved.

The practical implication is a cost-benefit framework for architecture conversion: the inference efficiency gains must be weighed against the specific capability regressions for the target deployment domain, and the continued pre-training budget (600B–1T tokens) is a significant investment — roughly 3–5% of the original 20T pre-training corpus. For a team considering converting an existing dense model to hybrid linear, Figure 9 provides a concrete estimate of what to expect.

The follow-up finding is equally important: the post-training phase (SFT + RL) is designed to recover and extend these lost capabilities. The paper's reasoning benchmarks (Tables 2–3) show that the final Ring-linear models achieve competitive or superior performance to softmax-attention counterparts. This implies that the forgetting is not permanent — it can be compensated through targeted fine-tuning on high-quality reasoning data and RL. The architecture conversion is not a one-way loss of capability; it is a reallocation of the model's capacity that requires re-specialization through post-training.

This innovation is incremental: the phenomenon of forgetting during continued training is known, and the recovery through fine-tuning is standard practice. The contribution is the specific quantification for the hybrid linear conversion context and the demonstration that the efficiency gains justify the temporary regression. The paper does not explore whether the forgetting could be mitigated through different continued pre-training strategies (e.g., mixed architecture training from the start, or distillation from the original model), leaving this as future work.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation is conducted across a suite of 17 benchmarks spanning three reasoning dimensions, as described in Section 6.1. For Mathematical Reasoning: AIME'24 (MAA, 2024), AIME'25 (MAA, 2025), OlympiadBench (He et al., 2024), CNMO'24 (CMNO, 2024), LiveMathBench (Liu et al., 2024c), and TheoremQA (Chen et al., 2023). For Agent and Coding: Humaneval+ (Liu et al., 2024b), MBPP+ (Liu et al., 2024b), LiveCodeBench (Jain et al., 2024), CodeForces (Codeforces, 2024), Spider (Yu et al., 2018), and BFCL-Live (Yan et al., 2024). For General Reasoning: GPQA-Diamond (Rein et al., 2024), SciBench (Wang et al., 2023), DROP (Dua et al., 2019), MuSR (Sprague et al., 2023), and Multi-LogiEval (Patel et al., 2024). The paper does not specify a held-out validation set or cross-validation protocol for these benchmarks — the evaluation appears to be a single-pass measurement on the standard test sets provided by each benchmark. For continued pre-training evaluation (Figure 9), the paper uses normalized scores across capability categories (NLU, Math, Code, Reasoning, Basic Knowledge, Professional Knowledge) but does not enumerate the specific constituent benchmarks for each category.

• Base model(s). Two models are evaluated: Ring-mini-linear-2.0 (16.4B total parameters, 1.6B activated, 957M non-embedding activated; 20 layers, d_model = 2048, 256 experts with top-8 routing, hybrid ratio 1:4, context length 128K) and Ring-flash-linear-2.0 (104.2B total parameters, 7.4B activated, 6.1B non-embedding activated; 32 layers, d_model = 4096, 256 experts with top-8 routing, hybrid ratio 1:7, context length 128K). Both models are the final post-trained versions after continued pre-training (initialized from Ling-base-2.0-20T checkpoints), supervised fine-tuning, and reinforcement learning. The models were chosen to represent two scale regimes: a compact model suitable for deployment efficiency comparisons and a larger model competitive with frontier reasoning models. The paper explicitly frames them as reasoning-oriented models that also handle non-reasoning tasks.

• Metrics. Each benchmark has its own standard metric as specified by the benchmark authors: accuracy (percentage of correct answers) for AIME'24, AIME'25, OlympiadBench, CNMO'24, GPQA-Diamond, SciBench, DROP, MuSR, and Multi-LogiEval; pass@1 or equivalent functional correctness for Humaneval+, MBPP+, LiveCodeBench (pass@1 on sampled solutions), and Spider (exact match accuracy on SQL queries); Elo score for CodeForces (a competitive programming rating derived from problem difficulty and solution correctness); and benchmark-specific accuracy for LiveMathBench, TheoremQA, and BFCL-Live. For the continued pre-training analysis (Figure 9), a single normalized score per capability category is reported relative to the Ling-base-2.0 performance, but the paper does not detail how multiple benchmarks within a category are aggregated (e.g., average, weighted average, best). For inference efficiency (Figures 7–8), the metric is normalized throughput: prefill throughput in tokens/second at batch size 1, and decode throughput in tokens/second at batch size 64, both normalized to a baseline model (Qwen3-8B for the mini comparison, Qwen3-32B for the flash comparison) whose throughput is set to 1.0.

• Baselines. For Ring-mini-linear-2.0 (Table 2): Ring-Mini-2.0 (the softmax-attention counterpart from the prior Ring series), Qwen3-8B-Thinking (Yang et al., 2025), and GPT-OSS-20B-Medium (Agarwal et al., 2025). For Ring-flash-linear-2.0 (Table 3): Ring-Flash-2.0 (softmax-attention counterpart), Qwen3-32B-Thinking (Yang et al., 2025), Gemini-2.5-Flash (Comanici et al., 2025), GPT-OSS-120B-Medium (Agarwal et al., 2025), Seed-OSS-36B-Instruct (ByteDance), and Qwen3-Next-80BA3B-Thinking (Qwen). All baselines are reasoning-oriented models of comparable parameter scale. For inference efficiency comparisons (Figures 7–8), additional baselines include the dense softmax-attention counterparts (Qwen3-8B and Qwen3-32B as the normalized baselines) and the prior Ring-2.0 series (Ring-mini-2.0, Ring-flash-2.0) with softmax attention. For the flash-size efficiency comparison, Qwen3-Next-80BA3B (also a hybrid linear architecture) is included as a same-architecture comparison point.

• Generation budget / compute accounting. For the benchmark evaluations, the paper does not report a consistent generation budget across all benchmarks — the inference procedure (sampling temperature, number of samples, whether chain-of-thought is used) is determined by each benchmark's standard evaluation protocol unless otherwise noted. For LiveCodeBench, the score is explicitly labeled as "sampled" in Figure 13c, implying multiple samples with pass@1. For inference efficiency measurements (Figures 7–8), the compute budget is controlled by fixing batch size (1 for prefill, 64 for decode) and measuring throughput at varying sequence lengths (4K, 8K, 16K, 32K, 64K, 128K, and in some cases up to 512K for prefill). Prefill throughput measures input tokens processed per second for a single prompt of the given length; decode throughput measures output tokens generated per second across 64 concurrent requests at the given generation length. For the FP8 training throughput comparisons (Figure 6), the compute budget is implicitly the fixed hardware configuration (32 H800 GPUs for mini, 288 H800 GPUs for flash) with specified global batch size, micro-batch size, and parallelism strategy; throughput is measured in tokens per second or effective training speed relative to the baseline.

• Cross-validation / statistical protocol. The paper does not report cross-validation, statistical significance tests, confidence intervals, or multiple evaluation runs with different seeds for any benchmark results. The benchmark scores in Tables 2–3 and Figures 1, 9, and 13 appear to be single-run measurements. For the scaling law fits (Figure 3), the paper follows Chinchilla methodology (fitting power-law curves to multiple training runs at different FLOP budgets) but does not report confidence intervals on the fitted parameters or the uncertainty in the crossover points between architectures. For the RL training curves (Figure 13), the x-axis is labeled "Step" without specification of the number of rollouts per step or the variance across different random seeds. The training-inference alignment ablation (Figure 10) shows reward curves for five configurations but does not indicate whether multiple runs were conducted to assess variance. The PPO clipping comparison (Figure 12) shows two reward curves but again without error bars or replicate information. This is a meaningful limitation: for the central claim that systematic alignment enables stable RL training, the paper provides qualitative curve shapes (monotonic improvement vs. collapse) but no quantitative stability metrics or statistical tests to distinguish reliable improvement from random fluctuation.

Main Quantitative Results

Benchmark Performance of Ring-mini-linear-2.0

Table 2 reports the performance of Ring-mini-linear-2.0 across 17 benchmarks, compared against three baselines of similar parameter scale. The headline result is that despite having only 1.6B activated parameters (957M non-embedding), Ring-mini-linear-2.0 achieves performance that is "comparable to its counterpart models on various reasoning tasks."

On mathematical reasoning benchmarks, the model is competitive with Ring-Mini-2.0 (its softmax-attention predecessor) and the baselines. On AIME'24: Ring-mini-linear-2.0 scores 79.95 versus Ring-Mini-2.0 at 79.69, Qwen3-8B-Thinking at 79.27, and GPT-OSS-20B-Medium at 77.86 — a marginal advantage over all three. On AIME'25: 73.65 versus 74.06 (Ring-Mini-2.0), 71.25 (Qwen3-8B-Thinking), and 73.85 (GPT-OSS) — essentially tied with Ring-Mini-2.0 and GPT-OSS, and ~2.4 points above Qwen3. On OlympiadBench: 82.91, tying Ring-Mini-2.0 and slightly ahead of Qwen3 (82.27) and GPT-OSS (80.59). On CNMO'24: 77.60, narrowly leading Ring-Mini-2.0 (76.91), Qwen3 (75.09), and GPT-OSS (75.87). On LiveMathBench: 83.64, essentially tied with Ring-Mini-2.0 (83.98) and GPT-OSS (83.40), slightly ahead of Qwen3 (82.92). On TheoremQA: 69.69, marginally behind Ring-Mini-2.0 (70.09) but ahead of Qwen3 (68.81) and GPT-OSS (66.81).

On agent and coding benchmarks, the pattern is uneven. Humaneval+: Ring-mini-linear-2.0 scores 91.16 versus 91.84 (Ring-Mini-2.0), 84.76 (Qwen3), 87.27 (GPT-OSS) — competitive but slightly behind Ring-Mini-2.0. MBPP+: 79.70 versus 80.79 (Ring-Mini-2.0), 79.63 (Qwen3), 79.86 (GPT-OSS) — essentially tied. LiveCodeBench: 59.53 versus 62.56 (Ring-Mini-2.0), 56.94 (Qwen3), 54.90 (GPT-OSS) — notably behind Ring-Mini-2.0 by ~3 points but ahead of the other baselines. CodeForces (Elo): 83.84 versus 84.80 (Ring-Mini-2.0), 73.31 (Qwen3), 82.25 (GPT-OSS) — close to Ring-Mini-2.0 and GPT-OSS, substantially ahead of Qwen3. Spider: 79.18 versus 77.64 (Ring-Mini-2.0), 79.41 (Qwen3), 78.81 (GPT-OSS) — all within ~1.8 points. BFCL-Live: 73.99 versus 74.26 (Ring-Mini-2.0), 75.99 (Qwen3), 54.64 (GPT-OSS) — tied with Ring-Mini-2.0, behind Qwen3 by 2 points, dramatically ahead of GPT-OSS.

On general reasoning benchmarks, the results are mixed. GPQA-Diamond: 65.69 versus 68.24 (Ring-Mini-2.0), 62.00 (Qwen3), 65.53 (GPT-OSS) — behind Ring-Mini-2.0 by ~2.6 points, ahead of the others. SciBench: 5.46 versus 4.74 (Ring-Mini-2.0), 4.39 (Qwen3), 4.43 (GPT-OSS) — all models score very low on this challenging benchmark, and Ring-mini-linear-2.0 holds a slight edge. DROP: 83.20 versus 88.55 (Ring-Mini-2.0), 87.13 (Qwen3), 76.49 (GPT-OSS) — notably behind Ring-Mini-2.0 by 5.35 points and Qwen3 by 3.93 points, but ahead of GPT-OSS. MuSR: 77.21 versus 75.99 (Ring-Mini-2.0), 76.92 (Qwen3), 76.53 (GPT-OSS) — narrowly leading. Multi-LogiEval: 73.49 versus 73.73 (Ring-Mini-2.0), 77.08 (Qwen3), 60.80 (GPT-OSS) — tied with Ring-Mini-2.0, notably behind Qwen3 by ~3.6 points.

Overall pattern for Ring-mini-linear-2.0: The model performs remarkably close to its softmax-attention counterpart (Ring-Mini-2.0), with differences typically within 1–3 percentage points. Where it lags (LiveCodeBench, DROP, GPQA-Diamond), the gap is moderate but noticeable. Where it leads (AIME'25, CNMO'24, SciBench), the margins are small. This is consistent with the paper's continued pre-training analysis (Figure 9a), which showed >98% performance recovery in most categories but minor deficiencies in reasoning. The benchmark results suggest those deficiencies are real but limited — Ring-mini-linear-2.0 is competitive with its softmax predecessor despite the architecture conversion, and the efficiency gains come at a modest capability cost.

Benchmark Performance of Ring-flash-linear-2.0

Table 3 presents the more consequential comparison: the 104B-parameter Ring-flash-linear-2.0 against six baselines, including frontier reasoning models. The headline from Section 1 and Figure 1 is that Ring-flash-linear-2.0 achieves "SOTA performance across multiple challenging complex reasoning benchmarks" while reducing inference cost to 1/10 of a dense 32B model.

On mathematical reasoning (the category with the most benchmarks and the most competitive baselines), the results are strong but not uniformly dominant. AIME'24: Ring-flash-linear-2.0 scores 90.73 versus 91.04 (Ring-Flash-2.0), 82.66 (Qwen3-32B-Thinking), 80.36 (Gemini-2.5-Flash), 80.40 (GPT-OSS-120B-Medium), 91.70 (Seed-OSS-36B-Instruct), and 92.08 (Qwen3-Next-80BA3B-Thinking). The model sits ~1.4 points below Qwen3-Next and ~1 point below Seed-OSS, but dramatically ahead of Qwen3-32B (+8 points) and Gemini-2.5-Flash (+10 points). Crucially, it is essentially tied with its softmax counterpart Ring-Flash-2.0 (90.73 vs. 91.04).

AIME'25 (the key benchmark highlighted in Figure 1): 86.51 versus 86.98 (Ring-Flash-2.0), 75.47 (Qwen3-32B), 72.00 (Gemini-2.5-Flash), 80.00 (GPT-OSS-120B), 84.70 (Seed-OSS-36B), 87.80 (Qwen3-Next-80BA3B). Ring-flash-linear-2.0 trails Qwen3-Next by 1.29 points and Seed-OSS by 1.81 points in absolute terms (the paper claims Seed-OSS at 84.70 vs. Ring-flash at 86.51, so Ring-flash leads by 1.81), and is essentially tied with Ring-Flash-2.0. The gap to the non-hybrid baselines (Qwen3-32B, Gemini-2.5-Flash, GPT-OSS-120B) is large: +11.04, +14.51, and +6.51 respectively.

OlympiadBench: 87.36 versus 88.10 (Ring-Flash-2.0), 84.69 (Qwen3-32B), 83.41 (Gemini-2.5-Flash), 82.32 (GPT-OSS-120B), 87.06 (Seed-OSS), 87.80 (Qwen3-Next). The model trails Qwen3-Next by 0.44 points, ties Seed-OSS, and is ~0.7 points behind Ring-Flash-2.0. CNMO'24: 84.98 versus 85.07 (Ring-Flash-2.0), 78.21 (Qwen3-32B), 82.38 (Gemini), 79.95 (GPT-OSS), 91.75 (Seed-OSS), 85.33 (Qwen3-Next). Notably behind Seed-OSS (-6.77 points) but competitive with others. LiveMathBench: 88.11 versus 87.84 (Ring-Flash-2.0), 86.85 (Qwen3-32B), 85.86 (Gemini), 85.89 (GPT-OSS), 88.70 (Seed-OSS), 88.08 (Qwen3-Next) — all models within ~3 points, with Ring-flash at parity. TheoremQA: 74.16 versus 74.91 (Ring-Flash-2.0), 73.88 (Qwen3-32B), 72.25 (Gemini), 72.16 (GPT-OSS), 77.16 (Seed-OSS), 75.97 (Qwen3-Next) — in the middle of the pack, ~3 points behind Seed-OSS.

On agent and coding benchmarks, the performance is generally stronger relative to baselines. Humaneval+: 91.84 versus 92.00 (Ring-Flash-2.0), 90.24 (Qwen3-32B), 91.77 (Gemini), 84.83 (GPT-OSS), 91.23 (Seed-OSS), 91.77 (Qwen3-Next) — all models except GPT-OSS are within ~1.8 points. MBPP+: 81.02 versus 81.28 (Ring-Flash-2.0), 82.28 (Qwen3-32B), 80.42 (Gemini), 79.99 (GPT-OSS), 80.89 (Seed-OSS), 81.55 (Qwen3-Next) — competitive but not leading. LiveCodeBench (featured in Figure 1): 70.37 versus 70.76 (Ring-Flash-2.0), 62.33 (Qwen3-32B), 61.40 (Gemini), 66.46 (GPT-OSS), 69.01 (Seed-OSS), 71.97 (Qwen3-Next). Ring-flash-linear-2.0 trails Qwen3-Next by 1.6 points but leads the non-hybrid-linear baselines by substantial margins: +8.04 over Qwen3-32B, +8.97 over Gemini, +3.91 over GPT-OSS. CodeForces (Elo): 90.24 versus 90.23 (Ring-Flash-2.0), 84.25 (Qwen3-32B), 81.59 (Gemini), 89.67 (GPT-OSS), 83.78 (Seed-OSS), 89.77 (Qwen3-Next) — the highest score among all baselines (marginally ahead of Qwen3-Next by 0.47 points). Spider: 80.86 versus 81.70 (Ring-Flash-2.0), 81.00 (Qwen3-32B), 77.27 (Gemini), 78.71 (GPT-OSS), 78.16 (Seed-OSS), 82.95 (Qwen3-Next) — mid-pack, behind Qwen3-Next by ~2.1 points. BFCL-Live: 75.51 versus 75.22 (Ring-Flash-2.0), 76.34 (Qwen3-32B), 75.26 (Gemini), 60.22 (GPT-OSS), 61.05 (Seed-OSS), 77.17 (Qwen3-Next) — significantly ahead of GPT-OSS and Seed-OSS (+15.3 and +14.5 points), competitive with the others.

On general reasoning, results are more variable. GPQA-Diamond (Figure 1): 74.49 versus 75.25 (Ring-Flash-2.0), 68.40 (Qwen3-32B), 82.80 (Gemini), 73.10 (GPT-OSS), 71.40 (Seed-OSS), 77.20 (Qwen3-Next). Notably behind Gemini-2.5-Flash (-8.31 points) and Qwen3-Next (-2.71 points), but ahead of Qwen3-32B and Seed-OSS. The large gap to Gemini is the most significant underperformance across any benchmark. SciBench: 5.13 versus 5.18 (Ring-Flash-2.0), 4.34 (Qwen3-32B), 4.11 (Gemini), 4.93 (GPT-OSS), 4.11 (Seed-OSS), 4.68 (Qwen3-Next) — all models score in the 4–5 range on this extremely challenging benchmark; differences are within ~1 point and likely not statistically meaningful. DROP: 89.66 versus 83.88 (Ring-Flash-2.0), 86.52 (Qwen3-32B), 84.16 (Gemini), 72.90 (GPT-OSS), 91.17 (Seed-OSS), 92.05 (Qwen3-Next). Ring-flash leads Ring-Flash-2.0 by ~5.8 points and is competitive with the leading Qwen3-Next (-2.4 points). MuSR: 84.05 versus 85.11 (Ring-Flash-2.0), 78.21 (Qwen3-32B), 84.98 (Gemini), 82.57 (GPT-OSS), 82.06 (Seed-OSS), 81.03 (Qwen3-Next) — competitive at the top of the group. Multi-LogiEval: 75.21 versus 76.18 (Ring-Flash-2.0), 77.01 (Qwen3-32B), 77.97 (Gemini), 73.96 (GPT-OSS), 77.90 (Seed-OSS), 72.62 (Qwen3-Next) — mid-pack.

Overall pattern for Ring-flash-linear-2.0: The model's performance is best characterized as competitive with frontier reasoning models while being a hybrid linear architecture, but not clearly dominant across the board. It essentially ties its softmax-attention counterpart (Ring-Flash-2.0) on most benchmarks, with differences typically within 0–2 points and no consistent direction — this validates that the architecture conversion causes negligible capability regression. Against external baselines, it leads on CodeForces, is competitive on AIME'24, OlympiadBench, LiveMathBench, Humaneval+, and MuSR, and notably trails on GPQA-Diamond (vs. Gemini), AIME'25 (vs. Qwen3-Next), and CNMO'24 (vs. Seed-OSS). The claim of "SOTA performance" (Section 1) is conditional: the model is state-of-the-art among hybrid linear architectures and competitive with the best softmax-attention models at similar scale, but it is not the single best model on any benchmark where a direct comparison is available (Qwen3-Next, Seed-OSS, and Gemini each lead on multiple benchmarks). The paper's positioning is more accurately that the hybrid architecture achieves efficiency gains without meaningful quality sacrifice, not that it surpasses all alternatives in quality.

Inference Efficiency: Prefill and Decode Throughput

Figures 7 and 8 provide the core evidence for the paper's efficiency claims. The measurements control for hardware (1× H20 for Ring-mini, 4× H20 for Ring-flash), batch size (1 for prefill, 64 for decode), and framework (SGLang), normalizing throughput to a dense baseline (Qwen3-8B or Qwen3-32B = 1.0).

For Ring-mini-linear-2.0 (Figure 7), normalized prefill throughput (Figure 7a) at context length 4K: approximately 0.7–0.8× baseline — meaning the hybrid linear model is slightly slower than the baseline dense model at short contexts. At 8K: ~1.0–1.2×. At 32K: ~4× versus Ring-mini-2.0 at ~2× and baseline at 1×. At 128K: ~14× versus Ring-mini-2.0 at ~6×. At 512K: the curve continues to rise (not shown numerically, but visually well above 14×). The crossing point where Ring-mini-linear-2.0 overtakes Ring-mini-2.0 is approximately at context length 8K, consistent with the paper's statement that linear attention advantages "only become pronounced at sequence lengths beyond 8K."

Normalized decode throughput (Figure 7b) at generation length 4K: ~4× baseline, Ring-mini-2.0 at ~3×. At 16K: ~10× baseline, Ring-mini-2.0 at ~5×. At 64K: ~16× baseline, Ring-mini-2.0 at ~7×. The paper states Ring-linear "delivers more than twice the throughput of Ring-2.0 and exceeds baseline performance by over tenfold" at 64K, which matches the figure: ~16× / 1× = 16× over baseline, and ~16× / ~7× ≈ 2.3× over Ring-mini-2.0.

For Ring-flash-linear-2.0 (Figure 8), normalized prefill throughput (Figure 8a) at 4K: all models cluster around 1–1.5× baseline. At 32K: Ring-flash-linear-2.0 ~3.5×, Ring-flash-2.0 ~2×, Qwen3-Next ~2×, baseline 1×. At 128K: Ring-flash-linear-2.0 ~8×, Ring-flash-2.0 ~4×, Qwen3-Next ~3.5×. Normalized decode throughput (Figure 8b) at 4K: Ring-flash-linear-2.0 ~2.5×, Ring-flash-2.0 ~2×, Qwen3-Next ~1.5×. At 32K: Ring-flash-linear-2.0 ~7×, Ring-flash-2.0 ~3.5×, Qwen3-Next ~3×. At 64K: Ring-flash-linear-2.0 ~9–10×, Ring-flash-2.0 ~5×, Qwen3-Next ~4×.

The comparison with Qwen3-Next-80BA3B (which also uses hybrid linear attention) is particularly informative: Ring-flash-linear-2.0 achieves roughly 2–2.5× higher decode throughput at 64K despite both being hybrid linear architectures. The paper attributes this to custom kernel fusion and optimization (Section 3.4) rather than fundamental architectural differences, implying that naive hybrid linear implementations leave substantial efficiency on the table that can be recovered through engineering.

FP8 Training Throughput

Figure 6 reports training throughput speedups from the linghe kernel library and associated optimizations. For Ring-mini-linear-2.0 training on 32 H800 GPUs: the baseline Megatron FP8 blockwise configuration achieves a normalized throughput of 1.0; adding fused kernels alone yields +21% (1.21×); fused kernels plus the ability to drop tensor parallelism from TP=2 to TP=1 (due to memory savings) yields +77% total (1.77×). For Ring-flash-linear-2.0 training on 288 H800 GPUs: baseline at 1.0; fused kernels yield +25% (1.25×); fused kernels plus enabling MBS increase from 1 to 2 and pipeline reconfiguration from PP=6/VPP=2 to PP=8/VPP=1 yield +57% total (1.57×). These are end-to-end throughput measurements for the full training pipeline (forward + backward + optimizer step), not just kernel microbenchmarks.

Continued Pre-Training Capability Recovery

Figure 9 reports normalized performance of Ring-linear-base-2.0 relative to Ling-base-2.0 (the dense softmax-attention models from which they were initialized) across six capability categories after continued pre-training of 600B tokens (mini) or 1T tokens (flash).

For Ring-mini-linear-base-2.0 (Figure 9a): NLU normalized score ~0.99 (essentially full recovery), Math ~0.98, Code ~0.98, Reasoning ~0.96, Basic Knowledge ~0.99, Professional Knowledge ~0.96. The paper states the models "restore more than 98% of the original models' performance in most categories," with the exceptions being Reasoning and Professional Knowledge at ~96%.

For Ring-flash-linear-base-2.0 (Figure 9b): NLU ~0.98, Math ~0.98, Code ~0.985, Reasoning ~0.96, Basic Knowledge ~0.99, Professional Knowledge ~0.97. The pattern is similar, with Reasoning and Professional Knowledge showing the largest regressions. The paper attributes this to "knowledge forgetting" during continued pre-training, compounded by the larger model having more parameters to adapt.

Reinforcement Learning Training Stability

Figures 10, 12, and 13 provide evidence for the training-inference alignment methodology's effectiveness.

Ablation of alignment components (Figure 10): The training reward over RL steps is plotted for five configurations: Original (unaligned), Fix KVCache & LM_Head, Fix KVCache & LM_Head & RMSNorm, Fix KVCache & LM_Head & RMSNorm & Attention, and Fix KVCache & LM_Head & RMSNorm & Attention & RoPE. The original configuration shows reward starting around 0.25 and growing slowly with high variance, reaching perhaps 0.35–0.40 by the end of training with substantial fluctuations. Each additional aligned module incrementally increases the final reward (roughly 0.45 after KV+LM fix, 0.50 after +RMSNorm, 0.55 after +Attention, 0.60+ after +RoPE) and visibly reduces the variance (the curves become smoother). The fully aligned configuration achieves a reward of approximately 0.60–0.65 by the end, with monotonic growth and minimal fluctuations. The paper does not report exact numerical values at each step, only the visual trend.

PPO clipping strategy comparison (Figure 12): The left panel compares training reward for two approaches: "Clipping with Rollout Probs" (Equation 5, using inference engine probabilities directly) and "Clipping with Training Probs" (Equation 6, recomputing probabilities through the training engine). Both start from similar reward levels (~0.15 at early steps), but the rollout-probability approach achieves higher rewards in the later stages of training (roughly 0.55–0.60 vs. 0.45–0.50 for training-probability clipping, by the end of the plotted steps). The right panel shows the proportion of tokens with absolute training-inference probability difference >0.8, a direct measure of training-inference disparity. The rollout-probability approach maintains this proportion at near-zero throughout training (the curve stays close to the x-axis), while the training-probability approach shows the proportion rising to approximately 0.0004–0.001 (0.04–0.1% of tokens) by the end of training, with a generally increasing trend.

RL training curves for Ring-mini-linear-2.0 (Figure 13): Three panels show consistent improvement over training steps. Training reward (Figure 13a): rises from ~0.54 to ~0.64. AIME'25 test score (Figure 13b): rises from ~0.70 to ~0.74. LiveCodeBench test score (Figure 13c): rises from ~0.58 to ~0.62. All three curves show monotonic improvement without degradation or collapse. The paper presents these as evidence that the alignment methodology enables "long-horizon stable RL training" that translates training reward improvements into held-out benchmark improvements.

Ablation Studies and Robustness Checks

• Layer group size (M) via scaling laws (Figure 3, right panel): Comparing different $M$ configurations shows that larger group sizes (more linear attention layers between softmax blocks) perform better at high FLOP budgets, while smaller $M$ may be preferable at lower budgets. This ablation justifies the choice of $M=4$ for the 16B model and $M=7$ for the 104B model. The non-obvious finding is that the optimal $M$ is not fixed but depends on total compute budget — a scaling law rather than a universal architectural constant.

• Hybrid vs. pure softmax attention (Figure 3, left panel): Even a modest hybrid ratio ($M=1$, groups of 2 layers: 1 linear + 1 softmax) consistently achieves lower training loss than pure softmax attention across all FLOP budgets tested. This is a robustness check confirming that the hybrid architecture is not merely an efficiency compromise but genuinely improves loss scaling.

• Rotary Position Embedding (RoPE) application (Section 2.2.3): Applying RoPE to only half of the Q and K dimensions (partial RoPE) versus full-dimensional RoPE reduced training LM loss by approximately 0.004. The ablation is not visualized in a figure but is reported quantitatively.

• Head-wise decay schedule (Section 2.2.3): Using power-law decay for assigning decay coefficients across attention heads versus linear decay reduced training LM loss by approximately 0.04 — an order of magnitude larger effect than the RoPE ablation. The paper notes this also had significant downstream task impact, making it one of the most consequential architectural decisions.

• Grouped RMSNorm for tensor parallelism (Section 2.2.3): Replacing standard RMSNorm (which requires all-reduce communication under TP) with grouped normalization (each rank normalizes its shard independently) eliminates the communication overhead. The paper describes this as a design choice rather than a controlled ablation, but the downstream effect is that it enables the TP=1 configuration that contributes substantially to the 77% training throughput improvement (Figure 6).

• Training-inference alignment components (Figure 10): As described above, fixing each module (KV cache, LM head, RMSNorm, Attention, RoPE) incrementally improves RL training reward and stability. The non-obvious result is the cumulative nature: fixing only one or two modules provides some benefit, but the full stability (monotonic reward growth with low variance) requires all five to be aligned. No single fix is sufficient.

• Rollout vs. training probabilities in PPO (Figure 12): After systematic alignment, using rollout probabilities directly (Equation 5) outperforms recomputing through the training engine (Equation 6) in both final reward and training-inference disparity control. This ablation confirms that the alignment is sufficient to make the theoretically correct PPO formulation practical. The non-obvious result is that the recomputed-probability approach, which is standard practice (verl, OpenRLHF), actually shows growing training-inference disparity over training steps despite recomputing probabilities on the same engine — suggesting that the recomputation itself introduces distortion, possibly due to prefill-decode attention differences.

• Ring-mini-linear-2.0 vs. Ring-Mini-2.0 benchmark comparison (Table 2): Across 17 benchmarks, the hybrid linear model performs within 0–5 points of its softmax-attention counterpart on every benchmark, with no systematic direction (sometimes slightly ahead, sometimes slightly behind). This serves as a robustness check for the continued pre-training procedure: the architecture conversion does not cause catastrophic capability loss. The largest negative gaps are on DROP (-5.35 points) and LiveCodeBench (-3.03 points); the largest positive gaps are on SciBench (+0.72, though all scores are very low) and MuSR (+1.22).

• Ring-flash-linear-2.0 vs. Ring-Flash-2.0 benchmark comparison (Table 3): The same pattern at larger scale: differences are typically within 0–2 points with no consistent direction. The largest gap is on DROP where Ring-flash-linear leads by +5.78 points — a notable positive result suggesting that the hybrid architecture may actually improve certain reading comprehension capabilities. The paper does not discuss this specific result or hypothesize why it occurs.

• Negative result: ReST$^{EM}$ training degradation (not reported in main text): The paper mentions in Section 5.2.2 and Figure 12 that the standard practice of recomputing training probabilities leads to growing training-inference disparity and lower reward — a negative result for the default verl/OpenRLHF approach when applied to long-output MoE models without alignment. Additionally, the paper notes in Section 5.2 that "smaller windows (e.g., 32K) introduce potential limitations, namely a high truncation rate and a lower performance ceiling" during RL training — a negative result for training with insufficient context length.

Critical Assessment

On the claim: "Systematic training-inference alignment enables stable long-horizon RL training"

The experiments provide evidence for this claim but with qualifications that limit the strength of the conclusion.

What is demonstrated: Figure 10 shows that incrementally aligning five modules (KV cache, LM head, RMSNorm, Attention, RoPE) improves RL training reward and reduces variance. Figure 13 shows that the fully aligned model achieves monotonic improvement in training reward and held-out test scores over the course of RL training without collapse. Figure 12 shows that after alignment, using rollout probabilities directly outperforms the standard training-probability recomputation approach.

What is not demonstrated: The paper does not compare the alignment-based approach against algorithmic mitigation methods from prior work (Zheng et al., 2025; Yao et al., 2025) on the same model and training setup. The claim that "no additional algorithmic modifications are necessary" (Section 5.2.2) is demonstrated only relative to an unmodified PPO baseline from verl/OpenRLHF that uses training-probability recomputation. It is not demonstrated that the alignment approach outperforms or is more practical than a well-tuned algorithmic mitigation strategy (e.g., GSPO from Zheng et al., 2025). The paper's ablation (Figure 10) compares increasingly aligned configurations to an "Original" baseline, but the baseline is not clearly described — it may already include some algorithmic mitigations, and the paper does not report what PPO hyperparameters were used or how they were tuned.

Statistical limitations: None of the RL training curves (Figures 10, 12, 13) include error bars, confidence intervals, or information about replicate runs. The number of training steps on the x-axes is not quantified, and the reward scale (0 to 1) is not defined (what is the reward function? Is it binary correctness? A shaped reward?). The improvement in Figure 13b from ~0.70 to ~0.74 on AIME'25 represents a shift from roughly 70% to 74% accuracy — a 4-percentage-point improvement. Without knowing the variance (across random seeds, across RL steps, or across checkpoint evaluations), it is unclear whether this improvement is statistically reliable or within the noise range of AIME evaluation (which typically has only 15–30 questions per test set). The paper does not report the number of AIME questions used for evaluation.

Generality concern: The alignment methodology is demonstrated on one model architecture (Ring-linear hybrid) with one training stack (Megatron) and one inference stack (vLLM/SGLang). The specific discrepancies identified (KV cache precision, RMSNorm epsilon, RoPE implementation, etc.) are plausible sources of error across many frameworks, but the paper does not provide evidence that the approach generalizes beyond this specific stack. Different training frameworks (FSDP, DeepSpeed) and inference engines (TensorRT-LLM, llama.cpp) may have different discrepancy profiles that require different fixes. The paper's contribution is a methodology (systematic activation comparison) rather than a universal solution, but the experiments only validate it for one configuration.

On the claim: "Hybrid architecture reduces inference cost to 1/10 of a 32B dense model and >50% vs. prior Ring series"

The inference efficiency measurements in Figures 7–8 support this claim at long context lengths (>64K), but the claim requires careful qualification regarding sequence length.

What is demonstrated: In Figures 7b and 8b, decode throughput at 64K generation length shows Ring-linear models at ~16× (mini) and ~10× (flash) the throughput of the baseline dense models. If we take throughput as inversely proportional to cost (higher throughput = lower cost per token), 16× throughput implies ~1/16 the cost per token, and 10× implies 1/10. The paper's "1/10 of a 32B dense model" claim is thus consistent with the flash model's decode throughput at 64K. A 50% reduction versus the Ring-2.0 series is also supported: at 64K decode, Ring-mini-linear-2.0 achieves ~16× vs. Ring-mini-2.0 at ~7× (a 2.3× throughput advantage, or ~57% cost reduction), and Ring-flash-linear-2.0 achieves ~10× vs. Ring-flash-2.0 at ~5× (a 2× throughput advantage, or 50% cost reduction).

What is not demonstrated or is conditional: The efficiency advantage is highly dependent on sequence length. At 4K context (Figures 7a, 8a), Ring-linear models are actually slightly slower than the baseline dense models (normalized prefill throughput <1.0) and only marginally faster during decode. The 1/10 cost claim applies specifically to long-context inference (64K+), and the paper does not provide a weighted average cost for a realistic workload mix. If most user requests are below 8K context (as the paper itself notes is common in pre-training), the average cost advantage could be substantially smaller. The paper does not report latency (time-to-first-token or time-per-output-token) at different batch sizes, only throughput at fixed batch sizes (1 for prefill, 64 for decode). For interactive applications with single requests (batch size 1), the throughput numbers may not translate directly to user-perceived latency.

Missing comparison: The 32B dense model comparison (1/10 cost) is mentioned in the abstract and Section 1, but a 32B dense model is not included in Tables 2–3 or Figures 7–8. The baseline models in the efficiency comparison are Qwen3-8B and Qwen3-32B. It is unclear whether the 32B dense model referenced is a specific model (e.g., Qwen3-32B without thinking, or a hypothetical 32B dense version of Ring) or an approximate scaling estimate. Without a direct FLOPs or cost comparison to a specific 32B dense model, the "1/10" figure is illustrative rather than rigorously validated.

On the claim: "SOTA performance across multiple challenging complex reasoning benchmarks"

The benchmark results in Table 3 partially support this claim but with important qualifications.

What is demonstrated: Ring-flash-linear-2.0 is competitive with frontier reasoning models across most benchmarks. On several (CodeForces, Humaneval+, LiveMathBench, MuSR), it achieves top or near-top scores. On others (AIME'24, AIME'25, OlympiadBench, LiveCodeBench), it is within the top tier but not the single best. The claim of "SOTA performance" in the plural (Section 1) can be interpreted as "among the state-of-the-art" rather than "unambiguously the best on all benchmarks," which is defensible given the results.

What is not demonstrated: The paper does not establish statistical significance for any of the benchmark comparisons. On AIME benchmarks with limited numbers of questions (AIME'24 and AIME'25 each have 15–30 questions), a difference of 1–2 percentage points is within sampling error and cannot reliably rank models. For example, on AIME'25, Ring-flash-linear-2.0 scores 86.51 versus Qwen3-Next at 87.80 — a gap of 1.29 points on a test set where each question is worth 3–7 percentage points. This difference could easily reverse with a different set of questions or a different random sampling seed. The paper does not report the number of evaluation samples per benchmark, making it impossible to assess the reliability of the rankings.

Missing baselines: The paper compares against several frontier models but omits others that would provide important context: DeepSeek-R1 (Guo et al., 2025), OpenAI O-series models, and Claude models are not included in the comparison. While Table 3 already has six baselines — a reasonable comparison set — the absence of DeepSeek-R1, which is widely considered SOTA on reasoning benchmarks and also uses efficient attention (MLA), is a notable gap for a paper claiming SOTA reasoning performance. The paper does not explain the baseline selection criteria.

Internal consistency check: Across all 17 benchmarks in Table 3, Qwen3-Next-80BA3B-Thinking achieves the highest score on 10 benchmarks (AIME'24, AIME'25, OlympiadBench, LiveCodeBench, MBPP+, Spider, BFCL-Live, GPQA-Diamond, DROP, Multi-LogiEval). Ring-flash-linear-2.0 leads on 3 (CodeForces, Humaneval+, MuSR), and Seed-OSS-36B leads on 2 (CNMO'24, TheoremQA). Ring-Flash-2.0 leads on 2 (LiveMathBench, SciBench). A casual summary would be that Qwen3-Next is the strongest model in this comparison group, with Ring-flash-linear-2.0 competitive but not dominant. The paper's framing in Section 1 ("consistently maintaining SOTA performance") overstates the case relative to the evidence — "consistently competitive with SOTA models" would be more accurate.

On the claim: "FP8 kernel library improves training efficiency by 50%"

Figure 6 supports this claim with specific configuration comparisons.

What is demonstrated: For Ring-flash-linear-2.0, the full optimization (fused kernels + increased MBS + pipeline reconfiguration) achieves a 57% throughput improvement over the Megatron baseline. For Ring-mini-linear-2.0, the full optimization achieves a 77% improvement. These are end-to-end training throughput measurements on a fixed hardware budget.

What is conditional: The 77% improvement for Ring-mini-linear-2.0 depends primarily on being able to drop tensor parallelism from TP=2 to TP=1, which is enabled by memory savings from kernel fusion. If the baseline had already used TP=1 (which might be possible with different model parallelism strategies or larger GPU memory), the improvement from fused kernels alone is 21%. The 57% improvement for Ring-flash-linear-2.0 similarly depends on increasing MBS from 1 to 2 and reconfiguring pipeline parallelism. The fused kernels alone provide 25% improvement. The paper presents the full-stack optimization as a package, but the largest gains come from system configuration changes enabled by memory savings, not from raw kernel speed improvements. This is not a weakness — it highlights the value of co-designed kernels and training configuration — but the "50% improvement" claim does not disentangle kernel performance from parallelism efficiency gains.

On the claim: "Hybrid architecture determined through scaling laws outperforms pure softmax at all FLOP budgets"

Figure 3 (left panel) directly supports this claim: the hybrid linear curve is consistently below (lower loss) the softmax attention curve across all FLOP budgets plotted. This is a robust finding that does not depend on specific hyperparameters.

Qualification: The scaling law curves are fitted to training runs at specific model sizes (not specified in Figure 3, but presumably at the scale of the Ring-mini or Ring-flash architecture). The paper does not explore whether the advantage holds at much smaller scales (hundreds of millions of parameters) or much larger scales (hundreds of billions). The right panel shows that the optimal layer group size depends on FLOP budget, implying that the advantage of hybrid over pure softmax might also depend on scale. If the advantage diminishes at very large scales (e.g., when softmax attention's overhead becomes a smaller fraction of total compute due to MoE dominance), the "all FLOP budgets" claim might not extrapolate. The paper does not explore the asymptotic behavior.

On the continued pre-training recovery claim

Figure 9 shows >98% performance recovery in most categories. The evidence is visual (bar charts with normalized scores) and the paper does not provide exact numerical recovery percentages per category, only "in most categories" and highlighting Reasoning and Professional Knowledge as exceptions.

Missing quantification: The paper does not report the absolute benchmark scores for Ling-base-2.0 and Ring-linear-base-2.0, only normalized ratios. Without knowing the baseline performance, it is impossible to assess whether a 96% recovery in Reasoning (apparently the worst category) represents a drop from 50% to 48% (acceptable) or from 10% to 9.6% (where a 0.4-point difference may be noise). The paper also does not report whether the normalized scores in each category represent a single benchmark or an aggregate of multiple benchmarks, making it difficult to assess the robustness of the recovery measurement.

Potential confounding factor: The continued pre-training uses the WSM learning rate scheduler (Tian et al., 2025b) instead of WSD (Hu et al., 2024b). The paper does not report whether the normalization baseline (Ling-base-2.0) was trained with WSD, or whether WSM would have improved or degraded the baseline performance if applied there. The scheduler change is a confounding variable in the comparison.

Missing experiments

Several experiments would strengthen the paper's claims:

• Ablation of hybrid ratio on downstream benchmarks, not just training loss. The scaling law experiments (Figure 3) measure training loss, which does not always correlate perfectly with downstream task performance. An ablation showing that $M=7$ vs. $M=4$ affects benchmark scores (not just loss) would connect the architecture design directly to the evaluation results.

• Inference efficiency at mixed batch sizes and with speculative decoding enabled. Figures 7–8 show throughput at fixed batch sizes without speculative decoding (the MTP layers are noted as disabled for efficiency measurements). Real-world serving involves varying batch sizes, and the paper develops speculative decoding support for hybrid linear models but does not benchmark it. Including speculative decoding throughput numbers would strengthen the practical deployment case.

• Statistical error analysis for benchmark results. Running each benchmark multiple times with different random seeds or bootstrap resampling of test questions would provide confidence intervals and distinguish meaningful differences from sampling noise, especially for small benchmarks like AIME.

• Comparison of alignment methodology against algorithmic mitigation. Training an RL run with a state-of-the-art algorithmic approach to training-inference disparity (e.g., GSPO from Zheng et al., 2025) on the same model and comparing stability/reward against the alignment approach would directly test the paper's claim that alignment is sufficient and algorithmic mitigation is unnecessary.

• Evaluation of pure linear attention (M = ∞) at scale. The paper argues that hybrid is necessary because pure linear attention underperforms on retrieval. But it does not report a pure linear attention baseline at the 104B scale to quantify how much performance is lost. The scaling law fits (Figure 3) include hybrid configurations but not a pure linear configuration, so the performance gap between $M=7$ and $M=\infty$ is unmeasured.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in Efficiency Claims

The assumption or constraint. The entire compute-optimal allocation framework depends on the ability to estimate prompt difficulty before deciding how to spend the inference budget. The paper's approach for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — constitutes an enormous upfront computation that is not included in any of the reported efficiency numbers. The authors explicitly acknowledge this in Section 3.2:

"We estimate the difficulty of a question... by sampling 2048 solutions per question and using either the ground-truth pass@1... or the PRM's predicted average final-answer correctness... This difficulty estimation step unfionrtunately still incurs additional computation cost during inference, and our experiments do not account for this cost largely for simplicity."

This is not a minor accounting oversight. At 2048 samples per question, the difficulty estimation cost exceeds the largest test-time budgets studied (256–512 generations). For a deployment processing a single query, the total cost would be 2048 + (strategy budget), not just the strategy budget — potentially making the compute-optimal approach more expensive than a simple best-of-N baseline with the same total budget.

The consequence. The headline 4× efficiency gains (e.g., "matching best-of-N weighted at 64 generations with only 16 generations" in Figure 4; "matching parallel best-of-N weighted at 256 generations with only 64 generations" in Figure 8) are computed after difficulty is already known, without amortizing the cost of learning it. If the difficulty estimation cost were included, the "optimal" strategy might simply be to use those 2048 samples as a standard best-of-N — especially since the paper shows that best-of-N weighted with 2048 samples already achieves strong performance (Figure 14, ~40% accuracy). The 4× figure should be understood as an upper bound on achievable efficiency that becomes realizable only when difficulty estimation can be done cheaply — a capability the paper does not demonstrate.

The problem becomes even more acute in the predicted (non-oracle) difficulty setting. While the paper shows that PRM-based predicted difficulty bins perform similarly to oracle bins (the curves largely overlap in Figures 4 and 8), generating and scoring 2048 samples through the PRM is still enormously expensive — in fact, it requires the very search infrastructure the paper is trying to optimize. The PRM must score every step of all 2048 candidate solutions, which for beam search or revision trajectories multiplies the cost further.

What evidence exists in the paper. Section 3.2 explicitly flags this issue. The curves in Figures 4 and 8 show that compute-optimal scaling with predicted difficulty works nearly as well as with oracle difficulty — but neither set of curves accounts for the cost of generating the difficulty estimate. The paper does not report the total cost (difficulty estimation + strategy execution) for any configuration, nor does it compare this total cost against a baseline that simply uses all available compute for best-of-N or beam search without difficulty estimation.

Mitigation status. The paper acknowledges this limitation and frames it as an open problem:

"This is still expensive; future work could explore more efficient difficulty estimation methods... such as pre-training or fine-tuning models to directly predict the difficulty of a question from its text."

No such method is developed or evaluated. The exploration-exploitation tradeoff — compute spent assessing difficulty versus compute spent solving the problem — is mentioned but not quantitatively analyzed. The difficulty estimation cost is the single largest unaccounted overhead in the paper's framework, and until it is addressed (either through cheap difficulty estimators or adaptive strategies that estimate difficulty as part of the solution process), the 4× efficiency claim operates in a closed-world where difficulty is magically known.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is fine-tuned exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). This means the model never sees examples of what to do when the current answer is already correct — it has no training signal for "stop revising, the answer is right." At test time, when the model occasionally produces a correct answer early in the revision chain, it may encounter that correct answer in its own context (since revisions condition on prior outputs) and treat it as an incorrect answer to be fixed, causing a regression to a wrong answer.

The paper reports a concrete measurement: approximately 38% of correct answers get converted back to incorrect ones using a naive approach of taking the final revision output (Section 6.1). This means that for every 100 correct answers produced during a revision chain, about 38 of them are subsequently "revised" into errors — a substantial and self-defeating error source.

The consequence. The 38% reversion rate imposes an upper bound on how much sequential revision can improve performance. Each additional revision step has a chance of overwriting a correct answer that was already found, creating a performance ceiling that pure "more revisions" cannot break through. In the extreme, if every revision step has a probability $p$ of converting a correct answer to incorrect, then as the chain length grows, the final output accuracy approaches some limit well below 100%. The paper's Figure 6 (left) shows pass@1 at each step gradually leveling off around 23–25% by steps 15–20 and remaining flat out to 64 steps — consistent with the reversion rate acting as a ceiling.

In practice, this ceiling means that simply increasing the sequential revision budget does not reliably improve accuracy. The paper's compute-optimal allocation for revisions (Figure 8) still shows improvement with increasing budget, but this is partly because the compute-optimal policy uses parallel sampling as well (the hybrid sequential-parallel strategies), and the verifier-based selection across chains mitigates the reversion problem by picking the best answer from anywhere in each chain rather than the last answer. The 38% reversion rate means that pure sequential strategies are inherently limited, and the gains from revisions come as much from diversity (multiple chains) as from iterative improvement.

What evidence exists in the paper. The 38% figure is reported in Section 6.1, though the exact measurement methodology (what model, what revision depth, on what dataset) is not detailed. Figure 6 (left) shows the flattening of per-step accuracy at higher revision depths. The paper's mitigation — majority voting or verifier-based selection across the chain rather than taking the last revision — is described but the residual reversion rate under those mitigations is not reported.

Mitigation status. The paper partially mitigates this through selection mechanisms: "the system uses a selection mechanism (majority voting or verifier-based selection) across the entire chain of revisions, picking the best answer from any point in the chain rather than always taking the last revision." This reduces the impact but does not fix the root cause — the model will still waste revision steps converting correct answers to incorrect ones, consuming compute budget without adding value. A more principled solution would be to train the model with both correct-incorrect-correct trajectories (teaching it to recognize and preserve correct answers) or to add an explicit "stop revision" token that the model learns to emit when no further improvement is needed. The paper does not explore these directions.

The ReST$^{EM}$ experiment (Appendix K and Figure 16) further demonstrates the fragility of revision training: attempting to optimize the revision model with RL-style data generation caused performance to degrade substantially with sequential revisions, suggesting the correct-to-incorrect reversion problem may be amplified rather than resolved by more training. This is a hard limitation that the paper identifies but does not solve.

---

The Method Provides Zero Benefit on Hardest Problems, Which Is Exactly Where It Would Be Most Valuable

The assumption or constraint. Test-time compute scaling — both search against verifiers and iterative revisions — works by amplifying the base model's existing capability: it helps find correct solutions that the model can already produce at some non-trivial rate but fails to identify among its generated candidates. Section 5.2 (search) and Section 6 (revisions) both demonstrate that the approach is ineffective when the base model's pass@1 on a problem is near zero.

The difficulty analysis reveals this starkly:

• Search (Figure 3, right): On difficulty bin 5 (hardest questions), beam search and best-of-N both hover at 1–3% accuracy regardless of budget from 4 to 256 generations. Neither method makes any meaningful progress.
• Revisions (Figure 7, right): On bin 5, accuracy remains at 2–3% for all sequential-to-parallel ratios at 128 generations.
• FLOPs-matched comparison (Figure 9): On the hardest difficulty bins (4–5 in the bar charts, bin 5 in the line plots), test-time compute with the smaller model shows negative relative improvement against the larger model at most $R$ values — meaning the smaller model + extra test-time compute is worse than just training a bigger model and using greedy decoding.

As the paper acknowledges:

"On the hardest questions, no amount of test-time compute helps — the base model simply cannot produce correct solutions regardless of how the budget is allocated."

And in the Section 7 takeaways:

"Test-time compute does not appear to be an effective substitute for pretraining on problems that are outside the base model's capability range."

The consequence. This is a fundamental limitation of the approach. The hardest problems — competition math problems that require novel reasoning strategies, tasks requiring knowledge the model lacks, problems where the base model's typical outputs are structurally far from any correct solution — are exactly the problems where one would most want additional compute to help. The paper shows that this is precisely where it fails. Test-time compute can make a model more reliably correct on problems it broadly understands, but it cannot make the model correct for the first time on problems it fundamentally misunderstands.

This limitation is not unique to this paper — it follows logically from the framework of modifying the proposal distribution or selecting among candidates. If no candidate is correct, no selection mechanism can find a correct answer, and no amount of iterative revision can transform an incorrect answer into a correct one if the model lacks the underlying knowledge. However, the paper's finding that beam search degrades accuracy on easy problems at high budgets (Figure 3, right, bins 1–2) adds a further twist: the approach is simultaneously harmful on easy problems (due to verifier over-optimization) and ineffective on hard problems, leaving only medium-difficulty problems as the sweet spot where test-time compute is unambiguously beneficial.

What evidence exists in the paper. The difficulty-bin analyses cited above (Figures 3 right, 7 right, 9) provide direct evidence. The bin-5 flatlines across all methods, budgets, and strategies are among the most robust and replicated findings in the paper — they appear in search experiments, revision experiments, and the FLOPs-matched comparison. The error bars are not reported, but the pattern is so consistent (always 1–3% regardless of method or budget) that it is clearly a genuine limitation rather than noise.

Mitigation status. The paper is transparent about this limitation and does not attempt to claim otherwise. Section 8 (conclusion) does not propose any solutions for hard problems. This is a limitation that likely requires fundamentally different approaches — curriculum learning, retrieval-augmented generation, explicit knowledge injection, or architectural improvements — that fall outside the scope of test-time compute optimization. The paper's contribution is characterizing where the boundary is, not overcoming it.

---

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*), Leaving Generalization Unproven

The assumption or constraint. Every experiment in the paper — the search algorithm comparisons, the revision model analysis, the compute-optimal scaling results, the difficulty-bin behavior, the verifier over-optimization characterization, the FLOPs-matched pretraining comparison — is conducted on the MATH benchmark (Hendrycks et al., 2021) using PaLM 2-S* (Codey) as the base model. The specific split used is that from Lightman et al. (2022): 12,000 training questions and 500 test questions.

The paper defends this choice in Section 4, arguing that MATH is an appropriate testbed because "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences," and that PaLM 2-S* is "representative of the capabilities of many contemporary LLMs." However, neither of these claims is empirically validated within the paper. In fact, several of the paper's specific findings suggest that the results may be very sensitive to the choice of model and benchmark.

The consequence. Multiple findings in the paper could be specific to the MATH-PaLM combination rather than general phenomena:

• The PRM's difficulty-dependent behavior (helping on medium problems, over-optimizing on easy problems) depends on the PRM's calibration properties, which are a function of the training data (Monte Carlo rollouts from PaLM 2-S*) and the base model's error distribution. A base model with different types of errors (e.g., primarily factual rather than reasoning errors) might produce a PRM with very different difficulty-dependent scaling curves.

• The revision model's 38% correct-to-incorrect reversion rate is a function of how the base model's correct and incorrect answers relate structurally. A model that produces qualitatively different types of errors might show different reversion behavior. The edit-distance-based pairing strategy for training data construction (Section 6.1) assumes structural similarity between incorrect and correct answers — this may not hold for all model families or task types.

• The optimal layer group size in the hybrid architecture (M = 7 for the larger model, M = 4 for the smaller model) was determined through scaling law fits on training loss. The optimal ratio likely depends on the relative importance of retrieval vs. sequential processing, which may differ across domains (code generation emphasizes retrieval of function definitions and variable bindings; mathematical reasoning emphasizes step-by-step deduction; dialogue emphasizes local coherence).

• The finding that beam search hurts easy problems depends on the specific failure modes of the trained PRM. A PRM trained on a different model's outputs or with a different methodology might exhibit over-optimization at different difficulty levels or budgets.

• The FLOPs-matched pretraining comparison uses a specific inference-to-pretraining token ratio $R$ and a specific 14× scaling factor. The tradeoff curves depend on the base model's scaling properties, which are PaLM-specific.

Most critically, MATH consists exclusively of competition-level mathematics problems with exact ground-truth answers. Many of the paper's core mechanisms depend on this: the PRM is trained using Monte Carlo rollouts that check against ground-truth answers (Section 5.1 and Appendix D: "Compute the fraction of rollouts that reach the correct final answer"); the difficulty bins are defined using pass@1 rates that require knowing which answers are correct; even the compute-optimal policy evaluation uses ground-truth answers via the MATH grading function (Appendix G). It is not obvious how to extend these mechanisms to tasks without clean correctness signals — code generation with unit tests is the closest analog, but open-ended generation, summarization, translation, dialogue, and creative writing pose qualitatively different challenges.

What evidence exists in the paper. None — the paper contains no out-of-domain evaluation, no cross-model replication, and no analysis of how findings might transfer to other domains. The authors' claim that PaLM 2-S* is "representative" (Section 4) is an assertion, not an empirical finding. A limited transfer check exists in Appendix K (the ReST$^{EM}$ experiment, Figure 16), but this is a negative result on the same model and dataset rather than a cross-domain generalization test. The paper acknowledges in Section 7 that "future work could explore more compute-optimal pretraining recipes" for the FLOPs-matched comparison, but does not discuss cross-domain generalization of the main findings.

Mitigation status. Not addressed. The paper makes no claims about generalization beyond MATH and PaLM 2-S*, but the findings are presented in general terms ("compute-optimal test-time scaling," "verifier over-optimization as a bottleneck") that implicitly suggest broader applicability. A domain limitation statement and discussion of which findings are likely to transfer versus which are likely MATH-specific would clarify the scope of the contributions. The paper's narrow scope is reasonable for a first systematic study, but it means that practitioners in other domains — code generation, scientific reasoning, dialogue agents, medical QA — have no direct evidence that the approach would work for them.

---

The 14× Larger Model Baseline Is Weakened by Parameter-Only Scaling and Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 asks: given a fixed total FLOPs budget, is it better to train a larger model or to keep the smaller model and spend the extra FLOPs on inference-time computation? The comparison is between PaLM 2-S* with compute-optimal test-time scaling and a model with approximately 14× more parameters, where the pretraining FLOPs of the larger model are matched by the combination of the smaller model's pretraining FLOPs plus additional inference FLOPs (accounting for the per-token cost).

The paper acknowledges a specific methodological choice:

"We fix training data and scale only model parameters when increasing pretraining compute, matching the approach of the LLaMA model series... They acknowledge that compute-optimal pretraining would scale both data and parameters equally (Hoffmann et al., 2022), and leave that comparison to future work."

Additionally, the larger model is evaluated using only greedy decoding — no majority voting, no best-of-N, no beam search, no test-time compute augmentation of any kind. The smaller model, by contrast, is given the full compute-optimal test-time scaling pipeline (search, revisions, or their adaptive combination) at a budget determined by the FLOPs matching equation.

The consequence. Both choices make the pretraining baseline weaker than it should be in a fair comparison. A compute-optimally trained larger model (scaling both parameters and data according to Chinchilla laws) would almost certainly perform better than a parameter-only-scaled model at the same total FLOPs budget. The pretraining FLOPs formula $X = 6ND$ is exact, but the mapping from FLOPs to loss depends on how those FLOPs are allocated between parameters and data — and the Chinchilla-optimal allocation (equal scaling of both) is known to produce lower loss than parameter-only scaling at a given budget. The paper's 14x larger model may thus be further from the pretraining Pareto frontier than an equivalently expensive model trained optimally.

Even more problematic is the greedy decoding baseline. By giving the larger model no test-time compute augmentation, the comparison is not "test-time compute vs. pretraining" but "test-time compute + small model vs. pretraining + no test-time compute." A fairer comparison would give the larger model some modest test-time compute budget as well — the FLOPs matching equation could account for this by allocating part of the larger model's inference budget to best-of-N or beam search. This is especially important because the paper's own results show that test-time compute provides substantial gains on easy-to-medium problems (the 4× efficiency improvement). If the larger model were allowed even best-of-8 (costing 8× its per-token inference cost), it might substantially close or reverse the gaps reported in Figure 9.

What evidence exists in the paper. Section 7 and Figure 9 present the comparison as described. The paper acknowledges the parameter-only scaling choice explicitly:

"This departs from compute-optimal pretraining... We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The greedy decoding baseline is not explicitly justified or acknowledged as a limitation; it is simply what the comparison uses.

The effect is visible in the results themselves. On medium-difficulty problems (bin 3 in Figure 9), the compute-optimal test-time scaling with the smaller model either matches or modestly exceeds the larger model's greedy performance. This is the regime where the paper draws its strongest conclusion — "test-time compute can substitute for pretraining." But the margin is narrow, and a larger model with even modest test-time compute (e.g., best-of-4) might reverse the finding. The paper does not test this.

Mitigation status. The parameter-only scaling choice is partially mitigated by the paper's explicit acknowledgment and flagging for future work. The greedy decoding baseline for the larger model is not acknowledged as a limitation. A sensitivity analysis — computing the FLOPs-matched comparison assuming the larger model uses best-of-K for various K, and seeing at what K the advantage of test-time compute disappears — would strengthen confidence in the conclusions. Without such analysis, the FLOPs-matched results should be interpreted as a lower bound on the value of pretraining (the pretraining baseline could be stronger) and an upper bound on the value of test-time compute relative to pretraining (the test-time compute advantage could be smaller or reversed with a fairer baseline).

---

The Revision Model and PRM Search Are Never Combined, Leaving Gains on the Table

The assumption or constraint. The paper studies two complementary mechanisms for test-time compute — modifying the proposal distribution through iterative revisions (Section 6) and improving candidate selection through PRM-guided search (Section 5) — but treats them entirely independently. Each is analyzed, optimized, and compared to baselines in isolation. The compute-optimal allocation policy selects between search strategies and revision strategies on a per-difficulty-bin basis but never combines them: there is no experiment where PRM beam search operates on revision model outputs, and no experiment where the revision model conditions on PRM feedback to guide its improvements.

The paper is explicit about this gap in Section 8:

"while we studied models that use both PRM search and revisions, we did not experiment with PRM tree-search techniques in combination with revisions."

This means the current results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary, difficulty-dependent strengths (revisions excel on easy problems where local refinement suffices; search excels on medium problems where broader exploration is needed), but whether their combination would be additive, multiplicative, or redundant is unknown.

The consequence. The paper's headline efficiency figures (4× over best-of-N) and the FLOPs-matched tradeoff analysis are computed for the best individual mechanism, not for the best combined system. If revisions + search together produce gains beyond either alone — which is plausible given their complementary strengths — then the efficiency advantage over pretraining could be substantially larger than reported. Conversely, if the mechanisms interfere (e.g., the PRM trained on base model outputs does not generalize to revision model outputs, which the paper partially demonstrates in Appendix J, Figure 15a), then the current analysis overstates what is achievable by treating them as separate options rather than a jointly optimized system.

The practical implication is that a practitioner reading this paper cannot determine the optimal deployment strategy for a system that has both a revision model and a PRM available. Should they run revisions first and then PRM search on the revised candidates? Use the PRM to guide revision trajectories (e.g., stopping a revision chain when the PRM score drops)? Run beam search where each leaf node conditions on prior rejected branches? The paper's framework provides no guidance.

Additionally, the compute-optimal allocation policy selects the best individual strategy per difficulty bin, but the truly optimal allocation might involve combinations: easy problems might benefit from sequential revisions followed by best-of-N verification; hard problems might benefit from PRM-guided search within each parallel revision chain. The current policy, which selects one strategy per bin, may leave gains on the table by treating the mechanisms as mutually exclusive.

What evidence exists in the paper. The revision model's PRM compatibility is partially tested in Appendix J (Figure 15a): the base-LM PRM underperforms a revision-specific ORM on revision model outputs, confirming distribution shift as a concern. However, this is a verifier issue (which verifier to use on revision outputs), not a mechanism combination issue (how to use search and revisions together). The paper does not test search over revision outputs, does not test revision models conditioned on PRM feedback, and does not test any joint allocation strategy.

Mitigation status. The paper acknowledges this as explicit future work in Section 8:

"Future work could explore combining these mechanisms, for instance by using the PRM to guide when to stop a revision chain or which revision branches to explore further."

No partial mitigation is provided. This is an understandable scoping choice — the paper is already dense with experiments — but it means the current results fundamentally underestimate the potential of the approach. A reader interested in deploying test-time compute should view the reported gains as achievable with either mechanism alone, with likely further improvements from combination, but with no empirical guidance on how large those improvements might be or how to realize them.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a fundamentally new attention mechanism or RL algorithm. Instead, it makes three contributions that collectively shift the conversation around efficient LLM deployment from "what architecture is theoretically optimal?" to "what architecture is practically optimal given the full engineering stack?" This is a methodological shift, not a paradigm shift — it changes how research should be conducted in this area rather than overturning established knowledge.

The first shift is the paper's demonstration that scaling law methodology should be applied to architecture design decisions, not just pretraining compute allocation. The field has long accepted that model size and data quantity should be determined through Chinchilla-style power-law fitting (Hoffmann et al., 2022). This paper extends that principle to the hybrid attention ratio: the optimal number of linear attention layers per softmax attention block is not a universal constant but a function of total training FLOPs (Figure 3, right panel). This means that architecture design — at least for the linear-vs-softmax attention tradeoff — should be an empirically determined outcome of scaling experiments, not a heuristic choice. The implication is that future work on hybrid architectures (linear attention, state-space models, gated linear units, etc.) should report FLOPs-matched comparisons across multiple architecture ratios, not just compare a single chosen configuration against a baseline. The choreography of "design architecture → train → evaluate" becomes "sweep architecture ratios at smaller scale → fit scaling laws → choose ratio → train at scale → evaluate," which requires substantially more experimental infrastructure but yields evidence-based rather than intuition-based design choices.

This also serves as a partial resolution to the contradictory evidence around hybrid architectures. Prior work showed that hybrid models work (Minimax M1, GPT-OSS, Qwen3-Next) but didn't explain why some ratios work better than others or whether the optimal ratio changes with scale. This paper's scaling law analysis provides a unifying explanation: different hybrid ratios dominate at different FLOP budgets, so heuristically chosen ratios will produce inconsistent results across model scales and training budgets. The apparent contradictions in prior work (some models benefit more from hybrid design than others) may simply reflect different positions on the scaling curve rather than fundamental disagreements.

The second shift — and the paper's most distinctive contribution — is the reframing of RL training stability for long-output models as an engineering discipline problem rather than an algorithmic one. Prior to this work, the dominant response to training-inference disparity in RLHF was algorithmic mitigation: use clipped importance sampling (standard PPO), recompute probabilities through the training engine (verl, OpenRLHF, Equation 6), develop off-policy corrections (Zheng et al., 2025; Yao et al., 2025). The implicit assumption was that numerical discrepancy between training and inference engines is an unavoidable cost of using specialized, optimized software stacks.

This paper makes a different diagnostic move: it argues that training-inference disparity is primarily correctable through systematic operator-level alignment — fixing KV cache precision, RMSNorm computation, RoPE implementation, attention backend consistency, and MoE routing determinism. Figure 10 demonstrates that each aligned module incrementally improves RL stability, and once alignment is complete, "no additional algorithmic modifications are necessary" (Section 5.2.2). The theoretically correct PPO formulation (using rollout probabilities, Equation 5) not only works but outperforms the standard recomputed-probability workaround (Equation 6, Figure 12). This changes the research priority: when RL training is unstable, the first diagnostic step should be activation-level comparison between training and inference engines, not tuning PPO hyperparameters or developing new clipping schedules.

The downstream effect is to make certain research directions more attractive and others less so. More attractive: systematic frameworks or tools for automated training-inference alignment checking (a "bit-exactness linter" for LLM frameworks), research on which specific operations are most prone to discrepancy (the paper's ablation provides a starting point: KV cache > LM head > RMSNorm > Attention > RoPE), and studies of whether alignment can be maintained across framework versions and hardware generations. Less attractive: developing increasingly complex off-policy PPO variants that accommodate discrepancy — the paper's results suggest this is solving the wrong problem, and that a few days of operator-level debugging can eliminate the need for algorithmic workarounds entirely. This does not mean algorithmic mitigation is useless — there may be irreducible discrepancies (e.g., from non-deterministic GPU operations) that cannot be eliminated — but it redirects effort toward understanding what can be aligned before developing algorithms to handle what cannot.

The third shift is more subtle: the paper demotes softmax attention from its status as the "default" attention mechanism to one option in a design space where hybrid architectures are not a compromise but an improvement. The left panel of Figure 3 shows that even a modest hybrid ratio ($M=1$, meaning one linear attention layer for each softmax attention layer) achieves consistently lower training loss than pure softmax attention across all FLOP budgets tested. This is not an efficiency-quality tradeoff — the hybrid architecture is genuinely better at language modeling, independent of efficiency considerations. If this finding holds across model scales and architectures (and the paper provides evidence only for the PaLM-derived Ring-linear models), it implies that pure softmax attention Transformers are Pareto-suboptimal: you can get better loss at the same compute by replacing some softmax layers with linear attention. This would be a significant architectural finding, equivalent in spirit to the discovery that GQA outperforms MHA at scale — not because GQA is more expressive, but because it allocates parameters more effectively. The paper's result is limited to one model family and scaling range, but if replicated, it would mean that future LLM architecture design should treat the hybrid attention ratio as a first-class hyperparameter alongside model width and depth, rather than defaulting to pure softmax or pure linear attention.

However, the paper does not fully resolve the question of why hybrid architectures outperform pure softmax. The stated rationale — that linear attention provides efficiency and softmax attention provides retrieval capability — is insufficient to explain why a 1:1 hybrid ratio outperforms pure softmax in terms of training loss. If softmax attention is strictly more expressive (it can implement exact token-level retrieval, which linear attention cannot), why does adding less expressive layers improve the overall model? Possible explanations (not discussed in the paper) include: linear attention provides a useful inductive bias for local processing that complements softmax attention's global retrieval; the hybrid architecture effectively increases model depth without proportionally increasing KV cache pressure; or the linear attention layers act as a form of regularization that prevents softmax attention from overfitting to spurious long-range correlations. This is an open question that the paper's empirical results surface but do not answer.

Follow-Up Research This Work Enables

Automated training-inference alignment diagnosis tools. The paper describes a manual, module-by-module process for identifying and fixing numerical discrepancies between training and inference engines (Section 5.2.1): provide identical inputs, compare activations layer by layer, trace discrepancies to their source (precision, nondeterminism, implementation difference). This process was applied to five specific modules (KV cache, LM head, RMSNorm, Attention, RoPE) across one training stack (Megatron) and one inference stack (vLLM/SGLang). A natural follow-up would be building a tool that automatically hooks into both engines' computation graphs and flags activation mismatches above a threshold, ideally with suggestions for the likely cause (e.g., "RMSNorm epsilon differs: training uses $10^{-5}$, inference uses $10^{-6}$" or "attention backend differs: training uses FlashAttention prefill kernel, inference uses decode kernel for earlier tokens"). The research question is whether such a tool could generalize across framework pairs (Megatron-vLLM, FSDP-TensorRT, DeepSpeed-vLLM) or whether each pair requires custom instrumentation. A strong follow-up would instrument three different training-inference pairs, report the number and severity of discrepancies found automatically vs. manually, and measure whether automated alignment produces RL training stability comparable to the paper's manual alignment. This would determine whether the alignment methodology scales to production settings where framework combinations change frequently.

Scaling law analysis of hybrid attention ratios across model families and domains. The paper's scaling law experiments (Figure 3) establish that the optimal layer group size $M$ depends on total training FLOPs for PaLM-derived architectures trained on the mix of data used for Ling-base-2.0. Three questions are left open. First, does the optimal $M$ depend on model architecture beyond scale? The paper uses $M=7$ for the 104B model ($d_{\text{model}}=4096$) and $M=4$ for the 16B model ($d_{\text{model}}=2048$). Is the larger $M$ optimal because the model is larger (more layers can be linear without hurting retrieval) or because the hidden dimension is larger (the $d \times d$ linear attention state has more representational capacity, so it can go longer without softmax "resets")? A controlled experiment training models at fixed total parameters but varying $d_{\text{model}}$ and measuring the optimal $M$ would disentangle these effects. Second, does the optimal ratio depend on training data distribution? The MATH-domain results suggest that retrieval capability is critical for mathematics (looking up theorems, definitions, earlier derivation steps). A code-generation training corpus might depend even more heavily on retrieval (function definitions, variable bindings), potentially shifting the optimal ratio toward more frequent softmax blocks. A comparison of optimal $M$ for models trained on math-heavy vs. code-heavy vs. general web-text corpora at the same FLOP budget would determine whether the hybrid ratio is a domain-dependent or universal architectural choice. Third, does the hybrid-vs-pure-softmax advantage persist at much larger scales? The paper's scaling law fits extrapolate to the training budgets used but not beyond; at 10× or 100× the FLOPs, the curves could cross or the advantage could saturate. Training a series of models at increasing scale with both pure softmax and optimal-hybrid configurations and fitting extrapolated scaling laws would establish whether hybrid architectures are a permanent improvement or a transient one at current scale.

The retrieval ceiling in hybrid linear models: quantifying and mitigating it. The paper states that "linear attention underperforms in retrieval" and that hybrid architectures "surpass the retrieval and extrapolation capabilities of the pure softmax attention architecture" (Section 2.2.2, citing Li et al., 2025). But the paper provides no direct measurement of retrieval capability — all downstream evaluations are on reasoning, coding, and QA benchmarks that mix retrieval with other skills. A targeted follow-up would construct a synthetic retrieval benchmark where the model must locate a specific fact, function definition, or value inserted at varying positions in a long context and use it to answer a query. By varying the context length and the distance between the inserted fact and the query, one could measure how retrieval accuracy degrades with distance for pure softmax, pure linear, and various hybrid ratios ($M=1, 3, 5, 7, \infty$). The hypothesis (suggested by the paper's architecture design) is that retrieval accuracy drops sharply when the query-to-fact distance exceeds the span from the last softmax attention block — the linear attention layers cannot perform exact token-level lookup, so information more than $M$ layers away from a softmax block becomes inaccessible. If this hypothesis holds, it implies a design principle: the hybrid ratio $M$ should be set so that the distance between softmax blocks (in terms of context tokens, not layers) is shorter than the typical retrieval distance in the target domain. A code model might need shorter $M$ than a math reasoning model because code generation involves frequent lookups of recently defined variables. This would convert the empirical scaling law finding into a principled, application-specific design rule.

Combining PRM-guided search with the revision model as the proposal distribution. The paper's Section 8 explicitly flags this gap: "we did not experiment with PRM tree-search techniques in combination with revisions." The paper provides separate evidence that revisions improve the proposal distribution (generating better candidates through iterative refinement) and that PRM search improves candidate selection (picking the best among generated candidates). The natural combination — using the revision model as the generator within beam search, or using the PRM to guide which revision branches to explore further — is an open experiment. A specific design: at each step of a revision chain, generate $k$ candidate revisions (top-k sampling), score each with the PRM, and keep the highest-scoring one for the next revision step (PRM-guided sequential revision). Alternatively: generate $N$ parallel revision chains, use beam search within each chain to explore revision branches, and select the best final answer across chains. The research question is whether the combined approach yields gains beyond the sum of individual gains — i.e., is the performance ceiling for combined search+revisions higher than the best of search-alone or revisions-alone? The paper's difficulty-dependent analysis suggests that combination might help most on medium-difficulty problems where both exploration (search) and refinement (revisions) are beneficial, but less on easy problems (where revisions alone suffice) or hard problems (where neither helps). A strong follow-up would replicate the compute-optimal scaling analysis (Figures 4 and 8) for a combined search+revision system and report whether the efficiency gain over best-of-N exceeds the 4× achieved by either mechanism alone.

Stress-testing training-inference alignment under distribution shift. The paper demonstrates that operator-level alignment produces stable RL training for Ring-linear models where training and inference run the same model architecture. But three real-world scenarios introduce distribution shift that may cause alignment to degrade. First, quantization: inference engines often run INT8 or FP8 quantized models while training uses BF16 or FP32. The paper's own FP8 training (linghe, Section 3.2) introduces quantization that is carefully matched between forward and backward passes, but inference-time quantization (e.g., GPTQ, AWQ) uses different quantization schemes that may not preserve bit-exactness with the training engine. Does alignment survive quantization? Second, speculative decoding: the paper developed tree-mask-compatible linear attention kernels for speculative decoding (Section 3.4), but does not report whether the alignment methodology extends to the draft-verification paradigm where tokens are proposed by a smaller model and verified by the large model. The probability distributions during speculative verification differ from standard autoregressive decoding (because the model sees multiple candidate tokens simultaneously). Third, model updates during online RL: the paper evaluates alignment at fixed checkpoints, but during online RL, the model parameters change between rollout generation and training — the rollout engine may be running a slightly stale checkpoint while the training engine uses the latest. A stress-test follow-up would measure training-inference probability disparity (as in Figure 12, right panel) across these three scenarios and determine whether the alignment methodology needs extension for each. The practical question is whether "align once at the start of RL training" is sufficient, or whether alignment must be continuously monitored and corrected.

Cheap difficulty estimation to make compute-optimal test-time scaling deployable. The paper's difficulty estimation method (2048 samples + PRM scoring) costs more than the test-time budgets being optimized, making the 4× efficiency gains unrealizable in deployment without amortization. The paper suggests "pretraining or finetuning models to directly predict difficulty of a question" (Section 3.2). A concrete follow-up: train a lightweight classifier (e.g., a small BERT-style model or an MLP on top of the frozen base model's embedding of the question text) to predict the difficulty quintile, using the 2048-sample pass@1 estimates as training labels on a held-out set of questions. Measure: (1) the classification accuracy of the difficulty predictor vs. the PRM-based method, (2) the compute cost of the predictor relative to 2048 PRM-scored samples, and (3) the downstream effect on compute-optimal allocation performance — i.e., does the predicted-difficulty compute-optimal curve (Figure 4, predicted bins) match the oracle curve when using the lightweight predictor instead of 2048 PRM samples? An alternative approach: adaptive difficulty estimation that starts with a small number of parallel samples (say, 4), uses the PRM score distribution on those samples as a quick difficulty proxy, and then allocates the remaining budget according to the estimated difficulty. This amortizes difficulty estimation into the solution process itself, potentially achieving the compute-optimal gains without additional upfront cost. The research question is how many initial samples are needed for reliable difficulty estimation, and whether the accuracy-vs-cost tradeoff favors offline estimation (one lightweight model, many queries) or online estimation (per-query sampling, no upfront cost).

Practical Applications and Downstream Use Cases

Long-context inference serving for reasoning models. The most directly actionable finding for practitioners is that the hybrid linear architecture with optimized kernels reduces decode cost at 64K generation length by approximately 10–16× compared to dense softmax attention baselines (Figures 7b, 8b), and by roughly 2× compared to the prior softmax-attention Ring-2.0 series, while maintaining competitive reasoning performance (Tables 2–3). For an organization serving a reasoning model that generates long chain-of-thought responses (thousands of tokens per query), switching from a softmax-attention architecture to the Ring-linear hybrid architecture at equivalent parameter scale would reduce per-query inference cost proportional to the throughput improvement — meaning a 10× cost reduction if generation lengths average 64K tokens. The key caveat is that this benefit only manifests at generation lengths beyond 8K (Figures 7b, 8b); for short-generation workloads (standard chat, simple QA), the throughput curves converge and the hybrid architecture may even be slightly slower (prefill throughput at 4K in Figure 7a is below the baseline). Practitioners should measure their actual generation length distribution before making the switch. This is a quantitative deployment decision that the paper's controlled throughput measurements at multiple sequence lengths (4K through 512K) enable, which generic claims about "linear attention is more efficient" do not support.

RL training infrastructure for long-output models. The paper's training-inference alignment methodology (Section 5.2.1) provides a concrete checklist for teams experiencing RL training instability with long-chain-of-thought models: (1) align KV cache precision (FP32 accumulation for recurrent states), (2) align LM head softmax computation (FP32 with register-level casting to avoid memory overhead), (3) align RMSNorm (FP32 computation, consistent epsilon, un-fuse residual), (4) align attention backend (same FlashAttention version, verify prefill-decode consistency), (5) align RoPE implementation (check rotation order, half-dimension handling, trig function computation). The paper's ablation (Figure 10) shows that each fix incrementally improves training stability, meaning teams can prioritize by checking these modules in order. The payoff is not just stability but also computational savings: after alignment, using rollout probabilities directly (Equation 5) eliminates the need to re-forward rollout data through the training engine — a 2× reduction in RL training compute cost since the inference engine already computed the needed probabilities. For a team running RL training on hundreds of GPUs for weeks, this is a substantial cost saving.

Cost-efficient batch inference for data generation pipelines. Organizations using LLMs to generate training data for self-improvement (STaR, ReST$^{EM}$, rejection sampling) or evaluation (scoring candidate solutions on MATH, generating code test cases) can use the Ring-linear architecture to process large batches of long-context prompts at substantially lower cost. The decode throughput advantage of ~10× at 64K (Figure 8b) means that generating 10,000 long reasoning traces costs roughly the same as generating 1,000 with an equivalent softmax-attention model. The paper's benchmark results (Tables 2–3) show that the hybrid architecture's generation quality is competitive with softmax attention at equivalent scale, so the cost savings do not come at the expense of data quality. The practical consideration is whether the data generation pipeline can batch requests to the batch size used in the paper's decode throughput measurement (64); at lower batch sizes, the throughput advantage may be smaller, and at batch size 1 (interactive use), latency rather than throughput becomes the bottleneck, which the paper does not measure.

On-device or edge deployment with the 16B model. Ring-mini-linear-2.0 has only 957M non-embedding active parameters (Table 1) — smaller than many dense models used in production — but achieves reasoning benchmark performance competitive with dense models having 3–5× more active parameters (Table 2 comparison with Qwen3-8B-Thinking). The combination of compact active parameters (low inference memory) and linear attention's constant KV state memory (Figure 4) makes this model particularly suitable for deployment scenarios with strict memory constraints: edge devices, consumer GPUs, or high-throughput serving where GPU memory limits the number of concurrent requests. At 64K context, the KV cache for a comparable softmax-attention model with 8B parameters might be ~2 GB (32 heads × 128 dimensions × 64K tokens × 2 bytes × 2 for K and V), while Ring-mini-linear-2.0's linear attention state is ~1 MB ($d \times d$ per head, $16 \times (128^2) \times 4$ bytes for FP32). This 2000× reduction in per-request state memory directly translates to higher maximum batch sizes or the ability to serve the model on hardware that cannot fit the softmax KV cache at all. The practical scenario is a developer running a coding assistant locally on a laptop GPU with 8 GB VRAM: a standard 8B model might exhaust memory with a 32K-token codebase in context, while Ring-mini-linear-2.0 can handle 128K context comfortably.

When to Prefer This Method

The paper articulates a clear tradeoff between the hybrid linear architecture and pure softmax attention, grounded in empirical results rather than theoretical claims. The decision rule is:

Prefer the Ring-linear hybrid architecture when:

• The dominant deployment cost is long-context inference (generation lengths >8K, context lengths >8K), where the throughput advantage of linear attention's constant-state memory becomes substantial (>2× at 32K, >10× at 64K; Figures 7–8). This includes reasoning models that generate long chain-of-thought, code agents that process large codebases, and document analysis applications.
• You have the engineering capacity to implement or adopt fused linear attention kernels (the paper shows that naive implementations through existing libraries like flash-linear-attention use 2–4 fragmented kernels that leave substantial efficiency on the table; Section 3.4).
• You are willing to invest in continued pre-training to convert an existing dense model (600B–1T tokens, Section 4) and accept a small capability regression in specialized reasoning and professional knowledge that can be recovered through post-training (Figure 9).
• For RL training of long-output models, you can invest the engineering effort to systematically align training and inference engines (Section 5.2.1) rather than relying on algorithmic workarounds for training-inference disparity.

Prefer a pure softmax attention architecture when:

• The dominant deployment length is below 8K tokens, where the throughput advantage of linear attention is minimal or negative (Figures 7a, 8a: prefill throughput at 4K is slightly below the softmax baseline, and the paper notes that linear attention advantages "only become pronounced at sequence lengths beyond 8K").
• You lack the infrastructure to maintain custom fused kernels across framework updates — the paper's efficiency gains depend on the linghe kernel library and Flood inference framework, which require ongoing maintenance as SGLang and vLLM evolve.
• The application requires high-fidelity retrieval of specific tokens from long contexts (e.g., exact string matching, needle-in-a-haystack retrieval), where softmax attention's exact pairwise similarity provides stronger retrieval guarantees than linear attention's compressed $d \times d$ state representation.

These conditions are directly extractable from the paper's measurements: the crossover point at ~8K sequence length (Figures 7–8), the continued pre-training cost of 600B–1T tokens (Section 4), the capability recovery of >98% in most categories but ~96% in reasoning and professional knowledge (Figure 9), and the kernel engineering requirement (Sections 3.1–3.2). The decision is quantitative and scenario-specific, not a universal recommendation.