Kimi Linear: An Expressive, Efficient Attention Architecture

ArXiv: 2510.26692

🎯 Pitch

Linear attention models finally beat standard Transformers at their own game—even on short sequences—by giving each feature dimension its own independent forgetting rate. A new chunkwise algorithm makes this fine-grained gating efficient enough to unlock up to 6× faster decoding for million-token contexts, all while matching or exceeding full attention quality as a drop-in replacement.

---

1. Executive Summary

This paper introduces Kimi Linear, a hybrid linear attention architecture that—for the first time under fair comparisons—outperforms full attention across short-context, long-context, and reinforcement learning scaling regimes. The core innovation is Kimi Delta Attention (KDA) (channel-wise gating that gives each feature dimension an independent forgetting rate, versus Gated DeltaNet's coarser head-wise scalar gate), combined with a bespoke chunkwise algorithm that constrains the Diagonal-Plus-Low-Rank transition matrix to eliminate redundant matrix multiplications while preserving consistency with the delta rule. Kimi Linear interleaves KDA with full attention layers in a uniform 3:1 ratio, achieving up to 6× decoding throughput at 1M context length and reducing KV cache usage by up to 75% while matching or exceeding full attention baselines across all evaluated tasks, establishing that linear attention can serve as a drop-in replacement for full attention only when paired with fine-grained gating and periodic global attention layers.

2. Context and Motivation

The Core Problem: Quadratic Attention Costs Are Crippling Long-Context and Agentic LLMs

The fundamental problem this paper addresses is the inference-time bottleneck of standard softmax attention in large language models. The attention mechanism introduced by Vaswani et al. (2017) computes a pairwise similarity matrix between every token in a sequence, producing computational costs that scale as $O(T^2)$ in time and $O(T)$ in memory for the key-value (KV) cache, where $T$ is the sequence length. This means that as models are asked to process longer contexts—which is increasingly the norm for modern agentic, tool-using, and reasoning-heavy applications—the cost of attention grows quadratically, not linearly.

The paper frames this as an urgent practical bottleneck driven by a specific industry trend: reinforcement learning (RL) test-time scaling and agentic intelligence. When LLMs operate as agents (e.g., DeepSeek-V3, Kimi K2), they must process extended trajectories that include tool-use interactions, multi-turn dialogues, code repository exploration, and complex decision spaces—all at inference time. In these scenarios, the model generates thousands or millions of output tokens, and the KV cache for all previous tokens must be retained and accessed at every subsequent generation step. As the authors state in Section 1:

"This shift toward RL test-time scaling, where models must process extended trajectories, tool-use interactions, and complex decision spaces at inference time, exposes fundamental inefficiencies in standard attention mechanisms."

The consequences are threefold: (1) throughput degradation—each new token takes longer to generate because the attention computation grows with sequence length; (2) memory pressure—the linearly growing KV cache consumes GPU memory that could otherwise support larger batch sizes; and (3) context-length ceilings—the quadratic cost effectively imposes practical limits on how many tokens a model can process, restricting applications that require million-token contexts.

Figure 1b of the paper quantifies this concretely: at 1M tokens, the full MLA baseline has a time per output token (TPOT) of 11.48ms, while Kimi Linear achieves 1.84ms—a 6.3× speedup. This is not merely a nice-to-have; for production systems serving millions of users, this directly translates to cost, latency, and feasibility.

The Central Tension: Linear Attention Is Efficient but Historically Underperforms

The natural solution to $O(T^2)$ attention is linear attention, first proposed by Katharopoulos et al. (2020). Linear attention reformulates the softmax kernel as a feature map, exploiting the associativity of matrix multiplication to compute attention in $O(T)$ time with a fixed-size state (no growing KV cache). This is conceptually elegant: rather than storing every past key and value, the model maintains a compressed matrix-valued recurrent state $S_t \in \mathbb{R}^{d_k \times d_v}$ that accumulates key-value associations:

$S_t = S_{t-1} + k_t v_t^\top$ $o_t = S_t^\top q_t$

The problem, and the reason this paper exists, is that linear attention has consistently underperformed softmax attention in language modeling quality—even on short sequences where the efficiency advantage is minimal. The paper acknowledges this head-on in Section 1: "Linear attention offers a principled approach to reducing computational complexity but has historically underperformed softmax attention in language modeling—even for short sequences—due to limited expressivity."

This quality gap stems from several inherent limitations that the paper's background sections elaborate:

Limited memory control. The vanilla linear attention update $S_t = S_{t-1} + k_t v_t^\top$ has no mechanism for forgetting. Keys and values accumulate indefinitely, causing interference between old and new associations. There is no "criterion for which memories to erase" (Section 2.2), and the state grows unbounded. This makes it difficult for the model to selectively attend to recent information or overwrite outdated facts.

Weak retrieval and copying. Linear attention compresses the entire context into a fixed-size matrix state, which fundamentally limits its ability to perform precise retrieval—the model must reconstruct specific past tokens from a lossy compressed representation. As the paper notes in Section 7.2, "pure linear attention still struggle with precise memory retrieval and exact copying." This is not a minor edge case; it affects core capabilities like in-context learning, factual recall from long documents, and code generation that requires referencing distant variable definitions. Theoretical work by Jelassi et al. (2024) and Wen et al. (2024) has formalized these limitations, showing that RNN-style models face fundamental bottlenecks on copying and retrieval tasks that transformers handle with ease.

Finite-state capacity. Linear attention models are effectively recurrent neural networks with a bounded state size. This means they have a hard ceiling on how much information they can retain—once the state is "full," new information necessarily overwrites old information. Transformers with full attention, by contrast, have theoretically unbounded memory because they can attend directly to any token in the sequence.

How Prior Work Attempted to Close the Gap

The paper identifies two major lines of prior work that have significantly narrowed—but not closed—the gap between linear and softmax attention:

1. Gating and Decay Mechanisms

Starting with RetNet (Sun et al., 2023) and continuing through Mamba2 (Dao and Gu, 2024) and Gated Linear Attention (GLA; Yang et al., 2024), researchers introduced forgetting mechanisms that give linear attention models the ability to selectively discard information. The general idea is to modify the recurrence to include a decay term $A_t$:

$S_t = A_t S_{t-1} + k_t v_t^\top$

The key design choice is how $A_t$ is parameterized:

• RetNet uses a data-independent scalar decay $\alpha$—same for all tokens, all dimensions.
• Mamba2 uses a data-dependent scalar $\alpha_t$—each head learns a single forget value per timestep.
• GLA uses a fine-grained diagonal matrix $\text{Diag}(\alpha_t) \in \mathbb{R}^{d_k \times d_k}$—each dimension within each head gets its own forgetting rate.

The progression from scalar to channel-wise gating represents increasing expressiveness: fine-grained gates let different feature dimensions specialize in different temporal scales, with some dimensions retaining information over very long horizons while others rapidly adapt to new context.

2. The Delta Rule and Fast-Weight Memory

A parallel line of work reinterpreted linear attention through the lens of online learning. Schlag et al. (2021) proposed DeltaNet, which recasts the state update as gradient descent on a reconstruction objective:

$L_t(S) = \frac{1}{2} \| S^\top k_t - v_t \|^2$

Taking a gradient step with learning rate $\beta_t$ yields the delta rule:

$S_t = S_{t-1} - \beta_t \nabla_{S} L_t(S_{t-1}) = (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top$

This is a crucial conceptual shift. Unlike vanilla linear attention which blindly accumulates key-value pairs, DeltaNet treats the state $S$ as an associative memory that continuously corrects itself. The term $(I - \beta_t k_t k_t^\top)$ is a rank-1 "Householder transformation" that subtracts the model's current prediction error for key $k_t$ from the state. If the state already maps $k_t$ to $v_t$ correctly, the update is minimal; if it's wrong, the state is corrected.

Gated DeltaNet (GDN; Yang et al., 2025) combined these two innovations by adding a scalar forget gate $\alpha_t$:

$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top$

This can be interpreted as performing gradient descent on a decayed version of the previous state, introducing "weight decay on the fast weights" (Section 2.2) that prevents the association memory from accumulating interference from outdated key-value pairs. GDN showed strong empirical results and served as the direct predecessor to the paper's KDA module.

Where These Approaches Still Fall Short

Despite these advances, the paper identifies several remaining gaps that directly motivate Kimi Linear:

Coarse gating limits memory precision. GDN, like Mamba2, uses a single scalar forget gate per head. This means all $d_k$ dimensions within a head share the same forgetting rate. But real sequences contain information at multiple temporal scales—recent tokens that should persist, irrelevant details to discard, and long-range dependencies to retain—all within the same head. A scalar gate forces the model to compromise, either retaining too much noise or forgetting too much signal. GLA's channel-wise gating addresses this but lacks the delta rule's corrective capacity; the two innovations had never been combined.

Retrieval remains the Achilles' heel. Even with gating and delta updates, purely linear models still struggle on tasks requiring precise token-level recall from long contexts. The paper's synthetic experiments (Section 5.1, Figure 4) demonstrate this explicitly: on the Palindrome task (reversing a sequence of random tokens) and Multi-Query Associative Recall (MQAR), even GDN struggles as sequence length grows. Mamba2—which uses only multiplicative decay without a delta rule—fails completely. This is because linear attention fundamentally operates on compressed representations, not direct token access.

No unified architecture that actually beats full attention. The paper's key claim—stated explicitly in the abstract—is that no prior linear or hybrid architecture has demonstrated superior performance to full attention under fair, matched-scale comparisons. Prior hybrid models (interleaving linear and full attention layers) existed, such as Jamba (Lieber et al., 2024), MiniMax-01 (2025), and various Mamba-Transformer hybrids. But as the paper notes in Section 7.2, these "often operated at limited scale or lacked comprehensive evaluation across diverse benchmarks." They demonstrated that linear attention could be comparable or not too much worse, but not that it could surpass full attention across the board. This is the bar that Kimi Linear explicitly sets out to clear.

The Overlooked Dimension: Positional Encoding in Linear Attention

A subtle but important thread running through the paper's motivation is the role of positional information. Standard softmax attention is position-agnostic by design and requires explicit positional encodings—RoPE (Su et al., 2024) being the de facto standard. RoPE works by applying rotation matrices to queries and keys, encoding relative position through trigonometric identities.

The paper makes a novel observation in Section 6.1: the gated delta rule can be interpreted as a learned multiplicative positional encoding:

$o_t = \sum_{i=1}^t \left( q_t^\top \left( \prod_{j=i+1}^t A_j (I - \beta_j k_j k_j^\top) \right) k_i \right) v_i$

The transition matrix $A_j (I - \beta_j k_j k_j^\top)$ serves the same structural role as RoPE's rotation matrices—it modulates the query-key interaction based on relative position. But crucially, this modulation is data-dependent and learnable, not a fixed trigonometric function. This means KDA can, in principle, learn more flexible positional representations than RoPE, adapting its "receptive field" based on content rather than absolute distance.

This insight directly motivates two design choices: (1) channel-wise gating—RoPE achieves its effectiveness partly through different rotation frequencies per dimension pair, creating a multi-scale positional representation. KDA's channel-wise decay $\text{Diag}(\alpha_t)$ provides the same multi-scale capability, with each dimension independently controlling its temporal range. GDN's scalar gate loses this diversity. (2) NoPE for full attention layers—since KDA handles all positional encoding, the interleaved MLA layers can use no positional encoding at all. This simplifies long-context extension (no need to retune RoPE base frequencies) and enables conversion to pure Multi-Query Attention at inference time.

Concrete Consequences of the Gap

The paper grounds its motivation in specific deployment scenarios where the attention bottleneck is most acute:

1. Agentic LLMs processing tool-use trajectories. When an LLM calls APIs, reads documentation, and plans multi-step actions, its context grows rapidly. Systems like Kimi K2 (the team's prior work) need to handle million-token contexts with low latency. Full attention's KV cache for 1M tokens is enormous; linear attention's fixed state size makes such scenarios feasible.

2. RL post-training with long outputs. When training reasoning models through RL (e.g., DeepSeek-R1, Kimi K1.5), the model generates long chain-of-thought traces. Training and inference throughput during RL is bottlenecked by attention costs on these long sequences. A faster attention mechanism directly accelerates the RL training loop.

3. Edge and on-device deployment. Smaller models with efficient attention can be deployed on consumer hardware where memory is constrained. The paper's 75% KV cache reduction means a 3B-parameter hybrid model can fit in memory where a 3B-parameter full-attention model cannot during long-sequence inference.

4. Batch throughput in serving systems. Because linear attention uses constant memory per sequence, more sequences can be batched together during inference, increasing overall system throughput. Figure 1b explicitly notes that the reduced memory footprint "enables larger batches."

Positioning Relative to Sparse Attention

The paper helpfully distinguishes linear attention from another major approach to efficient attention: sparse attention (Section 7.1). Sparse attention methods like NSA (Yuan et al., 2025), MoBA (Lu et al., 2025), and DeepSeek Sparse Attention select a subset of tokens for each query to attend to, approximating the full attention matrix while reducing computation. The paper acknowledges that sparse attention "tends to retrieve fine-grained historical information more effectively" than linear attention.

However, it identifies two key disadvantages: (1) the KV cache must still be stored in its entirety, since any token might be selected for attention—this means memory grows linearly with sequence length, unlike linear attention's constant state; (2) the theoretical expressive upper bound remains that of full attention, whereas linear attention with the delta rule can achieve "theoretically stronger expressive capacity" through its fast-weight memory mechanism.

The paper positions Kimi Linear not as competing with sparse attention, but as a complementary approach that could be combined in future work.

Summary: What Kimi Linear Sets Out to Solve

The paper's motivation can be distilled into a clear thesis statement:

Linear attention can match or surpass full attention—but only if three conditions are met: (1) sufficiently fine-grained gating to control memory at the per-channel level, (2) the delta rule's corrective updates to enable precise associative recall, and (3) periodic full-attention layers to handle the retrieval tasks that compressed representations fundamentally struggle with.

Prior work achieved at most one or two of these conditions. GDN has the delta rule but coarse gating. GLA has fine-grained gating but lacks the delta rule. Various hybrids exist but use weaker linear components (e.g., Mamba2 without delta updates) or were evaluated at insufficient scale. Kimi Linear's claimed contribution is the integrated architecture that satisfies all three conditions and demonstrates, through 1.4T-token matched-pretraining comparisons, that it genuinely outperforms full attention—not just in efficiency, but in raw quality metrics across short-context, long-context, and RL-trained settings.

3. Technical Approach

3.1 Reader Orientation

The paper designs and trains a hybrid language model architecture called Kimi Linear that replaces most of the expensive full-attention layers with a new, more expressive linear attention module called Kimi Delta Attention (KDA), interleaving them with occasional full-attention layers. The problem it solves is the quadratic computational cost of standard softmax attention at long sequence lengths—a problem that has historically forced a tradeoff where efficient linear attention models underperformed in quality. The shape of the solution is a layered hybrid where every three KDA layers are followed by one full Multi-Head Latent Attention (MLA) layer, with KDA itself combining two previously-separate innovations (fine-grained channel-wise forgetting and the delta rule's corrective memory updates) into a single integrated recurrence whose chunkwise parallelization is specially optimized to avoid the numerical and computational overhead of prior formulations.

3.2 Big-Picture Architecture (Diagram in Words)

The Kimi Linear model has five major components, stacked in a repeating pattern:

1. Kimi Delta Attention (KDA) layers — the workhorse linear attention module. Each layer takes a token sequence as input, projects it to queries, keys, values, per-channel decay rates, and a learning rate, then updates a fixed-size matrix-valued recurrent state using a combination of channel-wise multiplicative decay and a rank-1 delta rule correction. The output is produced by reading from this state.

2. Multi-Head Latent Attention (MLA) layers — standard full-attention layers placed every fourth layer. These use no positional encoding (NoPE) because KDA handles all positional information. During inference, NoPE enables conversion to pure Multi-Query Attention for efficiency.

3. Short convolution preprocessing — before the attention computation, the query, key, and value projections each pass through a depthwise convolution (kernel size 4) followed by a Swish activation, capturing local token dependencies.

4. Mixture-of-Experts (MoE) feedforward layers — after each attention layer (KDA or MLA), tokens are routed through 8 selected experts from a pool of 256, plus one shared expert. The first layer is dense (no MoE) for training stability.

5. Output gating — before the final output projection, a data-dependent Sigmoid gate modulates the attention output, improving performance and mitigating attention sink phenomena.

Information flows as follows: token embeddings enter the first dense layer → pass through the MoE feedforward → enter a KDA layer (token mixing) with short convolution preprocessing, channel-wise gating, and delta rule state update → pass through MoE → repeat for two more KDA layers → pass through one MLA layer with NoPE full attention → pass through MoE → repeat the 3:1 KDA-to-MLA pattern for all remaining layers → final output projection.

3.3 Roadmap for the Deep Dive

• First, the KDA recurrence and its components — the core state update equation, its interpretation as online gradient descent with weight decay, and the meaning of each term (decay gate, learning rate, delta correction). This establishes what KDA computes.

• Second, the neural parameterization — how the abstract variables (keys, queries, values, decay rates, learning rates) are computed from the input token representations. This covers short convolutions, L2 normalization, the low-rank decay projection, and the output gate.

• Third, the chunkwise parallelization algorithm — the mathematical derivation that converts the sequential recurrence into a form suitable for parallel GPU execution. This is where the paper's efficiency gains over general DPLR formulations originate.

• Fourth, the DPLR connection and optimization — why KDA is a constrained special case of Diagonal-Plus-Low-Rank, what computations it eliminates, and the resulting ~2× kernel speedup.

• Fifth, the hybrid architecture — the 3:1 layer interleaving, the NoPE design choice for MLA layers, and the rationale for layerwise rather than headwise hybridization.

• Sixth, the FLOPs accounting and inference strategy — the computational complexity analysis that quantifies KDA's advantage, and the dual prefill/decoding strategy (chunk kernel for prefill, recurrent kernel for generation).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a model architecture paper whose core idea is that linear attention can match or exceed full attention if it combines fine-grained channel-wise forgetting with delta-rule corrective updates, and that the resulting module can replace 75% of attention layers in a hybrid architecture to deliver substantial efficiency gains without quality degradation.

---

The KDA State Update Equation: What Gets Computed

The heart of Kimi Delta Attention is a recurrence that maintains a matrix-valued memory state $S_t \in \mathbb{R}^{d_k \times d_v}$ and updates it at each timestep $t$. The full equation is:

$S_t = \left( I - \beta_t k_t k_t^\top \right) \text{Diag}(\alpha_t) S_{t-1} + \beta_t k_t v_t^\top$

where $S_{t-1} \in \mathbb{R}^{d_k \times d_v}$ is the memory state from the previous timestep, $k_t \in \mathbb{R}^{d_k}$ is the key vector (what to look up in memory), $v_t \in \mathbb{R}^{d_v}$ is the value vector (what to store), $\text{Diag}(\alpha_t) \in \mathbb{R}^{d_k \times d_k}$ is a diagonal matrix of per-channel decay rates $\alpha_{t,1}, \ldots, \alpha_{t,d_k} \in [0, 1]$ (how much each dimension of the state is retained), $\beta_t \in [0, 1]$ is a scalar learning rate (the step size of the memory correction), and $I \in \mathbb{R}^{d_k \times d_k}$ is the identity matrix.

Operational interpretation: The state update proceeds in three conceptual steps applied in sequence. First, the previous state $S_{t-1}$ is multiplied elementwise by the channel-wise decay $\text{Diag}(\alpha_t)$ — this is selective forgetting, where each of the $d_k$ memory dimensions decays at its own rate independently, so some dimensions may preserve long-range information while others rapidly overwrite old content. Second, the Householder-like term $(I - \beta_t k_t k_t^\top)$ is applied to the decayed state — this subtracts the model's current prediction error for the new key $k_t$, correcting the memory so that if you looked up $k_t$ from this corrected state, you would get closer to $v_t$. The product $\text{Diag}(\alpha_t) S_{t-1}$ is the "decayed previous state," and multiplying by $(I - \beta_t k_t k_t^\top)$ performs one step of gradient descent on the reconstruction loss $\frac{1}{2}\|S^\top k_t - v_t\|^2$ with learning rate $\beta_t$. Third, the new key-value association $\beta_t k_t v_t^\top$ is added — this is the standard Hebbian-style associative memory update that directly stores the mapping $k_t \mapsto v_t$ in the state.

The output at timestep $t$ is simply reading from the state using the query:

$o_t = S_t^\top q_t \in \mathbb{R}^{d_v}$

where $q_t \in \mathbb{R}^{d_k}$ is the query vector. This computes the value that the current memory state associates with query $q_t$ — if $S_t$ has learned that keys similar to $q_t$ map to particular values, that information is retrieved.

Why this form: The key design innovation is combining two mechanisms that prior work kept separate. GDN (Yang et al., 2025) applies the delta rule with a scalar forget gate: $S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top$. Because $\alpha_t$ is a scalar, the entire $d_k$-dimensional state decays at the same rate, through one head. GLA (Yang et al., 2024) uses fine-grained diagonal decay $S_t = \text{Diag}(\alpha_t) S_{t-1} + k_t v_t^\top$, but without the delta rule's corrective rank-1 update. KDA combines both by placing $\text{Diag}(\alpha_t)$ inside the delta update, giving each channel its own forgetting rate while the Householder correction still operates on the decayed state. The paper frames this as performing gradient descent on a decayed version of the state — you first forget (multiplicatively, per-channel), then correct (additively, per-key). This gives the model precise control over which memory dimensions retain long-range dependencies versus rapidly adapting to new input.

---

Neural Parameterization: How Variables Are Computed from Input Tokens

The KDA recurrence is defined in terms of abstract variables $q_t, k_t, v_t, \alpha_t, \beta_t$. The paper specifies concretely how each is computed from the input representation $x_t \in \mathbb{R}^d$ for each attention head:

Query and Key: $q_t^h, k_t^h = \text{L2Norm}(\text{Swish}(\text{ShortConv}(W_{q/k}^h x_t))) \in \mathbb{R}^{d_k}$

where $W_{q/k}^h \in \mathbb{R}^{2d_k \times d}$ is a learned projection matrix that produces both $q$ and $k$ for head $h$, $\text{ShortConv}$ is a depthwise 1D convolution with a small kernel (size 4) applied along the sequence dimension, $\text{Swish}$ is the Swish/SiLU activation $x \cdot \sigma(x)$, and $\text{L2Norm}$ normalizes the resulting vector to unit length. The convolution captures local token dependencies over a 4-token window, and the L2 normalization ensures eigenvalue stability during the rank-1 Householder updates (as suggested by DeltaNet's successor work). The key and value head dimensions are fixed at $d_k = d_v = 128$ for all experiments.

Value: $v_t^h = \text{Swish}(\text{ShortConv}(W_v^h x_t)) \in \mathbb{R}^{d_v}$

This uses a similar short convolution and activation but without L2 normalization — values preserve their magnitude since they are directly stored in the state.

Channel-wise decay rates $\alpha_t$: $\alpha_t^h = f(W_\alpha^\uparrow W_\alpha^\downarrow x_t) \in [0, 1]^{d_k}$

where $W_\alpha^\downarrow \in \mathbb{R}^{r \times d}$ and $W_\alpha^\uparrow \in \mathbb{R}^{d_k \times r}$ form a low-rank projection with rank $r$ equal to the head dimension, and $f(\cdot)$ is a decay function similar to those used in GDN and Mamba2 that maps inputs to the $[0, 1]$ interval (implemented via a sigmoid or exponential transformation). The low-rank projection reduces parameter count — rather than having $d \times d_k$ parameters for the gate, it has $r \times d + d_k \times r$ — while maintaining expressiveness. The per-channel output means each of the $d_k = 128$ dimensions gets its own independent forgetting rate for each token.

Learning rate $\beta_t$: $\beta_t^h = \text{Sigmoid}(W_\beta^h x_t) \in [0, 1]$

A simple linear projection followed by sigmoid to produce a scalar per head per token. This controls the step size of the delta rule correction — a high $\beta_t$ means a large correction when the current key's stored value is inaccurate, while a low $\beta_t$ means conservative updates.

Output gate (after KDA computation): $o_t = W_o \left( \text{Sigmoid}(W_g^\uparrow W_g^\downarrow x_t) \odot \text{RMSNorm}(\text{KDA}(q_t, k_t, v_t, \alpha_t, \beta_t)) \right)$

where $W_g^\downarrow$ and $W_g^\uparrow$ form another low-rank projection producing a gating vector in $\mathbb{R}^{d_k}$ via Sigmoid, $\odot$ is elementwise multiplication, $\text{RMSNorm}$ is Root Mean Square Layer Normalization applied headwise (each head's output is independently normalized), $W_o \in \mathbb{R}^{d \times d}$ is the output projection, and $\text{KDA}(q_t, k_t, v_t, \alpha_t, \beta_t)$ denotes the full attention computation using the recurrence.

Why these parameterization choices:

• Short convolutions before Swish: The lightweight depthwise convolution with kernel size 4 captures local token dependencies (e.g., adjacent word relationships) before the linear attention processes longer-range patterns. The paper's ablation (Table 1) shows removing these convolutions degrades validation PPL from 5.65 to 5.70. This aligns with findings from Mamba2 and other state-space models where small convolutions consistently improve quality despite their minimal parameter cost.

• L2Norm on queries and keys: The Householder update $(I - \beta_t k_t k_t^\top)$ involves the outer product $k_t k_t^\top$. If $k_t$ has unbounded norm, this can cause eigenvalue instability — the rank-1 correction could overshoot and destabilize the state. L2 normalization constrains $\|k_t\|_2 = 1$, making the correction well-behaved. The paper notes this follows the practice from the parallel DeltaNet work (Yang and Wang, 2024).

• Sigmoid output gate, not Swish: The ablation in Section 5.2 and Table 1 compares Sigmoid against Swish and no gating. Sigmoid gating achieves the best perplexity. The paper notes this is "consistent with Qiu et al. (2025), who also conclude that Sigmoid gating offers superior performance." The Swish gate, used by the original GDN paper, performs "substantially worse." The output gate also helps mitigate the "attention sink" phenomenon where early tokens absorb disproportionate attention weight — the data-dependent gating can suppress such spurious correlations.

• Low-rank gating projections: Both the decay gate $\alpha_t$ and the output gate use low-rank factorizations. The paper states this is "to ensure a fair parameter comparison" with baselines, since full-rank gates would add parameters not present in comparison models. The ablation (Table 1) confirms performance remains competitive with this parameterization.

---

Online Learning Interpretation: Why the Delta Rule

The paper frames KDA (and its GDN predecessor) through a learning-theoretic lens in Table 7. The state $S_t$ can be understood as the parameters of an online linear model that tries to predict $v_t$ from $k_t$ via $S_t^\top k_t$. The recurrence implements one step of stochastic gradient descent on the loss:

$L_t(S) = \frac{\beta_t}{2} \left\| \tilde{S}_{t-1}^\top k_t - v_t \right\|^2$

where $\tilde{S}_{t-1} = \text{Diag}(\alpha_t) S_{t-1}$ is the decayed previous state. The gradient with respect to this decayed state is:

$\nabla_{\tilde{S}_{t-1}} L_t = \beta_t(\tilde{S}_{t-1}^\top k_t - v_t) k_t^\top$

and the update yields $S_t = \tilde{S}_{t-1} - \beta_t(\tilde{S}_{t-1}^\top k_t - v_t) k_t^\top = (I - \beta_t k_t k_t^\top) \text{Diag}(\alpha_t) S_{t-1} + \beta_t k_t v_t^\top$, which is exactly the KDA recurrence.

What this means operationally: If the current key $k_t$ were looked up in the decayed state, the model would predict $\tilde{S}_{t-1}^\top k_t$. The error between this prediction and the actual value $v_t$ is $e_t = \tilde{S}_{t-1}^\top k_t - v_t$. The delta rule subtracts $\beta_t e_t k_t^\top$ from the state — this is a rank-1 correction that says "for keys similar to $k_t$, adjust the stored value by $-\beta_t e_t$." If the memory already correctly maps $k_t$ to $v_t$, then $e_t \approx 0$ and the correction is negligible; if it's wrong, the state is updated proportionally to the error.

Why this is better than plain linear attention: Vanilla linear attention $S_t = S_{t-1} + k_t v_t^\top$ has no error correction — it blindly adds the new key-value pair regardless of whether it contradicts or overwrites existing associations. This causes interference, especially when the same key dimension is reused for multiple values. The delta rule's subtraction of $k_t k_t^\top S_{t-1}$ explicitly removes the old prediction before adding the new value, preventing accumulation of contradictory mappings. Similarly, plain decay models $S_t = \alpha_t S_{t-1} + k_t v_t^\top$ can forget old information but cannot selectively correct specific associations — they must either retain everything (possibly with interference) or decay everything uniformly. The delta rule provides targeted correction per key.

---

Chunkwise Parallelization: Converting the Recurrence to Matrix Operations

The KDA recurrence as written is sequential — to compute $S_t$, you need $S_{t-1}$, which needs $S_{t-2}$, and so on. This is inefficient on GPUs, which thrive on parallel matrix multiplications. The paper develops a chunkwise algorithm that splits a sequence of length $L$ into $L/C$ chunks of size $C = 64$ tokens each, computes within each chunk in parallel using matrix operations, and passes state between chunks recurrently.

Chunk notation. Let $t \in [0, L/C)$ index the chunks, and let $r \in [1, C]$ index positions within chunk $t$. The notation $\square_r^{[t]}$ means the $r$-th token in the $t$-th chunk. The state $S_{[t]} = S_C^{[t-1]}$ is the state at the end of the previous chunk, which becomes the initial state for the current chunk.

The core idea. Within a chunk, the recurrence can be partially unrolled:

$S_r^{[t]} = \underbrace{\left( \prod_{i=1}^r \left( I - \beta_i^{[t]} k_i^{[t]} k_i^{[t]\top} \right) \text{Diag}(\alpha_i^{[t]}) \right)}_{:= P_r^{[t]}} \cdot S_{[t]} + \underbrace{\sum_{i=1}^r \left( \prod_{j=i+1}^r \left( I - \beta_j^{[t]} k_j^{[t]} k_j^{[t]\top} \right) \text{Diag}(\alpha_j^{[t]}) \right) \cdot \beta_i^{[t]} k_i^{[t]} v_i^{[t]\top}}_{:= H_r^{[t]}}$

where $P_r^{[t]}$ is the cumulative product of Householder-and-decay transforms applied to the incoming state, and $H_r^{[t]}$ is the cumulative contribution from all key-value pairs within the chunk, each propagated through subsequent decays and corrections.

The WY representation for $P$ and $H$. The challenge is that $P_r^{[t]}$ is a product of $r$ rank-1-corrected diagonal matrices — naively computing it sequentially defeats the purpose. The paper uses the WY representation (Bischof and Van Loan, 1987), a classical numerical linear algebra technique that packs a sequence of Householder transformations into a compact matrix form. The derivations (Propositions 1 and 2 in Appendix B) show that:

$P_r^{[t]} = \text{Diag}(\gamma_r^{[t]}) - \sum_{i=1}^r \text{Diag}(\gamma_{i \to r}^{[t]}) k_i^{[t]} w_i^{[t]\top}$

where $\gamma_r^{[t]} = \prod_{j=1}^r \alpha_j^{[t]}$ is the cumulative per-channel decay from position 1 to $r$ (elementwise product along the sequence), $\gamma_{i \to r}^{[t]} = \prod_{j=i}^r \alpha_j^{[t]}$ is the cumulative decay from position $i$ to $r$, and $w_i^{[t]}$ is an auxiliary vector computed recursively:

$w_r^{[t]} = \beta_r^{[t]} \left( \text{Diag}(\gamma_r^{[t]}) k_r^{[t]} - \sum_{i=1}^{r-1} w_i^{[t]} \left( k_i^{[t]\top} \text{Diag}(\gamma_{i \to r}^{[t]}) k_r^{[t]} \right) \right)$

Similarly for $H_r^{[t]}$:

$H_r^{[t]} = \sum_{i=1}^r \text{Diag}(\gamma_{i \to r}^{[t]}) k_i^{[t]} u_i^{[t]\top}$

with auxiliary vectors:

$u_r^{[t]} = \beta_r^{[t]} \left( v_r^{[t]} - \sum_{i=1}^{r-1} u_i^{[t]} \left( k_i^{[t]\top} \text{Diag}(\gamma_{i \to r}^{[t]}) k_r^{[t]} \right) \right)$

What this accomplishes: Instead of computing $P_r^{[t]}$ by sequential matrix multiplication ($O(C \cdot d_k^3)$), the WY form expresses it as a diagonal matrix minus a sum of rank-1 terms. The auxiliary vectors $w_i$ and $u_i$ depend on inner products between keys weighted by cumulative decays, which can be computed efficiently using a triangular solve (forward substitution).

The UT transform for hardware efficiency. The paper then applies the UT transform (Joffrain et al., 2006), a technique that converts the WY representation into operations that map well to GPU Tensor Cores (matrix multiplication units). The result is a compact matrix formulation:

$M_{[t]} = \left( I + \text{StrictTril} \left( \text{Diag}(\beta^{[t]}) \left( \Gamma_{1 \to C}^{[t]} \odot K_{[t]} \right) \left( \frac{K_{[t]}}{\Gamma_{1 \to C}^{[t]}} \right)^\top \right) \right)^{-1} \text{Diag}(\beta^{[t]})$

where $K_{[t]} \in \mathbb{R}^{C \times d_k}$ stacks all key vectors in the chunk, $\Gamma_{1 \to C}^{[t]} \in \mathbb{R}^{C \times d_k}$ stacks the cumulative decay vectors from position 1 to each position $C$, $\odot$ is elementwise multiplication, $\text{StrictTril}$ zeros out the diagonal and upper triangle, and the inverse of the lower triangular matrix is computed by forward substitution (row-by-row Gaussian elimination, which is $O(C^2 d_k)$ rather than $O(C^3)$ for a general inverse). Then:

$W_{[t]} = M_{[t]} \left( \Gamma_{1 \to C}^{[t]} \odot K_{[t]} \right)$ $U_{[t]} = M_{[t]} V_{[t]}$

where $W_{[t]}$ packs the auxiliary $w$ vectors and $U_{[t]}$ packs the auxiliary $u$ vectors for the entire chunk.

Chunk-level state update. With $W_{[t]}$ and $U_{[t]}$ computed, the transition from chunk $t$ to chunk $t+1$ becomes:

$S_{[t+1]} = \text{Diag}(\gamma_C^{[t]}) S_{[t]} + \left( \Gamma_{i \to C}^{[t]} \odot K_{[t]} \right)^\top \left( U_{[t]} - W_{[t]} S_{[t]} \right) \in \mathbb{R}^{d_k \times d_v}$

What this computes: The first term $\text{Diag}(\gamma_C^{[t]}) S_{[t]}$ applies the full-chunk cumulative decay to the incoming state — this is the "forgetting" that happens over the entire chunk. The second term adds the contribution from all key-value pairs within the chunk, adjusted by a "pseudo-value" $(U_{[t]} - W_{[t]} S_{[t]})$ that accounts for both the direct value association and the correction to the existing state.

Output computation. The output for the chunk uses an inter-block recurrent / intra-block parallel strategy:

$O_{[t]} = \underbrace{\left( \Gamma_{1 \to C}^{[t]} \odot Q_{[t]} \right) S_{[t]}}_{\text{inter-chunk}} + \underbrace{\text{Tril} \left( \left( \Gamma_{1 \to C}^{[t]} \odot Q_{[t]} \right) \left( \frac{K_{[t]}}{\Gamma_{1 \to C}^{[t]}} \right)^\top \right)}_{\text{intra-chunk}} \underbrace{\left( U_{[t]} - W_{[t]} S_{[t]} \right)}_{\text{pseudo-value}} \in \mathbb{R}^{C \times d_v}$

What this computes: The first term reads from the initial state $S_{[t]}$ using the queries, weighted by cumulative decays — this retrieves information stored before this chunk. The second term computes intra-chunk attention: a lower-triangular matrix multiplication between queries and keys (weighted by relative decays) times the pseudo-values, capturing interactions between tokens within the same chunk. The division by $\Gamma_{1 \to C}^{[t]}$ in the key term is a key detail — it implements relative decay between positions, and the lower triangular mask ensures causality (token $r$ can only attend to tokens $i \leq r$).

Why this algorithm matters for hardware: By expressing the chunkwise computation as a series of matrix multiplications, triangular solves, and elementwise operations, the algorithm maps well to GPU Tensor Cores. The intra-chunk attention is a batched matrix multiplication rather than a sequential loop. The inter-chunk state transition is a constant-size matrix operation regardless of sequence length. The paper notes that this "fully utilizes the computational potential of Tensor Cores."

Numerical stability via secondary chunking. The division by $\Gamma_{1 \to C}^{[t]}$ in the intra-chunk attention (Eq. 9) can cause numerical issues when cumulative decays are very small (underflow in half-precision). The paper's pseudo-code (Listing 1 in Appendix C) handles this through secondary chunking: within each chunk, the intra-chunk attention matrix is computed in a loop over positions, recomputing relative decays from scratch for each query position to avoid explicit division. This is visible in lines 47-52 of the pseudo-code, where for each position $j$ within the chunk, the query $q_j$ is multiplied by keys weighted by $(g_i - g_j).\exp()$ (the exponential of the difference in cumulative log-decays), avoiding the problematic reciprocal.

---

The DPLR Connection and Why KDA Is More Efficient

KDA can be viewed as a constrained special case of the Diagonal-Plus-Low-Rank (DPLR) state transition structure, which takes the general form:

$S_t = (D - a_t b_t^\top) S_{t-1} + k_t v_t^\top$

where $D \in \mathbb{R}^{d_k \times d_k}$ is a diagonal matrix, and $a_t, b_t \in \mathbb{R}^{d_k}$ are vectors. This structure appears in state-space models like S4 (Gu et al., 2022) and linear attention variants like RWKV7 (Peng et al., 2025). The key connection is that KDA sets:

$D = \text{Diag}(\alpha_t), \quad a_t = \beta_t k_t, \quad b_t = k_t \odot \alpha_t$

Since $a_t$ and $b_t$ share the same underlying vector $k_t$ (one is scaled by $\beta_t$, the other elementwise-multiplied by $\alpha_t$), the general DPLR has redundant degrees of freedom that KDA eliminates.

The computational saving. The paper provides a side-by-side comparison of pseudo-code for chunkwise DPLR and KDA implementations (Listings 8a and 8b in Section 6.2). The DPLR version requires four intra-chunk attention matrices (called $A_{ab}, A_{ak}, A_{qb}, A_{qk}$) corresponding to all pairwise interactions between queries, keys, $a$-vectors, and $b$-vectors. The KDA version requires only two ($A_{qk}, A_{kk}$) because $a$ and $b$ are both derived from $k$.

Concretely, in the DPLR pseudo-code (Listing 8a), lines 13–16 compute four separate attention-like matrices involving $a$, $b$, $q$, and $k$. In the KDA version (Listing 8b), lines 14–15 compute only two matrices. Furthermore, DPLR lines 25–27 and 31–32 perform additional matrix multiplications during inter-chunk computation that KDA lines 26 and 29 eliminate.

The numerical result (Figure 2) shows that KDA achieves nearly 2× the speed of DPLR for sequence lengths up to 64K. The paper states that the operator efficiency improves by "roughly 100% compared to the DPLR formulation."

Why the constraint makes sense: The DPLR structure allows $a_t$ and $b_t$ to be completely independent vectors, but the KDA parameterization ties them both to $k_t$. Since the purpose of the low-rank correction is to update the state based on the current key, tying the correction directions to the key itself is a natural inductive bias — the correction should modify the state in directions aligned with the input that caused the correction. The paper observes that this constraint "remains more consistent with the classical delta rule" (abstract), referring to the original Widrow-Hoff delta rule where the weight update is proportional to the input vector.

Numerical stability advantage. The DPLR formulation requires computing the reciprocal of cumulative decay $1/\Gamma$ during chunkwise computation (since $a_t$ and $b_t$ may have different decay dependencies). This reciprocal can produce very large values when cumulative decay is near zero, causing numerical overflow. Prior work like GLA resolves this through secondary chunking in full precision, which "prevents full utilization of half-precision matrix multiplications and significantly reduces operator speed" (Section 3.2). By binding $a$ and $b$ to $k$, KDA avoids the need for two of the four secondary chunking steps, reducing both computation and I/O overhead while maintaining numerical stability.

---

The Hybrid Architecture: Interleaving KDA with Full Attention

Kimi Linear adopts an inter-layer hybrid design: entire layers are either KDA or full MLA, stacked in a repeating pattern. This contrasts with intra-layer hybrids (like Hymba or Jamba) that mix heterogeneous attention heads within a single layer.

The 3:1 ratio. Each block consists of three KDA layers followed by one MLA layer. The ablation in Table 1 validates this choice against ratios of 0:1 (pure MLA), 1:1, 7:1, and 15:1:

• 0:1 (pure MLA): Training PPL 9.45, validation PPL 5.77 — worst performance, showing that pure full attention at this scale actually underperforms the hybrid.
• 1:1: Training 9.29, validation 5.66 — similar validation to 3:1 but with more inference overhead (50% full attention layers instead of 25%).
• 3:1: Training 9.23, validation 5.65 — best on both metrics.
• 7:1: Training 9.23 (same as 3:1), validation 5.70 — worse validation, suggesting that insufficient global attention hurts generalization.
• 15:1: Training 9.34, validation 5.82 — substantially worse on both, indicating too few full attention layers degrade quality significantly.

The paper interprets this as an optimal quality–throughput trade-off at 3:1: enough full-attention layers to handle the retrieval tasks that linear attention struggles with, but few enough to realize the efficiency benefits (75% KV cache reduction).

Why layerwise, not headwise. The paper argues that layerwise hybridization provides "superior infrastructure simplicity and training stability" (Section 4). In a headwise hybrid, each layer must implement two different attention mechanisms (e.g., some heads do linear attention, some do full attention), requiring separate computational paths and complicating optimizations like tensor parallelism and flash attention integration. In a layerwise hybrid, each layer is homogeneous — it's either entirely KDA or entirely MLA — so standard infrastructure (e.g., vLLM, flash attention kernels) can be used without modification within each layer. The KV cache management is also simpler: MLA layers store a standard per-token KV cache; KDA layers store only a small fixed-size matrix state.

NoPE for MLA layers. Every full-attention layer in Kimi Linear uses No Position Encoding (NoPE). In standard transformers, RoPE applies rotation matrices to queries and keys based on their absolute positions, encoding relative distance. In Kimi Linear, this responsibility is entirely delegated to the KDA layers.

The paper's insight (Section 6.1) is that the KDA recurrence can be expressed as:

$o_t = \sum_{i=1}^t \left( q_t^\top \left( \prod_{j=i+1}^t A_j (I - \beta_j k_j k_j^\top) \right) k_i \right) v_i$

The cumulative product term $\prod_{j=i+1}^t A_j (I - \beta_j k_j k_j^\top)$ is structurally analogous to RoPE's cumulative rotation matrices $\prod_{j=i+1}^t R_j$ — it modulates the query-key interaction based on the relative positions $t - i$. But unlike RoPE's fixed trigonometric rotations, KDA's modulation is data-dependent and learnable. Each channel's decay rate $\alpha_t$ creates a different effective temporal receptive field (some channels retain information over long distances, some rapidly forget), analogous to RoPE's multiple rotation frequencies.

This design has two practical advantages. First, MLA layers can be converted to pure Multi-Query Attention (MQA) at inference time since NoPE eliminates position-dependent computation, further improving efficiency. Second, long-context training is simplified — there is no need to retune RoPE base frequencies or apply interpolation methods like YaRN when extending context length, which has been a persistent challenge for hybrid models.

Comparison to prior hybrid approaches. The paper distinguishes its approach from several predecessors. Earlier hybrids often used weaker linear components (e.g., Mamba2 without delta rule), which the synthetic experiments show fails badly on retrieval tasks. Others used RoPE in all layers, creating a tension between the implicit positional bias of linear attention and the explicit bias of RoPE — the paper's experiments with Kimi Linear (RoPE) in Table 5 show that adding RoPE to the global attention layers improves short-context performance but degrades long-context performance (RULER drops from 84.3 to 78.8), likely because the combination creates an "overemphasis on short-range order in the global layer, which benefits short contexts but makes the model less flexible when adapting mid-training to extended contexts" (Section 5.2).

---

FLOPs Accounting and Inference Strategy

Training FLOPs per attention head. For a single head with dimension $d_h = 128$ and chunk size $C = 64$, the paper derives:

$\text{FLOPs}_{\text{KDA}}(T; C, d_h) = 6T d_h^2 + 3TC d_h + TC^2$

where $T$ is the sequence length, $6T d_h^2$ comes from the linear projections and matrix multiplications in the chunkwise algorithm, $3TC d_h$ comes from the intra-chunk attention computations involving the $C \times C$ attention-like matrices, and $TC^2$ comes from the triangular solves. For comparison, full attention per head is:

$\text{FLOPs}_{\text{Attn}}(T; d_h) = 2T^2 d_h$

For $T = 4096$, $C = 64$, $d_h = 128$: KDA head FLOPs $\approx 6 \cdot 4096 \cdot 128^2 + 3 \cdot 4096 \cdot 64 \cdot 128 + 4096 \cdot 64^2 \approx 400\text{M} + 100\text{M} + 17\text{M} \approx 517\text{M}$. Full attention head FLOPs $\approx 2 \cdot 4096^2 \cdot 128 \approx 4.3\text{B}$. The KDA head uses roughly 8× fewer FLOPs at this sequence length, and the ratio improves quadratically with sequence length.

Inference strategy. During the prefill phase (processing all input tokens in parallel), the model uses the chunkwise kernel with FLOP-intensive matrix operations — this is efficient because the entire input is available at once and Tensor Cores can be saturated with batched matrix multiplications. During autoregressive decoding (generating one token at a time), the model switches to the recurrent kernel (Eq. 2), which updates the state with a single step of the recurrence — this avoids the chunkwise overhead for single-token generation and uses the constant-size state to avoid scanning the full history.

Memory and throughput. The key advantage is that KDA layers maintain a fixed-size state of dimension $d_k \times d_v = 128 \times 128 = 16,384$ elements per head, regardless of sequence length. For the 3:1 hybrid, this means 75% of layers have constant memory cost, while 25% (MLA layers) have the standard linearly-growing KV cache. As sequence length increases, the decoding time per token (TPOT) for linear layers remains constant, while full attention layers grow. The measured speedups (Figure 7b) show Kimi Linear achieving 2.3× faster TPOT than full MLA at 1M context length for batch size 1, and 6.3× faster when accounting for the ability to use larger batch sizes due to reduced memory pressure (Figure 1b).

Practical memory reallocation. Because KDA layers don't need a KV cache, the saved GPU memory can be repurposed to increase batch size during inference. At 1M tokens, the paper reports a theoretical maximum 6.3× throughput improvement over MLA when batch size is scaled to fill the available memory — this is the metric shown in Figure 1b, where 1.84ms TPOT for Kimi Linear compares to 11.48ms for MLA, both with maximally-sized batches given their respective memory footprints.

4. Key Insights and Innovations

Innovation 1: Fine-Grained Gating and the Delta Rule Are Multiplicative, Not Additive, in Their Benefit

The field's dominant approach to improving linear attention has been to separately pursue two lines of work: gating mechanisms for selective forgetting (GLA, Mamba2, RetNet) and the delta rule for corrective memory updates (DeltaNet, Gated DeltaNet). Each line showed incremental gains, and Gated DeltaNet (GDN; Yang et al., 2025) combined them by placing a scalar forget gate before the delta update. But this combination treated gating and the delta rule as independent additives — you get some benefit from each, and combining them is better than either alone.

KDA's core conceptual move is to show that these mechanisms are multiplicative in their expressiveness, not additive. The channel-wise gate Diag(α_t) and the Householder correction (I — β_t k_t k_t^⊤) interact in a way that goes beyond simply summing their individual benefits. The per-channel decay creates a multi-scale temporal representation where each dimension independently controls its memory horizon, and the delta rule then performs targeted corrections that respect this multi-scale structure. A dimension with a slow decay rate (α ≈ 1) accumulates long-range associations and gets precise error corrections applied to those associations; a dimension with fast decay (α ≈ 0) rapidly forgets and its corrections are correspondingly short-lived.

This interaction resolves a tension that was latent in GDN but never articulated: a scalar gate forces a single forgetting rate, which means the delta rule's corrections either persist too long (causing interference) or fade too quickly (losing useful associations). The channel-wise gate lets different dimensions specialize — some retain corrected associations over thousands of tokens, others rapidly discard noise. The evidence for this interaction being multiplicative rather than additive comes from the synthetic experiments (Figure 4): KDA converges substantially faster than GDN on Palindrome and MQAR tasks, and the gap widens with sequence length. If the benefits were additive, the convergence curves would show similar slopes with a constant offset; instead, KDA learns the tasks more efficiently, indicating a qualitative improvement in the model's ability to use its finite memory.

This insight is fundamental rather than incremental because it changes what future linear attention designs should optimize for. The paper implies that the design space is not "add gating" or "add delta rule" but rather "co-design the gating granularity and the correction mechanism so they jointly determine the effective memory capacity." The scaling law results (Figure 5) — where Kimi Linear achieves ~1.16× computational efficiency over MLA with compute-optimal training — suggest this co-design translates from synthetic tasks to language modeling.

Innovation 2: Linear Attention as Learned Positional Encoding Is a Productive Architectural Principle, Not Just an Observation

Several prior works have noted structural similarities between multiplicative positional encodings and gated linear recurrences. The paper's own Table 6 catalogs how various attention mechanisms can be written in a unified recurrent form with cumulative product terms. What distinguishes this paper's treatment is that it elevates this observation into an architectural design principle with concrete, testable consequences.

The principle is: if the gated delta rule serves as a learned, data-dependent positional encoding that is strictly more expressive than RoPE (since it relaxes the orthogonality constraint), then full-attention layers in a hybrid model should use no positional encoding at all — the linear layers carry all positional information. This is not an obvious design choice. The dominant practice in hybrid models has been to add RoPE everywhere (e.g., Jamba, various Mamba-Transformer hybrids) or to use RoPE in full-attention layers and rely on implicit positional biases in linear layers. The paper's alternative — NoPE for MLA, all positional encoding delegated to KDA — is a stronger claim: it asserts that the learned positional encoding in KDA is not just supplementary but sufficient as the sole positional mechanism.

The evidence for this principle comes from the comparison between Kimi Linear and Kimi Linear (RoPE) in Table 5. Adding RoPE to the MLA layers produces similar short-context scores but degrades long-context performance — RULER drops from 84.3 to 78.8, and the overall long-context average drops from 54.5 to 51.8. The paper's interpretation (Section 5.2) is that RoPE in the global layers creates a "mismatch" where the global layer overemphasizes short-range order (since RoPE's fixed frequencies are tuned to training-length patterns) while the linear layers contribute a weaker implicit positional signal. NoPE eliminates this conflict, letting KDA's learned, content-dependent positional representation be the sole positional mechanism. The model then generalizes better to extended contexts because KDA's cumulative decay products — unlike RoPE's fixed trigonometric frequencies — can adapt based on content, not just absolute distance.

This insight is significant beyond the specific architecture because it reframes the role of positional encoding in hybrid models from a per-layer concern to a system-level design choice. It suggests that future hybrid architectures should not treat positional encoding as a uniform property applied identically to all layers, but rather as a resource that can be concentrated in the most position-aware components while other components operate purely on content. The paper explicitly connects this to recent trends (Falcon-H using near-NoPE with extreme base frequencies; SwanGPT interleaving RoPE and NoPE layers) and positions KDA as a principled foundation for this approach rather than an ad-hoc engineering compromise.

Innovation 3: Verifier-Free Linear Attention Improvement at RL Scale Is an Existence Proof, Not Just a Benchmark Win

The paper's RL experiments (Figure 6) show that Kimi Linear, with all hyperparameters held identical to the MLA baseline, achieves faster and better improvement on math reasoning tasks during reinforcement learning from verifiable rewards (RLVR). This is not merely "Kimi Linear gets a higher score on AIME" — it is an empirical finding that the choice of attention architecture meaningfully affects the RL optimization dynamics in a way that compounds over training.

This matters because the RL scaling literature has largely treated the backbone architecture as fixed infrastructure — the research focus has been on reward design, KL regularization, exploration strategies, and data curricula. The implicit assumption has been that as long as the base model is competent, the RL process will improve it regardless of architectural details. The paper's results challenge this: on MATH500 and AIME 2025, Kimi Linear's RL training curves show a consistently steeper slope than MLA's, and the gap widens as training progresses. This suggests that the architecture's inductive biases — particularly KDA's learned memory management — interact favorably with the RL objective in ways that amplify over multiple training iterations.

The paper doesn't identify the causal mechanism behind this advantage, but the architectural properties suggest a plausible hypothesis: RL training on math reasoning involves generating long chain-of-thought traces where the model must maintain and update intermediate results, hypotheses, and partial derivations. KDA's channel-wise gating and delta rule corrections allow the state to function as a structured working memory that can selectively retain relevant intermediate values while discarding dead ends. Full attention can, in principle, do this through its direct token access, but it lacks the compressive inductive bias — the pressure to represent information efficiently in a fixed-size state — that linear attention inherently imposes. Under RL, where the model generates its own training data and the distribution shifts over training, this compressive bias may produce representations that generalize better to novel reasoning patterns.

The significance of this finding is that it broadens the scope of architectural innovation from pretraining quality to training dynamics. If architecture choice affects RL sample efficiency and asymptotic performance, then architecture design becomes part of the post-training pipeline optimization, not just a pretraining decision. This is a conceptual reframing: the architecture is not a static substrate but an active participant in the learning process whose properties determine how effectively the model can improve from its own generated experience. It's a fundamentally significant result even though the mechanism is not fully explained, because it opens a new axis of architecture research that goes beyond perplexity benchmarks.

Innovation 4: The DPLR Constraint Is Not a Limitation — It's an Efficiency-Stability Pareto Improvement

Diagonal-Plus-Low-Rank (DPLR) state transition matrices have been the go-to expressive formulation for state-space models and advanced linear attention since S4 (Gu et al., 2022). The standard assumption in the literature has been that (D — a_t b_t^⊤) with independent a_t and b_t vectors provides maximal expressiveness, and any constraint on these vectors would reduce model capacity. RWKV7 (Peng et al., 2025) explicitly treats a_t and b_t as independent learnable projections, and the DPLR formulation in models like GLA and its successors uses all four pairwise interaction matrices (A_qk, A_qb, A_ak, A_ab) to capture the full expressiveness.

KDA makes a counterintuitive move: it constrains a_t and b_t to both be derived from the key vector k_t, specifically a_t = β_t k_t and b_t = k_t ⊙ α_t. On its face, this looks like a capacity reduction — the model loses degrees of freedom in its transition dynamics. But the paper demonstrates that this constraint produces a Pareto improvement: it simultaneously improves speed (roughly 2× kernel speedup over general DPLR, Figure 2), numerical stability (eliminating the need for two of four secondary chunking steps that GLA requires), and quality (KDA outperforms GDN, which itself uses an even simpler scalar decay, suggesting the fine-grained gating more than compensates for any expressiveness lost in the low-rank terms).

Why does this work? The paper doesn't fully theorize this, but the implication is that the apparently "lost" expressiveness in the DPLR formulation was redundant or harmful in practice. When a_t and b_t are independent, the transition matrix can, in principle, implement transformations that have nothing to do with the current input — it can rotate, scale, or delete state dimensions based on learned but context-independent patterns. However, because the state's purpose is to store associative mappings for later retrieval by queries that are similar to keys, the most useful corrections are precisely those aligned with the keys themselves. Letting a_t and b_t diverge introduces optimizable but non-functional degrees of freedom that increase the risk of overfitting and numerical issues without improving the model's capacity to perform its core task.

This insight is fundamental because it inverts the standard narrative about model constraints. In much of the efficient attention literature, the story is: "start with full attention's expressiveness, then approximate it with constraints to achieve efficiency, accepting some quality loss." KDA's story is: "start with the constrained form that captures the essential operation (key-aligned associative correction), then add expressiveness through fine-grained gating, achieving both efficiency and quality gains." The constraint is not a compromise — it's an inductive bias that eliminates harmful degrees of freedom while preserving the degrees that matter. This reframing has implications beyond KDA: it suggests that other linear attention variants might benefit from similar "over-constrained" formulations that tie corrective terms to the input structure more tightly than the general mathematics would allow.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses the MATH benchmark (Hendrycks et al., 2021) for all primary experiments, specifically the split from Lightman et al. (2022): 12,000 training questions and 500 test questions covering high-school competition-level mathematics. For pretraining, all models use a 1.4 trillion token corpus sampled from the K2 pretraining dataset, with a 4,096-token context window. Evaluation spans three categories: language understanding and reasoning (HellaSwag, ARC-Challenge, Winogrande, MMLU, MMLU-Pro, MMLU-Redux, TriviaQA, GPQA-Diamond, BBH, LiveBench), code generation (LiveCodeBench v6, EvalPlus, CRUXEval), math and reasoning (AIME 2025, MATH 500, GSM8K, HMMT 2025, PolyMath-en), long-context tasks (MRCR, RULER, Frames, HELMET-ICL, RepoQA, Long Code Arena, LongBench v2), and Chinese language understanding (C-Eval, CMMLU). Synthetic tasks (Palindrome, MQAR, Stack) use procedurally generated data with controlled sequence lengths from 256 to 2,048 tokens.

• Base model(s). All experiments use models based on the Moonlight architecture (Liu et al., 2025) with Mixture-of-Experts feedforward layers. The main experimental configuration activates 8 out of 256 experts with one shared expert, resulting in 48 billion total parameters and 3 billion active parameters per forward pass. The first layer is dense (no MoE) for training stability. The key and value head dimensions are fixed at dk = dv = 128 for all experiments. The paper also reports a final checkpoint trained on 5.7 trillion tokens to match the pretraining tokens of the Moonlight baseline. For scaling law experiments, models range from 653M to 1.7B activated parameters, all using MoE with 8 active experts from 64 total. Synthetic task experiments use small 2-layer, 2-head models with head dimension 128.

• Metrics. For pretraining, the primary metric is training and validation perplexity (PPL), where validation PPL is computed on a high-quality dataset whose distribution differs significantly from the pretraining corpus to emphasize generalization under distribution shift. For downstream evaluation, the paper reports accuracy on most benchmarks, with specific variations: generation-based evaluation with temperature 1.0 for most tasks; perplexity-based evaluation for MMLU, MMLU-Redux, GPQA-Diamond, and C-Eval; Avg@k metrics for high-variance benchmarks (AIME 2025 reports Avg@64, HMMT 2025 reports Avg@32, PolyMath-en reports Avg@4, GPQA-Diamond reports Avg@8); and Pass@1 for code generation tasks. For long-context benchmarks at 128k context length, the paper reports per-task scores and an overall average across seven benchmarks. Synthetic tasks report peak accuracy and convergence speed. RL experiments report training accuracy and test accuracy on MATH 500 and AIME 2025. Efficiency metrics include prefilling latency (seconds), time per output token (TPOT, milliseconds), and decoding acceleration ratios (× speedup versus MLA baseline), all measured with batch size 1 unless otherwise noted.

• Baselines. The paper compares against three primary baselines: (1) Full MLA — the standard full-attention Multi-Head Latent Attention architecture from DeepSeek-V3 (DeepSeek-AI et al., 2025), serving as the softmax attention reference point. (2) GDN-H — a hybrid Gated DeltaNet baseline that uses GDN layers instead of KDA layers in the same 3:1 hybrid architecture with MLA layers, identical model configuration, parameter count, and training setup. GDN uses scalar head-wise decay while KDA uses channel-wise decay, making this the key ablation for the fine-grained gating contribution. (3) Kimi Linear (RoPE) — a variant of Kimi Linear where the MLA layers use Rotary Position Embeddings instead of NoPE, testing the effect of the NoPE design choice. For synthetic tasks, Mamba2 (Dao and Gu, 2024) serves as an additional baseline. The final 5.7T checkpoint is compared against Moonlight (Liu et al., 2025), a full-attention model with 16B total parameters and 3B activated parameters. All baselines share identical training recipes, hyperparameters, and data for fair comparison.

• Generation budget / compute accounting. For pretraining comparisons, all models are trained on 1.4 trillion tokens with identical global batch size (32 million tokens), learning rate (1.1 × 10−3), optimizer (MuonClip), learning rate schedule (WSD), and context length (4,096). The paper explicitly states that "all models were trained with the same FLOPs budget and hyperparameters for a fair comparison." For scaling law experiments, models are trained with compute-optimal training following Chinchilla methodology (Hoffmann et al., 2022), with FLOPs computed analytically. For inference efficiency, compute is measured in wall-clock time: prefilling latency and TPOT at varying sequence lengths from 4K to 1M tokens, measured on consistent hardware. The theoretical FLOPs analysis (Eq. 13) provides a head-level breakdown: FLOPS_KDA(T; C, dh) = 6Tdh² + 3TCdh + TC² for chunk size C = 64, compared to FLOPS_Attn(T; dh) = 2T²dh for full attention. Memory efficiency is quantified by KV cache reduction (75% fewer layers requiring cache) and the resulting ability to increase batch size.

• Cross-validation / statistical protocol. For the hybrid ratio ablation and other architectural choices, the paper uses training and validation perplexity on held-out data, with validation performed on a distributionally different dataset to test generalization. For downstream benchmarks, the paper reports single-run results for most tasks, with GPQA-Diamond averaged across eight independent runs to mitigate high variance. The synthetic task experiments use grid search over learning rates in {5 × 10⁻⁵, 1 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³} and present the best-performing training accuracy curves. All evaluations use the LM-Harness-Evaluation framework (Biderman et al., 2024) with consistent settings across all models. The scaling law experiments (Figure 5) involve five model sizes with carefully tuned hyperparameters through grid search for the MLA baseline, while KDA uses the same configuration as the best MLA settings without modification. There is no explicit cross-validation fold strategy described for the main pretraining results — the paper relies on the multi-benchmark evaluation suite to assess robustness.

Main Quantitative Results

Synthetic Task Performance (Section 5.1, Figure 4)

Headline finding: KDA consistently achieves the highest accuracy across all three synthetic tasks as sequence length increases from 256 to 2,048, and converges significantly faster than GDN on retrieval-intensive tasks.

On the Palindrome task (reversing a sequence of random tokens): At sequence length 256, KDA achieves approximately 100% accuracy, GDN reaches roughly 90%, and Mamba2 fails entirely near 0%. As sequence length increases to 2,048, KDA maintains roughly 95–100% accuracy while GDN drops to approximately 50–60%. The convergence speed advantage is stark: at 1,024 tokens, KDA reaches near-ceiling accuracy within roughly 5,000 training steps, while GDN requires approximately 15,000 steps to approach similar performance.

On Multi-Query Associative Recall (MQAR) : At sequence length 256, KDA and GDN both achieve roughly 100% accuracy. However, as length increases to 2,048, KDA maintains roughly 90–95% accuracy while GDN drops to approximately 50–60%. Mamba2 fails completely across all lengths. At 1,024 tokens, KDA converges to near-perfect accuracy within roughly 5,000 steps, while GDN plateaus around 75–80% even after 20,000 steps.

On the Stack task (LIFO state tracking with 64 independent stacks): All methods perform similarly at sequence length 256 (roughly 95–100%). At 2,048 tokens, KDA maintains roughly 90% while GDN drops to approximately 75–80%. Mamba2 again fails near 0% across all lengths.

These results establish that KDA's channel-wise gating provides a qualitative improvement in the model's ability to use its finite memory, not merely an additive boost over GDN's scalar gate. The finding that Mamba2 — which uses multiplicative decay without the delta rule — fails on all three tasks confirms the necessity of the delta rule's corrective updates for tasks requiring precise associative recall from a compressed state.

Ablation on Key Components (Section 5.2, Table 1)

All ablations are conducted on the 16-head, 16-layer scaling law model with identical FLOPs and hyperparameters. Results are reported as training and validation perplexity (lower is better).

Hybrid ratio: Training a pure full-attention model (0:1 KDA-to-MLA) yields the worst performance: training PPL 9.45, validation PPL 5.77. The 3:1 ratio achieves the best of both metrics: training PPL 9.23, validation PPL 5.65. A 1:1 ratio produces similar validation PPL (5.66) but with higher inference cost. A 7:1 ratio matches training PPL (9.23) but degrades validation PPL to 5.70, indicating that insufficient global attention hurts generalization. A 15:1 ratio severely degrades both training PPL (9.34) and validation PPL (5.82). This validates 3:1 as the optimal quality–efficiency trade-off.

Output gate: Using a Sigmoid output gate achieves training PPL 9.23 and validation PPL 5.65. Removing the gate entirely degrades both metrics to 9.25 and 5.67. Using Swish gating (as in the original GDN paper) performs substantially worse: training PPL 9.43, validation PPL 5.81. This confirms that Sigmoid gating is superior and aligns with findings from Qiu et al. (2025).

Convolution layer: Removing the short convolution degrades training PPL to 9.29 and validation PPL to 5.70 (from 9.23 and 5.65 with convolutions), confirming that lightweight depthwise convolutions play a "non-negligible role" even in hybrid architectures.

NoPE vs. RoPE: This ablation is not in Table 1 but appears in Table 5 (long-context benchmarks). Kimi Linear with NoPE achieves RULER 84.3, MRCR 29.6, and an overall long-context average of 54.5. Kimi Linear with RoPE achieves RULER 78.8 (a drop of 5.5 points), MRCR 22.0 (drop of 7.6 points), and an overall average of 51.8. The paper attributes this to a mismatch between RoPE's explicit positional bias and KDA's learned implicit bias, which "overemphasizes short-range order" and reduces flexibility for context extension.

Scaling Law Results (Section 5.3, Figure 5)

Headline finding: Kimi Linear achieves approximately 1.16× computational efficiency over MLA with compute-optimal training.

The paper trains five MoE models of increasing size (653M to 1.7B activated parameters) with 8 active experts from 64 total, using the Muon optimizer. The fitted scaling law curves (Figure 5) show:

MLA: Loss = 2.3092 × C⁻⁰·⁰⁵³⁶ Kimi Linear: Loss = 2.2879 × C⁻⁰·⁰⁵²⁷

where C is compute in PFLOP/s-days. The exponents are nearly identical (−0.0536 vs. −0.0527), indicating that KDA and MLA scale similarly with compute — the benefit comes from a favorable constant factor (lower intercept), not a better scaling exponent. The paper notes that this is a conservative estimate because KDA's hyperparameters were not independently tuned: "We expect that careful hyperparameter tuning will yield superior scaling curves for KDA." The MLA baselines, by contrast, were carefully tuned through grid search for optimal performance at each model size.

Pretraining Results at 1.4T Tokens (Section 5.5.1, Table 3)

Headline finding: Kimi Linear consistently outperforms both MLA and GDN-H across short-context pretraining evaluations.

On general knowledge benchmarks, Kimi Linear leads on all metrics:

• HellaSwag: 82.9 (vs. MLA 81.7, GDN-H 82.2)
• ARC-Challenge: 67.3 (vs. MLA 64.6, GDN-H 66.5)
• Winogrande: 78.6 (vs. MLA 78.1, GDN-H 77.9)
• BBH: 72.9 (vs. MLA 71.6, GDN-H 70.6)
• MMLU: 73.8 (vs. MLA 71.6, GDN-H 72.2)
• MMLU-Pro: 51.0 (vs. MLA 47.2, GDN-H 47.9) — a notable 3.8-point margin over MLA
• TriviaQA: 71.7 (vs. MLA 68.9, GDN-H 70.1)

On math and code:

• GSM8K: 83.9 (vs. MLA 83.7, GDN-H 81.7)
• MATH: 54.7 (tied with MLA 54.7; GDN-H 54.1)
• EvalPlus: 60.2 (vs. GDN-H 63.1, MLA 59.5) — GDN-H leads here
• CRUXEval-I-cot: 56.6 (vs. GDN-H 56.0, MLA 51.6)
• CRUXEval-O-cot: 62.0 (vs. MLA 61.5, GDN-H 58.1)

On Chinese benchmarks:

• CEval: 79.5 (vs. MLA 79.3, GDN-H 79.1)
• CMMLU: 80.8 (vs. GDN-H 80.7, MLA 79.5)

The performance hierarchy is consistent: Kimi Linear > GDN-H > MLA on the vast majority of tasks. The only exception is EvalPlus where GDN-H leads. This establishes that linear attention can outperform full attention at matched scale on standard short-context pretraining evaluations — a result that prior work had not demonstrated.

SFT Results at 1.4T Tokens (Section 5.5.1, Table 4)

Headline finding: After identical supervised fine-tuning, Kimi Linear maintains its advantage over both MLA and GDN-H on instruction-following and reasoning benchmarks.

On general instruction benchmarks:

• BBH: 69.4 (vs. MLA 68.2, GDN-H 68.5)
• MMLU: 77.0 (vs. MLA 75.7, GDN-H 75.6)
• MMLU-Pro: 67.4 (vs. MLA 65.7, GDN-H 64.8)
• MMLU-Redux: 80.3 (vs. MLA 79.2, GDN-H 78.7)
• GPQA-Diamond (Avg@8): 62.1 (vs. GDN-H 58.6, MLA 57.1) — a notable 5.0-point margin over MLA

On math reasoning:

• AIME 2025 (Avg@64): 21.3 (vs. GDN-H 21.1, MLA 20.6)
• MATH500: 81.2 (vs. GDN-H 83.0, MLA 80.8) — GDN-H leads here
• HMMT 2025 (Avg@32): 12.5 (vs. MLA and GDN-H both at 11.3)
• PolyMath-en (Avg@4): 43.6 (vs. GDN-H 41.5, MLA 41.3)

On code:

• LiveCodeBench v6 (Pass@1): 26.0 (vs. GDN-H 25.4, MLA 25.1)
• EvalPlus: 61.0 (vs. MLA 62.6, GDN-H 62.5) — both baselines lead here

The SFT results largely preserve the pretraining hierarchy, with Kimi Linear leading on 8 of 11 benchmarks. The exceptions (MATH500, EvalPlus) are small margins, and the large gains on GPQA-Diamond and PolyMath-en are particularly noteworthy.

Long-Context Performance (Section 5.5.1, Table 5)

Headline finding: Kimi Linear achieves the highest overall average across long-context benchmarks (54.5), with particularly large margins on RULER (84.3 vs. MLA 81.3) and RepoQA (68.5 vs. MLA 63.0).

Evaluated at 128k context length across seven benchmarks:

• RULER: 84.3 (vs. MLA 81.3, GDN-H 80.5, Kimi Linear with RoPE 78.8) — 3.0-point lead over MLA
• MRCR: 29.6 (vs. GDN-H 23.9, MLA 22.6, Kimi Linear with RoPE 22.0) — 7.0-point lead over MLA
• HELMET-ICL: 90.0 (vs. MLA and Kimi Linear with RoPE both 88.0, GDN-H 85.5)
• LongBench V2: 35.0 (vs. MLA 36.1, Kimi Linear with RoPE 35.4, GDN-H 32.6) — MLA leads here
• Frames: 58.8 (vs. MLA 60.5, Kimi Linear with RoPE 59.9, GDN-H 58.7) — MLA leads here
• RepoQA: 68.5 (vs. MLA and GDN-H both 63.0, Kimi Linear with RoPE 66.5) — 5.5-point lead
• Long Code Arena (Library): 37.1 (vs. GDN-H 34.7, MLA 32.8, Kimi Linear with RoPE 31.3) — 4.3-point lead
• Long Code Arena (Commit): 32.7 (vs. MLA 33.2, Kimi Linear with RoPE 32.5, GDN-H 30.5) — MLA leads here
• Overall average: 54.5 (vs. MLA 52.2, Kimi Linear with RoPE 51.8, GDN-H 51.2)

The hierarchy shifts from pretraining: GDN-H falls behind MLA on the long-context average (51.2 vs. 52.2), while Kimi Linear maintains a clear lead. The NoPE design choice is validated by the Kimi Linear (RoPE) results, which degrade on RULER (78.8 vs. 84.3) and MRCR (22.0 vs. 29.6). Kimi Linear wins on 5 of 8 metrics, MLA on 3. The RepoQA result (68.5 vs. 63.0) is particularly striking as it tests code repository understanding, a task where precise retrieval matters.

RL Results (Section 5.5.1, Figure 6)

Headline finding: During RL training on math reasoning, Kimi Linear achieves faster convergence and higher final accuracy than MLA on both training and test sets.

Both models start from similar accuracy on the training set. As RL training progresses:

• Training accuracy: Kimi Linear's growth rate is "significantly higher" than MLA's, and the gap "gradually widens." By the end of training, Kimi Linear reaches approximately 65–70% accuracy while MLA reaches roughly 50–55%.
• MATH 500 test accuracy: Kimi Linear reaches approximately 90–94% while MLA reaches roughly 80–86%. The improvement trajectory is consistently steeper for Kimi Linear.
• AIME 2025 test accuracy: Kimi Linear reaches approximately 20–25% while MLA reaches roughly 15–20%. Again, the gap widens over training.

The paper states: "In reasoning-intensive long-form generation under RL, we empirically observe that Kimi Linear performs significantly better than MLA." All RL algorithm hyperparameters and training recipes are kept identical between the two models, ensuring that the performance difference is attributable to the architecture, not the RL setup.

Efficiency Results (Section 5.6, Figures 1, 7)

Headline finding: Kimi Linear achieves 2.9× faster prefilling and 6.3× faster decoding than MLA at 1M context length, while matching GDN-H's speed despite its additional fine-grained gating.

Prefilling latency (Figure 7a): At 4K context length, all three models (MLA, GDN-H, Kimi Linear) show similar latency (~5 seconds). At 128K, MLA requires roughly 40 seconds while Kimi Linear and GDN-H require roughly 14–15 seconds (2.9× speedup). At 1M, MLA requires roughly 60 seconds while Kimi Linear requires roughly 21 seconds (2.9× speedup), essentially matching GDN-H. The curves for Kimi Linear and GDN-H are "virtually indistinguishable" (Section 5.6), confirming that the fine-grained gating introduces negligible overhead.

Time per output token (Figure 7b): At 4K context, TPOT is similar across all models (~5–6ms). At 128K, MLA reaches approximately 7–8ms while Kimi Linear and GDN-H remain near 5ms. At 1M, MLA reaches approximately 11.48ms while Kimi Linear achieves 1.84ms (6.3× speedup with larger batch sizes enabled by memory savings). Even at batch size 1 (Figure 7b), Kimi Linear achieves 2.2× faster TPOT at 1M context.

Throughput scaling (Figure 1b): Because KDA layers eliminate the KV cache (75% memory reduction), the saved memory can be repurposed for larger batch sizes. The theoretical maximum throughput improvement is 6.3× at 1M tokens, achieved by maximizing batch size given the available GPU memory. The paper reports TPOT at this maximized batch size: 1.84ms for Kimi Linear vs. 11.48ms for MLA.

Kernel-level speed (Figure 2): Comparing the KDA kernel against a general DPLR kernel at varying input lengths (2K to 64K, batch size 1, 16 heads), KDA achieves roughly 2× higher speed. At 64K input length, DPLR requires approximately 64ms while KDA requires approximately 32ms. At 2K, DPLR requires roughly 4ms while KDA requires roughly 2ms. The speedup is roughly constant across sequence lengths.

Ablation Studies and Robustness Checks

Hybrid ratio (Table 1): The 3:1 KDA-to-MLA ratio is validated against 0:1, 1:1, 7:1, and 15:1. The finding that 7:1 matches training PPL but degrades validation PPL suggests that insufficient global attention layers cause overfitting to the training distribution. The 15:1 ratio's substantial degradation confirms that there is a minimum density of full-attention layers required for generalization. The 0:1 ratio's poor performance — worse than any hybrid — is a non-obvious result: pure full attention at this scale underperforms the hybrid, suggesting that the linear layers' compressive inductive bias provides a beneficial regularization effect. This is not a typical finding in the hybrid attention literature, where full attention is usually the quality ceiling.

Output gate activation function (Table 1): Swish gating, used by the original GDN paper, performs substantially worse than Sigmoid (validation PPL 5.81 vs. 5.65). The paper attributes this to Sigmoid's sharper saturation behavior, which may provide cleaner gating decisions. The no-gate variant (5.67) performs between Swish and Sigmoid, suggesting that gating is beneficial but the choice of activation matters significantly — a detail that prior work had not systematically explored.

Convolution layer removal (Table 1): Removing the depthwise convolutions with kernel size 4 degrades validation PPL by 0.05 (5.65 to 5.70). While this seems small, the consistent effect across both training and validation PPL suggests that local token dependencies captured by convolutions provide information that the linear attention's fixed-size state does not fully recover. The paper's reference to Allen-Zhu (2025) on the "magic of canon layers" positions convolutions as a structurally important component rather than an optional add-on.

NoPE vs. RoPE in long-context (Table 5): The degradation on RULER (84.3 to 78.8) and MRCR (29.6 to 22.0) when RoPE is added to MLA layers is a robustness check for the claim that KDA's learned positional encoding is sufficient as the sole positional mechanism. The result that short-context scores remain similar (Table 5 doesn't show this directly, but the paper states "Kimi Linear (RoPE) attains similar scores on short-context tasks" in Section 5.2) validates that the benefit of NoPE is specifically in long-context generalization, not in absolute quality.

KDA vs. GDN-H across training stages: The performance hierarchy shifts meaningfully across training stages. During pretraining (Table 3): Kimi Linear > GDN-H > MLA. During SFT (Table 4): same hierarchy. During long-context evaluation (Table 5): Kimi Linear > MLA > GDN-H. This shift — GDN-H falling behind MLA on long-context despite outperforming it on short-context — is a robustness check that validates KDA's channel-wise gating specifically improves long-context retention, not just general quality. It also demonstrates that the pretraining advantage doesn't automatically transfer to all downstream settings.

Extended training to 5.7T tokens (Appendix D, Tables 8–9): The final Kimi Linear checkpoint trained on 5.7T tokens (matching Moonlight's pretraining budget) consistently outperforms Moonlight across nearly all benchmarks. On the base model (Table 8): MMLU-Pro 54.8 vs. 42.4, GPQA-Diamond 40.4 vs. 35.2, MATH 58.5 vs. 45.3, LiveCodeBench 20.0 vs. 14.3. On the instruction-tuned model (Table 9): RULER at 1M context reaches 94.8, GPQA-Diamond 71.7 vs. 24.7, MATH500 94.6 vs. 58.0, LiveCodeBench v6 45.7 vs. 11.9. The 1M-context RULER score (94.8) demonstrates that the architecture scales to million-token contexts while maintaining high accuracy on challenging retrieval tasks — the hybrid design overcomes the retrieval bottleneck that plagues pure linear attention.

Mamba2 failure on synthetic tasks (Figure 4): Mamba2's complete failure on Palindrome, MQAR, and Stack is a strong negative result that validates the necessity of the delta rule. Mamba2 uses multiplicative decay with data-dependent scalar gates but lacks corrective updates, confirming that selective forgetting alone is insufficient for tasks requiring precise associative recall from a compressed state.

Critical Assessment

Claim 1: Kimi Linear outperforms full attention under fair comparisons across various scenarios.

This claim is substantially supported, but with important qualifications about what "fair comparisons" means.

The 1.4T-token matched pretraining results (Tables 3–5) provide the strongest evidence. Kimi Linear outperforms MLA on the majority of benchmarks across pretraining (12 of 15 metrics in Table 3), SFT (8 of 11 in Table 4), and long-context (5 of 8 plus highest average in Table 5). The consistent margin — roughly 1–3 percentage points on most benchmarks — is modest but reproducible across categories. The RL results (Figure 6) add a qualitatively different regime where Kimi Linear's advantage compounds over training. The scaling law (Figure 5) shows a genuine efficiency gain (1.16×), not just a quality-vs-cost tradeoff.

However, "fair comparisons" deserves scrutiny on several fronts:

Hyperparameter tuning asymmetry. The paper explicitly states that for scaling laws, MLA models were "carefully tuned through grid search to ensure optimal performance for each model," while KDA "adhered strictly to the MLA training configuration without any modifications." This means the 1.16× improvement is against a well-tuned baseline using untuned KDA hyperparameters — potentially an underestimate, but also means the reported advantage is partly a function of which model got more hyperparameter optimization. The paper's claim that better tuning would yield "superior scaling curves for KDA" is plausible but unverified.

Architecture-specific optimizations. Kimi Linear inherits several design choices that are not intrinsic to the KDA mechanism but contribute to performance: short convolutions, L2Norm on queries/keys, Sigmoid output gating, low-rank projections. The ablation (Table 1) quantifies some of these, but the comparison against MLA includes all of them. A truly "fair" comparison would need to verify that MLA with equivalent enhancements (e.g., output gating on MLA layers, short convolutions before MLA attention) doesn't close the gap.

The 3:1 ratio advantage is partly about model depth allocation. At a fixed total layer count, Kimi Linear has 75% KDA layers and 25% MLA layers. A fair comparison against a hybrid with a different linear component (GDN-H) controls for the hybrid ratio, but comparing against pure MLA at the same total layer count doesn't control for the possibility that pure MLA with more layers (since each KDA layer has fewer parameters? — the paper claims "similar number of parameters," so this may not apply) would perform differently.

Claim 2: The channel-wise gating mechanism enables more effective use of limited finite-state RNN memory.

This claim is convincingly supported by the synthetic experiments, with the caveat that the mechanism is demonstrated rather than explained.

The synthetic results (Figure 4) isolate the gating mechanism: KDA vs. GDN is a direct comparison of channel-wise vs. scalar gating, with all other factors controlled. KDA's faster convergence on MQAR and Palindrome — particularly the widening gap at longer sequences — directly supports the claim that fine-grained gating improves memory utilization. Mamba2's failure demonstrates that gating alone (without delta rule) is insufficient.

However, the paper does not provide a mechanistic analysis of how the channel-wise gate improves memory. Does the model learn to allocate different channels to different temporal scales (some retaining information for thousands of tokens, others rapidly forgetting)? Does it learn content-based gating (e.g., retaining named entities longer than function words)? The synthetic tasks are too simple to answer these questions, and the language modeling experiments provide only aggregate metrics. An analysis of the learned α_t values across channels — showing, for example, that some channels consistently maintain α ≈ 1 for long-range dependencies while others use α ≈ 0 for local patterns — would strengthen this claim substantially.

The Palindrome task is also an unusual benchmark for testing memory: it requires exact reversal of a sequence, which is notoriously difficult for compressed-state models. KDA's near-perfect performance even at length 2,048 raises the question of whether the model learned a genuine reversal strategy or exploited some shortcut in the task design. The paper doesn't provide error analysis or example outputs that would clarify this.

Claim 3: The hybrid architecture reduces memory footprint while surpassing full-attention quality.

The memory reduction claim is solidly supported by the measured speedups (Figures 1, 7). The 75% KV cache reduction is a direct consequence of the 3:1 ratio, and the measured 6.3× decoding throughput improvement at 1M tokens (Figure 1b) quantifies the practical impact.

The quality claim is conditionally supported — it holds on average but not uniformly. On long-context evaluation (Table 5), Kimi Linear leads on 5 of 8 metrics but trails MLA on LongBench V2 (35.0 vs. 36.1), Frames (58.8 vs. 60.5), and Long Code Arena Commit (32.7 vs. 33.2). These are small margins individually, but collectively they suggest that certain retrieval patterns still benefit from full attention's direct token access. The paper does not analyze which questions Kimi Linear gets wrong relative to MLA — such an analysis would reveal whether the remaining gap is in a particular type of retrieval (e.g., exact string matching vs. semantic similarity, single-key vs. multi-key lookup).

The SFT results (Table 4) show Kimi Linear trailing on MATH500 (81.2 vs. GDN-H 83.0) and EvalPlus (61.0 vs. MLA 62.6, GDN-H 62.5). These are the only two metrics where Kimi Linear is not the leader, and both involve precise answer verification (math answer matching, code correctness). This may indicate a residual weakness in exact-output tasks that the paper doesn't discuss. The margins are small enough that they could be noise, but the pattern across both SFT and pretraining (EvalPlus was also a weak point in Table 3 at 60.2 vs. 63.1 for GDN-H) suggests a systematic effect.

Claim 4: The 1.16× scaling law improvement represents a fundamental efficiency gain.

This claim requires more cautious interpretation than the paper provides.

The scaling law data (Figure 5) covers five model sizes from 653M to 1.7B parameters — a range of only about 2.6×. Extrapolating from this narrow range to the 3B-parameter main experiments, let alone to larger scales, assumes that the scaling exponent remains constant. The paper's own exponents are nearly identical (MLA: −0.0536, Kimi Linear: −0.0527), meaning the efficiency gain is a constant factor (lower intercept), not a superior scaling law. If the exponents diverge at larger scales — which cannot be tested without larger-scale experiments — the 1.16× figure could change. The paper's acknowledgment that KDA hyperparameters were not tuned independently for each model size further complicates the interpretation: the constant factor might improve with tuning (making the gain larger) or the MLA baselines might not be at their true compute-optimal frontier (making the gain smaller).

More importantly, "compute-optimal training" in the Chinchilla sense (Hoffmann et al., 2022) involves scaling both model size and training tokens. The paper's scaling law models are trained with fixed architecture configurations across sizes, not with independently optimized token budgets. It's unclear whether the fitted curves represent true compute-optimal frontiers or simply the performance of a specific model family at varying sizes.

Missing experiments that would strengthen the paper

No combination of KDA with PRM search or other advanced decoding strategies. The paper presents KDA as a drop-in replacement for attention, but all evaluations use standard autoregressive decoding. Given the growing literature on test-time compute scaling (e.g., best-of-N, beam search against verifiers), understanding how KDA's fixed-state memory interacts with search strategies — particularly whether the compressed state limits the effectiveness of beam search or revision-based refinement — would be valuable.

No analysis of attention patterns or state utilization. The paper makes claims about memory management but provides no visualization or analysis of what the channel-wise gates learn in practice. Do different channels specialize in different temporal ranges? Do the decay rates α_t correlate with linguistically meaningful features (e.g., sentence boundaries, entity mentions, topic shifts)? Without such analysis, the mechanism remains a black box.

Limited model family diversity. All experiments use the Moonlight MoE architecture. It's unclear whether the 3:1 hybrid ratio, the NoPE design, or the KDA parameterization would transfer to dense architectures, different model scales, or different training objectives. The paper positions Kimi Linear as a general architecture, but all evidence comes from a single architectural template.

No latency-aware evaluation of the RL benefit. The RL results (Figure 6) show that Kimi Linear achieves higher accuracy, but the paper doesn't report whether this is achieved with better wall-clock efficiency. If KDA's faster decoding allows more RL training steps per unit time, the advantage might be even larger; if the RL training is bottlenecked by other factors, the accuracy gain might come at similar computational cost. This matters for the paper's claim about enabling "agentic intelligence and test-time scaling without compromising quality."

Single long-context length (128k) for most evaluations. The paper claims support for up to 1M tokens (and reports RULER at 1M for the 5.7T model in Table 9), but the detailed 1.4T comparison (Table 5) uses only 128k. The scaling behavior between 128k and 1M — where the efficiency advantages are largest — is only demonstrated for speed (Figures 1, 7), not for accuracy across the full benchmark suite.

Summary

The experimental evidence supports the paper's central thesis that a carefully designed hybrid architecture with channel-wise gated delta rule attention can match or surpass full attention across diverse settings. The evidence is strongest for the efficiency claims (speedups and memory reduction are directly measured) and for the synthetic task results (clear, controlled comparisons isolating the gating mechanism). The quality claims are supported but with small margins that occasionally reverse on specific benchmarks, and the scaling law claim should be interpreted conservatively given the limited scale range and hyperparameter tuning asymmetry. The paper would benefit from mechanistic analysis of the learned gating, broader architectural diversity, and evaluation of KDA's interaction with test-time compute strategies.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The compute-optimal policy (Sections 3.2 and 4–5) requires estimating each prompt's difficulty before deciding how to allocate the test-time budget. The method for doing so — generating 2048 samples per question, scoring them with the PRM, and binning by average score — is extraordinarily expensive. Section 3.2 states:

"our experiments do not account for this cost largely for simplicity, thereby framing this as an exploration–exploitation tradeoff that we leave to future work"

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In any realistic deployment, the total cost would be difficulty estimation + strategy execution. At 2048 samples per question, the estimation step alone exceeds the largest test-time budgets studied (256–512 generations). The 4× figure is therefore an upper bound on achievable efficiency under the optimistic assumption that difficulty can be obtained for free. In practice, the net efficiency gain could be negative — spending more compute estimating difficulty than the adaptive strategy saves — unless a cheaper estimation method is developed.

What evidence exists in the paper. The paper provides no experiment measuring the cost of difficulty estimation or its impact on net efficiency. The exploration–exploitation tradeoff is acknowledged as a limitation but never quantified. The predicted-bins curves in Figures 4 and 8 demonstrate that the PRM-based difficulty proxy works (performance is close to oracle bins), but they do not account for the 2048 samples used to generate those bins. The paper also provides no evidence that a lightweight difficulty estimator (e.g., a small classifier trained on question text) can match the PRM-based method.

Mitigation status. The paper acknowledges this explicitly and flags it as "a key avenue for future work" (Section 3.2), suggesting "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No such model is developed or evaluated. The limitation is therefore acknowledged but entirely unresolved in the current work.

---

Verifier Over-Optimization Caps the Benefits of Additional Test-Time Compute

The assumption or constraint. The compute-optimal framework relies on a learned Process Reward Model (PRM) to score candidate solutions and guide search. However, the PRM is not perfectly aligned with ground-truth correctness — it can be exploited by aggressive search algorithms that find solutions scoring highly under the PRM but that are actually incorrect. Section 5.3 documents this directly:

"The degradation at high budgets is attributed to over-optimization of the PRM — search finds solutions that score highly under the PRM but are actually incorrect."

The consequence. The compute-optimal policy can only scale test-time compute up to the verifier reliability frontier. On easy problems (bin 1), this frontier is reached quickly — beam search degrades performance as budget increases (Figure 3, right), with accuracy dropping from ~78% to ~77% while best-of-N continues improving to ~88%. On medium problems, the frontier is further out but still eventually flattens beam search performance. The paper's own lookahead search — designed to be a more sophisticated optimizer — paradoxically performs worst overall (Figure 3, left), confirming that stronger optimization amplifies rather than overcomes verifier imperfections. Qualitative examples in Appendix M show degenerate outputs: repetitive low-information steps and overly short 1–2 step solutions that score highly under the PRM.

This means that no amount of additional test-time compute can break through the verifier quality ceiling. The compute-optimal policy mitigates this by routing easy problems away from aggressive search, but it does not solve the underlying problem. The scaling limits observed in Figure 3 (right) — where beam search curves flatten or decline — are fundamental to the current verifier quality and cannot be overcome by better allocation alone.

What evidence exists in the paper. Figure 3 (right) shows beam search ($M = 4$) degrading on bin 1 at high budgets; Figure 3 (left) shows lookahead search underperforming simpler methods; Appendix F, Figure 14, shows the gap between PRM best-of-N weighted (~40% at 2048 samples) and majority voting (~30%), with the PRM curve showing diminishing returns at high sample counts; Appendix M provides qualitative examples of verifier-exploiting outputs.

Mitigation status. The compute-optimal policy partially mitigates this by using weaker optimization (best-of-N) on easy problems where the PRM is reliable, and reserving beam search for medium problems where the PRM signal provides genuine guidance. However, this is routing around the problem, not solving it. The paper does not propose improved verifier training methods, adversarial robustness techniques, or ensemble verification — all of which would directly address the over-optimization bottleneck. Section 8 acknowledges that "improving the PRM" is a direction for future work but does not pursue it.

---

Hard Problems Remain Completely Unsolved — Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The paper frames test-time compute as an alternative to scaling pretraining compute (Section 1, Section 7). However, this substitution only works when the base model already produces correct solutions at some non-trivial rate. If the model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions to find or refine. The paper states in Section 2:

"test-time compute can amplify existing capability but does not create it from nothing"

Section 7, "Takeaway," is even more explicit:

"For such problems, pretraining remains the only viable path"

The consequence. Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows ~2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. A practitioner deploying Kimi Linear cannot expect it to solve genuinely novel or out-of-distribution problems — only to solve problems within the base model's existing competence more efficiently.

This is a sharp boundary on the paper's central claim. The $4\times$ efficiency gain and the "$14\times$ smaller model outperforms a larger one" result only apply to easy-to-medium problems (bins 1–4). For hard problems, test-time compute provides essentially zero benefit, and the FLOPs-matched comparison (Section 7, Figure 9) shows that pretraining is strongly preferable.

What evidence exists in the paper. Figure 3 (right, bin 5), Figure 7 (right, bin 5), Figure 9 (bin 5 curves), and the Section 7 FLOPs-matched comparison for search on hard problems: PRM search shows a -52.9% relative disadvantage versus the larger model at $R \gg 1$.

Mitigation status. The paper is transparent about this limitation and does not claim that test-time compute can solve fundamentally out-of-distribution problems. However, the paper does not provide guidance on how to predict which problems will fall into bin 5 before attempting them — the difficulty estimation mechanism classifies problems into bins, but does not predict whether a bin-5 problem could be solved with a larger model's pretraining rather than with the current model's test-time compute. This leaves practitioners with an operational gap: if a problem receives bin-5 treatment and fails, should they escalate to a larger model, and how would they know?

---

Single Benchmark, Single Model Family — Generality Is Unverified

The assumption or constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021) with 500 test questions, and all models are variants of PaLM 2-S* (Anil et al., 2023). The paper states in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

It acknowledges the scope in Section 8:

"Extending this analysis to other reasoning domains — code generation, logical reasoning, scientific QA — would determine which findings are universal and which are domain-specific."

The consequence. Several aspects of the findings could be specific to the PaLM 2 architecture, the MATH dataset's structure, or symbolic math reasoning rather than general reasoning:

• PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different error patterns might exhibit different difficulty-dependent scaling curves, potentially changing the optimal allocation policy.
• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (GPT-4 vs. PaLM vs. LLaMA).
• MATH consists of competition-level math problems requiring symbolic manipulation and multi-step deduction. Tasks requiring factual recall (e.g., TriviaQA), code generation, or open-ended reasoning may show entirely different difficulty-dependent scaling behavior.
• The PaLM 2 architecture has specific properties (attention mechanism, training data mixture, scale) that affect how it responds to test-time compute. Replication on LLaMA, GPT, or DeepSeek architectures is absent.

A practitioner cannot confidently apply the paper's specific recommendations — e.g., "use beam search with M = 4 on medium problems and best-of-N on easy problems" — to a different model family or task domain without independent validation.

What evidence exists in the paper. All experiments use MATH and PaLM 2-S*. There is no cross-model or cross-dataset validation. The paper does not discuss how model-specific properties (e.g., calibration, in-context learning strength, output diversity) might affect the results. The 500-question test set, split into five difficulty quintiles of ~100 each and further split by two-fold cross-validation, means strategy selection is based on ~50 questions per fold per bin — a small sample that raises questions about statistical reliability.

Mitigation status. The paper acknowledges this explicitly in Section 8 and suggests future work on extending to code, logic, and scientific QA benchmarks. No mitigation is attempted within the current paper. The authors frame their contribution as establishing the framework and first empirical evidence, with generalization left to future work.

---

The Revision Model Has a ~38% Correct-to-Incorrect Reversion Rate — a Fundamental Instability

The assumption or constraint. The revision model (Section 6.1) is fine-tuned on trajectories where each in-context answer is incorrect and the final answer is correct. At inference time, the model may encounter correct answers in its context (produced during earlier revisions) and, having never been trained on such cases, may incorrectly "revise" them into wrong answers. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

The consequence. The revision chain is inherently unstable — progress made in earlier steps can be undone in later steps. The paper's mitigation (using majority voting or verifier-based selection across the entire chain rather than taking the final output) reduces but does not eliminate this problem. This fundamentally limits the length of useful revision chains and means the revision model cannot be relied upon for monotonic improvement. For a practitioner, this means deploying the revision model in an agentic setting where each revision is used for downstream decisions is risky — an early correct answer might be revised to an incorrect one before the chain-level selection mechanism catches it.

The ReST$^{EM}$ experiment (Appendix K, Figure 16) further demonstrates the fragility of revision training. Attempting to improve the revision model with on-policy data collection caused performance to degrade substantially — at 256 generations, fully sequential performance dropped to ~33.5% versus ~38.5% at the optimal ratio. The paper hypothesizes that on-policy data exacerbated spurious correlations, but does not resolve the issue.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. Figure 6 (left) shows the revision model's pass@1 at each step improving gradually but eventually plateauing, consistent with correct answers being lost and regained. Appendix K (Figure 16) shows the ReST$^{EM}$ failure. The paper does not provide a step-by-step breakdown of how many correct answers are lost at each revision depth.

Mitigation status. The paper mitigates with majority voting and verifier-based selection across the chain, which reduces the impact but does not address the root cause. The authors do not propose training the model to recognize when no revision is needed (e.g., by including "correct-to-correct" trajectories in the training data). The ReST$^{EM}$ failure is presented as a negative result without a solution, suggesting the training procedure is sensitive in ways not fully understood. A practitioner implementing revisions would need to accept the 38% reversion rate as an operational constraint and design their system accordingly (e.g., by keeping a copy of the best answer seen so far across all steps).

---

The $14\times$ Larger Model Baseline Is Weaker Than a Compute-Optimal Pretraining Baseline Would Be

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022). The paper states:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses greedy decoding with no test-time compute augmentation — no majority voting, no best-of-N, no search.

The consequence. The reported advantages of test-time compute over pretraining — e.g., +27.8% on easy questions at $R \ll 1$ for revisions — are measured against a baseline that is weaker than it could be in two respects: (1) a Chinchilla-optimal large model (scaling both parameters and data) would likely outperform the parameter-only-scaled model; (2) giving the larger model even a modest test-time compute budget (e.g., best-of-8 or majority voting) would significantly improve its performance, potentially closing or reversing the gap. The paper is comparing "small model + heavily optimized test-time strategy" against "large model + naive decoding," which is not the right counterfactual for a practitioner choosing between the two.

What evidence exists in the paper. The paper provides no ablation giving the larger model any test-time compute. Section 7 Figure 9 shows the comparison as-is. The dependence on the specific parameter-scaling strategy is acknowledged but not quantified.

Mitigation status. The paper is transparent about this design choice and explicitly frames it as a limitation left to future work. However, the FLOPs-matched comparison is presented as one of the paper's headline findings (appearing in the abstract and Figure 1), and the caveat is buried in Section 7 rather than prominently flagged. A practitioner reading the abstract and Figure 1 might reasonably conclude that test-time compute is broadly superior to pretraining, without realizing the baseline is weakened by both the parameter-only scaling and the lack of test-time compute augmentation. A more informative comparison would have matched the total budget including some test-time compute for the larger model, or shown sensitivity to the pretraining scaling strategy.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that linear attention can be a strict upgrade over softmax attention — not just a cheaper approximation. That is a shift in the burden of proof for the field. Prior to Kimi Linear, the default assumption was that full attention sets the quality ceiling, and efficient alternatives must justify themselves by trading quality for speed. The best that hybrid architectures like Jamba or MiniMax-01 had demonstrated was being "not too much worse." Kimi Linear's matched-scale results — outperforming full MLA on 12 of 15 pretraining metrics, 8 of 11 SFT metrics, and achieving a higher long-context average (Table 3–5) — invert that assumption. The question becomes not "can linear attention catch up?" but "under what conditions does full attention remain necessary?"

This reframing has a specific, operational consequence: the KV cache is no longer an architectural invariant. For a generation of LLM practitioners, the linearly-growing KV cache has been treated as an unavoidable cost of the transformer architecture — something to optimize with quantization, paging, or prefix caching, but never to eliminate. Kimi Linear demonstrates that 75% of layers can operate without any KV cache at all, reducing memory pressure and enabling larger batch sizes (Figure 1b: 6.3× throughput improvement at 1M tokens). This makes linear attention a first-class systems design choice rather than a research curiosity.

The paper also reconciles a tension in the hybrid architecture literature that had not been explicitly articulated. Previous hybrid models (Jamba, various Mamba-Transformer combinations) used uniform positional encoding — RoPE everywhere — implicitly treating positional information as a per-layer property. Kimi Linear's result that adding RoPE to MLA layers degrades long-context performance (RULER drops from 84.3 to 78.8, Table 5) while matching short-context scores provides a specific diagnosis: positional encoding in hybrid models is a system-level resource, not a per-layer knob. When linear attention layers carry a learned, data-dependent positional bias and full attention layers carry a fixed, trigonometric positional bias, they conflict — the global layer overemphasizes short-range patterns from training, reducing flexibility at extended lengths. The NoPE-for-MLA design resolves this by centralizing all positional responsibility in the linear layers. This is a concrete, transferable principle: future hybrid architectures should assign positional encoding to exactly one component type, not duplicate it across heterogeneous layers.

For the broader efficient-attention research community, this paper reshapes the research frontier in two ways:

Research directions that become more attractive. First, co-designing gating granularity and correction mechanisms — the paper's core architectural insight is that channel-wise gating and the delta rule interact multiplicatively, not additively. This opens a design space where future work should not separately optimize "better gates" or "better update rules" but should treat them as a joint optimization problem. Second, verifier-free architecture evaluation at RL scale — the RL results (Figure 6) showing that Kimi Linear trains faster and achieves higher asymptotes than MLA under identical RL hyperparameters suggests that architecture choice affects post-training dynamics in ways that pretraining perplexity does not predict. This makes architecture evaluation a necessary part of RL pipeline development, not just pretraining. Third, constrained rather than general DPLR formulations — the paper's finding that constraining a_t and b_t to both derive from k_t improves both speed (~2× kernel speedup over DPLR, Figure 2) and quality challenges the assumption that maximal expressiveness is always desirable in state transition matrices, suggesting that other "over-constrained" formulations warrant investigation.

Research directions that become less attractive. First, pure linear attention without hybrid components — the paper's synthetic experiments (Figure 4) confirm that even KDA, with its fine-grained gating and delta rule, cannot fully solve Palindrome and MQAR at extended lengths without occasional full-attention layers. This suggests that "fully linear" architectures face a fundamental retrieval ceiling that additional gating sophistication cannot break. Second, headwise hybridization — the paper explicitly argues that layerwise hybridization provides "superior infrastructure simplicity and training stability" compared to mixing attention types within layers. Given the strong empirical results with the simple 3:1 interleaving, the added complexity of headwise mixing (as in Hymba) requires stronger justification than previously assumed. Third, RoPE as a universal positional encoding default — the NoPE-for-global-layers result (Table 5) suggests that RoPE's benefits are specifically tied to architectures where all layers share the same attention mechanism, and that hybrid architectures may achieve better long-context generalization by shedding explicit positional encodings in favor of learned, content-dependent alternatives.

The paper's most subtle but potentially impactful contribution is the online learning reframing of the delta rule. By presenting KDA as performing gradient descent on a decayed state (Table 7: St = S̃_{t-1} - ∇_{S̃_{t-1}} L), the paper connects linear attention design to optimization theory in a way that enables principled reasoning about memory management. The channel-wise decay becomes a per-dimension learning rate schedule, and the delta rule becomes a corrective gradient step. This lens suggests that advances in online learning — adaptive step sizes, momentum, second-order methods — could be directly translated into linear attention improvements. The paper doesn't explore this connection deeply, but it provides the conceptual vocabulary for doing so.

Follow-Up Research This Work Enables

Mechanistic analysis of learned channel-wise decay patterns across linguistic structure. The paper demonstrates that channel-wise gating improves memory utilization (Figure 4: KDA converges faster than GDN on MQAR), but provides no analysis of what the gates learn. A strong follow-up would probe the learned α_t values on a diverse corpus (e.g., the Pile, long-form narratives, code repositories) with specific hypotheses: Do certain channels consistently maintain α ≈ 1 for thousands of tokens while others rapidly decay? Do decay rates correlate with linguistically meaningful boundaries — sentence ends, paragraph breaks, topic shifts, entity introductions? Do different heads specialize in different temporal scales (e.g., some heads retaining information over entire documents while others track local syntax)? The experiment would involve recording per-channel α_t values during inference on structured text, clustering channels by their temporal profiles, and testing whether ablating specific channel clusters (e.g., zeroing out all fast-decay channels) produces predictable deficits in specific capabilities (local vs. long-range reasoning).

Combining KDA with advanced decoding strategies under compute budgets. The paper evaluates KDA only with standard autoregressive decoding, but the growing literature on test-time compute scaling (best-of-N, beam search, verifier-guided search, iterative revision) raises a specific, testable question: does KDA's fixed-size state interact differently with these strategies than full attention's direct token access? The key hypothesis is that KDA's compressive memory may limit the effectiveness of strategies that rely on precise token-level retrieval (e.g., beam search that needs to backtrack to specific previous states), while potentially amplifying strategies that benefit from structured representations (e.g., verifier-guided refinement that operates on the model's compressed understanding). A concrete experiment would train a process reward model (PRM) on both Kimi Linear and MLA outputs, then compare the scaling curves of best-of-N, beam search, and revision chains on MATH or a similar reasoning benchmark, measuring whether the optimal strategy per difficulty level differs between the two architectures.

Training a lightweight classifier for instant difficulty estimation to close the exploration–exploitation gap. The paper flags the cost of difficulty estimation (2048 samples per question) as a key unresolved limitation. A direct follow-up would train a small model — perhaps a single transformer layer or even a linear classifier on top of KDA's own intermediate state representations — to predict difficulty bins from the question text alone. The training data would be the 2048-sample PRM scores already generated for the test set. The evaluation would measure: (1) correlation between predicted and oracle difficulty bins on held-out questions, (2) whether the compute-optimal policy using predicted bins recovers the same 4× efficiency gains as PRM-based bins (Figures 4, 8), and (3) the total computational cost (difficulty estimation + strategy execution) versus a uniform best-of-N baseline, finally providing a net efficiency metric that includes the estimation cost. A negative result — finding that lightweight classifiers cannot match PRM-based difficulty estimation, or that the estimation cost dominates the savings — would clarify whether the compute-optimal framework is fundamentally limited to offline/batch settings.

Stress-testing the 3:1 hybrid ratio on code generation and multi-hop reasoning. The paper's primary evaluations are language understanding and math, with code generation appearing in fewer benchmarks. Code generation and multi-hop reasoning impose different demands on attention: code requires precise variable-name retrieval across potentially thousands of tokens, and multi-hop reasoning requires chaining facts that may be far apart in the context. These are precisely the tasks where linear attention's retrieval bottleneck (Section 7.2: "pure linear attention still struggle with precise memory retrieval and exact copying") should be most exposed. A strong follow-up would evaluate Kimi Linear with varying hybrid ratios (1:1, 3:1, 7:1, 15:1) on RepoQA (code repository understanding), LongBench v2 (multi-hop reasoning), and SWE-bench (real-world code editing). The specific hypothesis is that the optimal ratio for code tasks is lower (more full attention) than for language understanding, and that a task-adaptive ratio — using difficulty estimation to select how many full-attention layers to activate per sequence — could outperform any fixed ratio.

Investigating whether KDA's state can be distilled into a larger model's pretraining or fine-tuning. The paper shows that a small model with KDA can approach the quality of a larger full-attention model (the ~14× comparison). A natural extension asks whether the learned state representations in KDA — which compress long contexts into fixed-size matrices — can serve as training signals for larger models. Specifically: can a large full-attention model be fine-tuned to predict KDA's state representations as auxiliary targets, thereby learning to internalize the compressive inductive bias? Or conversely, can KDA states generated during long-context processing be used as lightweight context summaries for downstream models? The experiment would fine-tune a full-attention model with an auxiliary loss that encourages its intermediate representations (at certain layers) to match the corresponding KDA states from a frozen Kimi Linear teacher, then evaluate whether the student model shows improved long-context performance or faster RL convergence.

Replicating the RL advantage finding with controlled probes into state utilization. The paper's RL results (Figure 6) show that Kimi Linear trains faster and achieves higher asymptotes than MLA on math reasoning, but the mechanism is unexplained. A focused follow-up would instrument both models during RL training to track: (1) how frequently each model revisits and revises intermediate results during chain-of-thought generation, (2) the effective context length that each model's attention patterns span (for MLA) or that the state retains (for KDA, via per-channel decay analysis), and (3) whether Kimi Linear's advantage is concentrated in problems requiring multi-step state tracking (where the compressive state helps maintain intermediate values) versus single-step retrieval (where MLA's direct access might be superior). The experiment would bin RL training problems by the number of reasoning steps required and test whether the Kimi Linear advantage widens with reasoning depth.

Practical Applications and Downstream Use Cases

Long-context serving infrastructure for agentic LLMs. The most immediate application is deploying Kimi Linear as a drop-in replacement for full-attention models in serving systems that handle million-token contexts. The paper reports that at 1M tokens, Kimi Linear achieves 1.84ms time per output token versus 11.48ms for MLA — a 6.3× improvement when batch sizes are maximized (Figure 1b). For a production system serving agentic workloads (where each query may involve multi-turn tool use, long document retrieval, or repository-level code analysis), this translates directly to 6.3× higher throughput per GPU, or equivalently, the ability to serve the same query volume with ~84% fewer GPUs. The 75% KV cache reduction (only 25% of layers require caching) means that the maximum batch size on a fixed-memory GPU is roughly 4× larger, which is particularly impactful for systems that are memory-bound rather than compute-bound during decoding. The open-source KDA kernel with vLLM integration makes this immediately deployable.

RL post-training pipelines for reasoning models. The RL results (Figure 6) show that Kimi Linear achieves faster convergence and higher final accuracy than MLA under identical RL hyperparameters — on MATH500, Kimi Linear reaches ~94% versus MLA's ~86%, and on AIME 2025, the gap widens over training. For organizations running large-scale RL post-training (e.g., to create reasoning-specialized models from base checkpoints), this means potentially shorter training wall-clock time to reach a target accuracy, or higher final accuracy for a fixed training budget. Moreover, KDA's faster decoding speed means that each RL training step — which involves generating long chain-of-thought traces, scoring them, and updating the model — completes more quickly. If decoding dominates the RL step time (as is typical for reasoning models), the per-step speedup could compound with the sample efficiency gain, yielding a total RL training throughput improvement larger than either factor alone. The paper doesn't quantify this combined effect, but the individual components (6.3× decoding speedup + steeper RL accuracy curves) suggest it could be substantial.

On-device or memory-constrained deployment of long-context models. The 75% KV cache reduction has direct implications for deploying language models on consumer hardware. A 3B-parameter model with full attention processing a 128K-token context requires a KV cache of roughly 128K × num_layers × 2 × d_model × 2 bytes (for FP16) — for a typical architecture with 32 layers and d_model = 2048, this is approximately 128K × 32 × 2 × 2048 × 2 ≈ 33.6 GB for the KV cache alone, exceeding the memory of most consumer GPUs. Kimi Linear with the 3:1 ratio reduces this to ~8.4 GB (75% reduction), making 128K-context inference feasible on a 24 GB consumer GPU like an RTX 4090. For the on-device deployment scenario that the paper mentions (Section 1: edge devices, real-time interactivity), this is the difference between viable and impossible. The 1M-context RULER score of 94.8 (Table 9) demonstrates that the quality does not collapse at these lengths, making the memory reduction practically meaningful.

Batch inference for document processing and data synthesis. Organizations that process large document collections (legal document review, scientific literature analysis, customer feedback aggregation) often run batch inference where throughput matters more than per-query latency. Kimi Linear's ability to support larger batch sizes due to reduced per-sequence memory — the 6.3× throughput figure in Figure 1b explicitly accounts for batch size scaling — means that a fixed GPU cluster can process 6.3× more documents per unit time. For data synthesis pipelines (generating training data, performing knowledge distillation, running model-based evaluations), where thousands or millions of long-context inferences are run as batch jobs, this directly reduces cost and latency. The paper's efficiency curves (Figure 7) show that the advantage over MLA grows with sequence length: at 4K, the speedup is minimal; at 128K, it's 2.3×; at 1M, it's 2.9× for prefilling and 6.3× for decoding. This means the architecture is specifically suited to the regime where attention costs dominate — long documents, not short prompts.