MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling

ArXiv: 2602.11761

🎯 Pitch

A 9B hybrid model combining sparse and linear attention achieves up to 3.5× faster inference than full-attention models at 256K tokens on a single GPU, and handles 1M-token contexts where same-sized Transformers crash from memory overload. The trick is a 1:3 layer ratio and a continual training recipe that repurposes an existing Transformer at a quarter of the usual cost.

---

1. Executive Summary

This paper introduces MiniCPM-SALA, a 9B-parameter hybrid architecture that integrates sparse attention—InfLLM-V2 for high-fidelity long-context modeling—with linear attention—Lightning Attention for global O(N) efficiency—in a 1:3 layer ratio to overcome the quadratic compute and KV-cache memory bottlenecks of full-attention Transformers on ultra-long sequences. Using a continual-training framework that converts a pre-trained Transformer (MiniCPM-4.0) into the hybrid architecture, the model reduces training cost by approximately 75% relative to training from scratch while matching or exceeding the general capabilities of comparable full-attention models like Qwen3-8B. On a single NVIDIA A6000D GPU, MiniCPM-SALA achieves up to 3.5× the inference speed of Qwen3-8B at 256K tokens and supports context lengths of up to 1M tokens—a scale where Qwen3-8B fails due to out-of-memory errors—establishing that hybrid sparse-linear architectures can democratize million-token inference on consumer-grade hardware while retaining competitive standard-benchmark performance.

2. Context and Motivation

The Core Problem: Full-Attention Transformers Hit a Computational Wall at Ultra-Long Contexts

The fundamental challenge this paper addresses is that the standard Transformer architecture—the foundation of virtually all modern LLMs—becomes prohibitively expensive at ultra-long sequence lengths. This is not a minor inefficiency that faster hardware can paper over; it is an architectural bottleneck that creates hard feasibility boundaries, preventing models from processing contexts of millions of tokens on anything but the most resource-rich infrastructure.

The paper identifies two distinct dimensions of this bottleneck (Section 1):

The compute bottleneck: quadratic scaling. In standard scaled dot-product attention, the computational cost grows as O(N²) with sequence length N. Every token must attend to every other token, producing an N × N attention matrix. When N reaches hundreds of thousands or millions of tokens, this becomes computationally intractable even on high-end GPUs. The paper frames this concretely: for a typical 8B-parameter model, the attention computation alone becomes the dominant latency cost, making interactive or batch inference impractical at these scales.

The memory bottleneck: the KV-cache explosion. During autoregressive generation, Transformers cache the key and value states of all previously processed tokens to avoid recomputing attention from scratch at each step. Even with optimizations like Grouped Query Attention (GQA)—which reduces the number of distinct key-value heads by sharing them across query heads—the KV-cache for millions of tokens can reach dozens or hundreds of gigabytes for a typical 8B model. This is not a computation problem; it is a physical memory limit. When the cache exceeds available VRAM, the model cannot run at all, regardless of how long one is willing to wait. The paper emphasizes this failure mode explicitly in its experiments: Qwen3-8B encounters out-of-memory (OOM) errors at 512K tokens on an A6000D (96GB VRAM) and at just 128K tokens on an RTX 5090 (32GB VRAM), while MiniCPM-SALA continues processing to 1M tokens on both.

These two bottlenecks are coupled but distinct. A model might theoretically tolerate high latency on a one-time batch evaluation, but refusing to even load due to insufficient memory is a hard barrier. The paper's practical demonstrations on consumer-grade hardware (RTX 5090) make this distinction vivid: the memory bottleneck, not the compute bottleneck, is what prevents full-attention models from running on edge devices at ultra-long contexts.

Why This Problem Matters: The Shift Toward Holistic, Information-Intensive Applications

The motivation is not purely architectural curiosity. The paper argues that the application landscape for LLMs is undergoing a paradigm shift toward scenarios that fundamentally require ultra-long contexts (Section 1, paragraph 1). The examples given are concrete:

• Whole-document understanding: models that can ingest entire technical manuals, legal documents, or research papers at once, rather than processing fragmented sections. This is necessary for tasks like multi-document summarization, cross-referencing, and comprehensive question-answering over large corpora.

• Repository-scale code engineering: modern software systems involve tens of thousands of files with complex dependency graphs. Models must understand the full project context—not just a single file—to generate correct code, identify bugs across module boundaries, or refactor system-wide invariants. The paper references tools like SWE-bench (Jimenez et al., 2024) and RepoBench (Liu et al., 2024) that evaluate precisely this capability.

• Long-horizon autonomous agents: agents operating over multi-day timescales need to maintain coherent task states, remember past observations and actions, and reason about evolving plans across thousands of interaction turns. The context window becomes the agent's working memory, and truncating it means forgetting.

The common thread is that these applications demand holistic contextual information—the model must see everything at once to maintain coherence. Chunking strategies (e.g., retrieval-augmented generation, sliding windows) are partial workarounds but fundamentally lose the ability to reason across arbitrarily distant pieces of information that both happen to be relevant.

Beyond these specific use cases, there is a broader economic and deployment motivation. The paper explicitly positions itself within the edge computing narrative: making million-token inference feasible on consumer GPUs (RTX 5090, 32GB VRAM) rather than requiring datacenter-scale hardware. This is not just about enabling new applications; it is about democratizing access to them. If only organizations with racks of H100s can run long-context models, the applications listed above remain research demonstrations rather than deployed products.

Prior Approaches: The False Dichotomy of Sparse vs. Linear Attention

The paper identifies two primary paradigms that attempt to mitigate the attention bottleneck, each with a characteristic strength and a characteristic weakness. The framing is that the field has been caught in a trade-off without a satisfactory resolution.

Sparse Attention: "Sparse Computation, Dense Storage"

Sparse attention methods reduce computation by computing only a subset of the full N × N attention matrix. Typical strategies include sliding windows (each token attends to a local neighborhood), global anchor tokens (select tokens attend to the full sequence), or learned sparsity patterns (the model dynamically selects which positions to attend to). The paper specifically incorporates InfLLM-V2 (Zhao et al., 2025), which represents the current state of this line of work.

Strength: Sparse attention preserves the standard softmax attention formulation on the computed subset, meaning the model's representational capacity for the information it does attend to is unchanged from full attention. It can model long-range dependencies faithfully—if the sparse pattern is cleverly designed to capture them.

Weakness: The paper identifies a subtler problem than mere sparsity: "sparse computation, dense storage." Even if a token only computes attention over a small fraction of the context, the model still needs to store the keys and values for all historical tokens in the KV-cache. This is because the relevant subset may differ per query—a token at position 1000 might need to attend to position 5, while a token at position 1001 might need position 800. The KV-cache must retain everything to support arbitrary sparse retrieval. This means sparse attention solves the compute bottleneck but leaves the memory bottleneck largely intact. For ultra-long contexts, the dense KV-cache remains the hard limiting factor.

A second, more implicit weakness is that designing the sparsity pattern itself requires careful engineering. InfLLM-V2 addresses some of this with its dense-sparse switchable mechanism, but the fundamental tension remains: the more tokens you discard from storage, the more you risk missing critical long-range dependencies.

Linear Attention: "O(N) Efficiency Through Lossy Compression"

Linear attention takes a fundamentally different approach: instead of computing the full attention matrix and then sparsifying it, it reformulates the attention computation to avoid materializing the N × N matrix entirely. Methods like Lightning Attention (Qin et al., 2024)—which MiniCPM-SALA uses—achieve this by rewriting the attention as a recurrent computation where a fixed-size state is updated with each token and used to produce the output. This reduces both computation and memory to O(N), with a constant-size state rather than a linearly growing KV-cache.

Strength: The efficiency is genuine and dramatic. The paper's results bear this out: MiniCPM-SALA's linear-dominated architecture achieves 3.5× speedup at 256K tokens and processes 1M tokens on consumer GPUs.

Weakness: The state is a fixed-size compression of all past context. Mathematically, linear attention can be seen as replacing the softmax nonlinearity with a kernelized inner product that factorizes across the sequence, enabling the recurrent formulation. But this factorization loses information compared to exact attention—the fixed-size state cannot perfectly preserve all past token representations. The paper characterizes this as "lossy compression of contextual information" that "inevitably results in performance degradation" (Section 1, paragraph 3). For long-range dependencies that require precise retrieval of specific past tokens (e.g., "what was the value of variable X defined on line 3?"), linear attention's compressed state may blur or lose the relevant detail.

This is the fundamental tradeoff that the paper sets up: sparse attention is precise but memory-hungry; linear attention is memory-efficient but lossy. Prior work largely chose one or the other, accepting the corresponding limitation.

Additional Paradigms: Hybrids and Distillation (In Their Infancy)

The paper acknowledges that "several works have begun exploring the integration of sparse and linear attention" (Section 1, paragraph 4), citing Hu et al. (2025), Hou et al. (2025), and He & Garner (2025). However, it argues these are preliminary explorations that "to the best of our knowledge" have not demonstrated through large-scale experimentation that hybrids can match the performance of full-attention baselines. The paper positions MiniCPM-SALA as the first demonstration at scale that the hybrid approach can close the performance gap while retaining the efficiency benefits.

The paper also discusses the training paradigm distinction. Some prior work on hybrids trains from scratch (Zuo et al., 2025; Qwen Team, 2025; Kimi Team et al., 2025; NVIDIA et al., 2025b), which maximizes architectural flexibility but incurs the full cost of training a large model. Others use cross-architecture distillation (Wang et al., 2024a; Hoshino et al., 2025; Li et al., 2025; Gu et al., 2025), where a pre-trained full-attention Teacher model trains a hybrid Student. Distillation reduces training cost but typically involves training an auxiliary model and may not fully transfer the Teacher's capabilities.

MiniCPM-SALA takes a third path: continual training with direct weight inheritance and architecture conversion, which avoids both the cost of training from scratch and the complexity of cross-architecture distillation.

Specific Gaps That Prior Work Leaves Unaddressed

Reading between the lines of the paper's introduction and related work discussion, several specific gaps emerge that MiniCPM-SALA explicitly targets:

1. No systematic layer-allocation strategy for sparse-linear hybrids. The paper does not simply intermix sparse and linear layers uniformly. It uses a "layer selection algorithm" (from Chen et al., 2026) to determine which layers become sparse vs. linear, and reports that this non-uniform placement produces superior downstream performance compared to naive uniform interleaving. Prior hybrid work had not systematically studied this placement problem.

2. Positional encoding conflicts across attention types. Sparse attention layers typically benefit from RoPE for local positional awareness, but RoPE's distance-based decay can harm long-range retrieval—exactly the capability sparse layers are meant to provide. The paper's Hybrid Positional Encoding (HyPE) applies RoPE only to linear layers and removes it (NoPE) from sparse layers, resolving this tension. Prior work had not addressed this attention-type-specific positional encoding strategy.

3. The Transformer-to-hybrid conversion problem is underspecified. Converting a pre-trained full-attention model to a hybrid architecture without catastrophic forgetting of acquired capabilities is nontrivial. The paper's multi-stage training pipeline (Table 1)—architecture conversion via HALO, continual stable-training at short context, short-decay training with high-quality data, long-decay training with progressive context extension, and supervised fine-tuning—represents a specific recipe for this conversion that had not been documented at this scale.

4. Lack of evidence that hybrid models match full-attention models on standard benchmarks. The paper explicitly claims that "to the best of our knowledge, MiniCPM-SALA is the first to demonstrate through large-scale experimentation that these hybrids can match the performance of full-attention baselines." This is a strong claim that frames prior work as having demonstrated feasibility but not competitive performance.

5. No demonstration of million-token inference on consumer hardware. The memory bottleneck analysis in the paper's experiments (Figures 2 and 3) makes a concrete practical claim: full-attention 8B models do not merely run slowly at 1M tokens on consumer GPUs—they fail entirely. Demonstrating that a 9B hybrid model can function at this scale on an RTX 5090 (32GB) is a direct response to an unaddressed deployment gap.

How the Paper Positions Itself

MiniCPM-SALA positions itself not as a new attention mechanism (both InfLLM-V2 and Lightning Attention are adopted from prior work) but as an integration and scaling effort that addresses the engineering and training challenges necessary to make the hybrid approach work at a competitive level. The paper's contribution framing in Section 1 reflects this:

• The hybrid attention mechanism with a 1:3 ratio is presented as striking "a balance between throughput and precision"—a design choice informed by prior architectural work (Qwen3-Next, Kimi-Linear) and validated at scale.

• The Transformer-to-hybrid training paradigm is positioned as a cost-saving alternative to training from scratch, reducing the training budget to approximately 25% of the de novo cost. This is an engineering contribution with direct practical implications for organizations that have already invested in pre-training large Transformers.

• The HyPE and architectural modifications (QK-Normalization, output gates) are presented as solving specific sub-problems (positional encoding conflicts, training stability, attention sink) that arise when combining heterogeneous attention types.

• The empirical demonstrations—matching standard benchmark performance, achieving 3.5× speedup at 256K, and supporting 1M-token inference on consumer GPUs—are the evidence that this integration actually works at a level that matters for deployment.

The paper explicitly connects to the HALO framework (Chen et al., 2026) and InfLLM-V2 (Zhao et al., 2025) for architectural details, making clear that its novelty lies in the orchestration and scaling of these components rather than in inventing new attention primitives. This is intellectually honest positioning: the paper does not claim to have solved everything from scratch, but rather demonstrates that careful system-building—choosing the right ratio, the right placement, the right positional encoding strategy, and the right training recipe—can produce a practically useful model that prior, more preliminary efforts had not achieved.

3. Technical Approach

3.1 Reader Orientation

MiniCPM-SALA is a 9-billion-parameter language model whose internal architecture replaces most of the standard attention layers with a mixture of two cheaper alternatives—sparse attention and linear attention—and whose weights are obtained by converting an already-trained Transformer rather than training from scratch. The system solves the problem that full-attention Transformers become impossible to run on very long inputs (millions of tokens) because they consume too much computation and memory; the solution is a hybrid layer stack where 75% of layers use O(N) linear attention (fast and memory-light but lossy) and 25% use sparse attention (slower but precise for long-range dependencies), trained via a multi-stage pipeline that inherits a pre-trained model's knowledge while teaching the new attention layers to work together.

3.2 Big-Picture Architecture (Diagram in Words)

The model has five major architectural components, arranged as a stack of Transformer-style layers:

1. Input embedding and normalization — tokens are embedded as usual; the first normalization is applied at the input to each layer.

2. A hybrid sequence of 75% linear attention layers and 25% sparse attention layers — these are interleaved in a non-uniform pattern determined by a layer selection algorithm. Linear attention layers use Lightning Attention (recurrent O(N) formulation with a fixed-size state); sparse attention layers use InfLLM-V2 (selective attention over a subset of the full context, but with the standard softmax formulation).

3. Feed-Forward Network (FFN) blocks — one standard FFN follows each attention block, preserved from the Transformer architecture to maintain representational capacity.

4. Architectural enhancements applied uniformly — QK-Normalization on all attention layers, output gates after every attention block, and HyPE (Hybrid Positional Encoding) that applies RoPE only to linear layers while removing it from sparse layers.

5. A training pipeline (not part of the model architecture per se but essential to how the model comes to exist) — architecture conversion via HALO, continual stable-training, short-decay training, long-decay training with progressive context extension, and supervised fine-tuning. This pipeline starts from a MiniCPM-4.0 checkpoint and produces the final MiniCPM-SALA weights.

Information flows through the model layer by layer: token embeddings enter the first layer, pass through either a linear attention sublayer or a sparse attention sublayer (with QK-Normalization and output gating), then through the FFN, then to the next layer. The type of attention at each layer position is fixed at training time.

3.3 Roadmap for the Deep Dive

• First, the hybrid layer allocation strategy — the 1:3 ratio, the layer selection algorithm, and why 75% linear is the sweet spot — because the layer composition defines the architecture's efficiency-performance tradeoff.
• Second, the sparse attention mechanism (InfLLM-V2) — what it computes, how it reduces computation, and critically, why it still requires dense KV-cache storage despite sparse computation — since understanding this limitation motivates the hybrid strategy.
• Third, the linear attention mechanism (Lightning Attention) — how the recurrent formulation achieves O(N) complexity, what information is compressed into the fixed-size state, and why this compression causes precision loss — because this is the efficiency engine that makes 1M-token inference possible.
• Fourth, the architectural enhancements — QK-Normalization, output gates, and HyPE — explaining what each solves and why they are necessary specifically in a hybrid setting where sparse and linear layers coexist.
• Fifth, the training pipeline in detail — the HALO conversion, the five training stages, the data volumes, sequence lengths, and learning rates at each stage, and the rationale for the progressive context extension — because the training recipe is what makes the conversion work without catastrophic forgetting.
• Sixth, the training-data composition and the reasoning behind it — since data quality and mixture changes across stages are key to the model's ability to retain general capabilities while acquiring long-context skills.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and engineering paper whose core idea is that a hybrid sparse-linear attention architecture, obtained by converting a pre-trained full-attention Transformer through a carefully staged continual-training pipeline, can match the general capabilities of full-attention models while achieving dramatic efficiency gains on ultra-long contexts and enabling inference at scales where full-attention models fail due to memory constraints.

---

Hybrid Layer Allocation: The 1:3 Ratio and Layer Selection

The defining structural decision in MiniCPM-SALA is which layers use which attention type. The paper does not naively alternate sparse and linear layers in a fixed pattern. Instead, it makes two deliberate choices: a global ratio and a per-position selection algorithm.

The 1:3 mixing ratio. The paper states that 25% of layers use sparse attention (InfLLM-V2) and 75% use linear attention (Lightning Attention). This ratio is "inspired by the architectural designs of recent representative studies, such as Qwen3-Next (Qwen Team, 2025) and Kimi-Linear (Kimi Team et al., 2025), as well as our internal small-scale preliminary experiments" (Section 2.1). The reasoning is a balance: linear attention provides the bulk of the efficiency—its O(N) complexity and constant memory make long-context processing viable—while sparse attention layers are sprinkled in to "facilitate effective modeling of long-range dependencies" that the lossy linear compression would otherwise miss. Too few sparse layers, and the model loses precision; too many, and the memory benefits evaporate because sparse attention still requires dense KV-cache storage.

The layer selection algorithm. The paper explicitly states that it does "not naively uniformly interleav[e] the two attention variants." Instead, it uses the "layer selection mechanism proposed by Chen et al. (2026)" to determine which specific layer positions become sparse attention layers. The HALO framework (Chen et al., 2026) provides this selection algorithm. The paper provides partial detail: the first and last layers are forced to remain unconverted (i.e., they stay as full-attention layers during the initial conversion, presumably to preserve the input-processing and output-projection capabilities that the Transformer's first and last layers are known to specialize in). For the remaining layers, the HALO selection algorithm determines which are converted to linear and which are preserved as softmax attention (to later become sparse attention).

The paper does not disclose the full selection criteria (referring the reader to Chen et al., 2026), but the operational consequence is clear: the sparse attention layers are placed at positions that the algorithm determines are most important for long-range dependency modeling, rather than being evenly distributed. This is a critical design choice because in a deep Transformer, different layers learn different linguistic abstractions—early layers tend to handle local syntax, middle layers semantic composition, and late layers task-specific integration. Placing sparse attention at layers where long-range information is most needed (presumably middle-to-late layers) rather than arbitrarily maximizes the benefit of the 25% allocation.

---

Sparse Attention: InfLLM-V2

For the layers assigned to sparse attention, MiniCPM-SALA uses InfLLM-V2 (Zhao et al., 2025), a mechanism designed specifically for long-context processing.

What InfLLM-V2 computes. InfLLM-V2 restricts the standard softmax attention computation to a subset of the full context rather than attending over all N positions. The key-value pairs that are actually accessed are determined by a selection mechanism: the model chooses which historical tokens are most relevant to the current query and computes attention only over those. The mechanism is described as "dense-sparse switchable" in the cited paper's title, meaning it can dynamically choose between full attention (when the context is short enough) and sparse attention (when the context is long). This switchability is "highly compatible with our conversion process" (Section 2.1) because during the early training stages when sequence lengths are short (512–4K tokens), the model can operate in dense mode, and the switch to sparse mode happens naturally when sequence length increases in long-decay training.

Why InfLLM-V2 introduces no additional parameters. The paper highlights that InfLLM-V2 "offers the distinct advantage of introducing no additional parameters to the architecture" (Section 2.1). This is crucial for the Transformer-to-hybrid conversion paradigm: because the sparse attention mechanism reuses the existing key, query, and value projection weights from the pre-trained Transformer, there is no architectural mismatch during weight inheritance. The layer can start from the full-attention weights and gradually learn to operate under sparsity constraints, rather than needing newly initialized parameters that would not benefit from pre-training.

The "sparse computation, dense storage" limitation. This is the critical weakness that the hybrid architecture is designed to compensate for. InfLLM-V2 reduces computational cost by only computing attention over a subset of positions, but the model still stores the full KV-cache for all historical tokens. The keys and values for every past token must be retained in memory because the subset of tokens selected for attention may differ at each generation step and for each query. The paper's term "sparse computation, dense storage" captures this precisely: the computation is sparse, but the memory footprint is unchanged from full attention.

This means that sparse attention layers do not solve the memory bottleneck—they only solve the compute bottleneck. A model using only sparse attention could run faster than a full-attention model (because it computes fewer attention dot-products), but it would still run out of memory at the same sequence length because the KV-cache size is identical. This is why MiniCPM-SALA uses only 25% sparse attention layers: the 75% linear attention layers are the ones that actually reduce memory consumption, while the sparse layers provide precision on a minority of the total layers where the memory cost is manageable.

Training-mode behavior. During the continual stable-training and short-decay training stages (Table 1), sparse attention is "disabled"—meaning these layers operate in standard full-attention mode because the sequence lengths (512–4K) are short enough that full attention is both feasible and more precise. Sparse attention is "enabled" only in the long-decay training stage, when sequence lengths extend to 32K and beyond. This staged approach allows the sparse attention layers to first stabilize their parameters in dense mode (inheriting and refining the pre-trained weights) before learning to operate under the sparsity constraint at long contexts.

---

Linear Attention: Lightning Attention

For the remaining 75% of layers, MiniCPM-SALA uses Lightning Attention (Qin et al., 2024), a linear-attention variant that achieves O(N) complexity in both computation and memory.

The core mathematical idea. Standard softmax attention computes, for each query position, a weighted sum over all key-value pairs:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$

This requires materializing the N × N attention matrix, which is the source of the quadratic cost. Linear attention avoids this by replacing the softmax nonlinearity with a kernel function that factorizes across the sequence. Specifically, if the attention weights can be written as:

$\text{Attention}(Q_i, K, V) = \frac{\sum_{j=1}^{N} \phi(Q_i)^T \phi(K_j) V_j}{\sum_{j=1}^{N} \phi(Q_i)^T \phi(K_j)}$

where $\phi(\cdot)$ is a feature map (a nonlinear transformation applied element-wise to the query and key vectors), then the numerator and denominator can be computed recurrently without storing the full N × N matrix. At each position, the model updates a fixed-size state $S_t$ (the accumulated outer product of key features and value vectors) and a normalization term $z_t$ (the accumulated key feature vector for the denominator), producing the output for the current position from these running aggregates.

Operational behavior. In Lightning Attention specifically, the mechanism is implemented as a hardware-efficient kernel that computes attention in blocks, maintaining the recurrent state across sequence positions. During training, the computation is parallelized within blocks, while during inference, the state is updated token-by-token. The critical property is that the memory required does not grow with sequence length—the state $S_t$ has a fixed size determined by the key/value dimensionality and the feature map dimension, independent of how many tokens have been processed. This is what enables 1M-token inference on consumer GPUs: the linear attention layers consume constant memory regardless of context length.

Why Lightning Attention was chosen over other linear attention variants. The paper gives a specific rationale: "Given our Transformer-to-hybrid conversion paradigm, Lightning Attention is selected for its functional proximity to the standard softmax attention. This structural alignment is intended to mitigate the complexities of parameter adaptation, thereby preserving pre-trained knowledge and ensuring robust downstream performance" (Section 2.1). In plain language: Lightning Attention's formulation is close enough to standard attention that the pre-trained weights (which were learned under softmax attention) can be adapted with less disruption than a more radically different architecture would require. The paper also notes that Lightning Attention "provides better length generalization capabilities according to Chen et al. (2026), which may improve data efficiency during long-context continual-training"—meaning the model can extrapolate to sequence lengths longer than those seen during training more effectively.

The lossy compression tradeoff. The fixed-size state $S_t$ is a compressed representation of all past context. Unlike the dense KV-cache in standard attention (which stores each token's key and value vector exactly), the linear attention state must summarize everything into a single matrix of fixed dimensions. This is mathematically a lossy operation: information from earlier tokens can be overwritten or blurred as new tokens are processed. The paper characterizes this as a "precision bottleneck" and "lossy compression of contextual information" that "inevitably results in performance degradation." This is why the 25% sparse attention layers exist: they provide precise, uncompressed access to historical tokens that the linear layers might have blurred, specifically for long-range dependencies that require exact retrieval.

Layer conversion under HALO. In the initial architecture conversion stage, the HALO framework converts the selected linear attention layers by modifying their attention computation from standard softmax attention to the Lightning Attention formulation. This involves adjusting the internal computation while preserving as much of the pre-trained weight structure as possible. The converted layers' parameters are trainable during the architecture conversion stage (while all other parameters are frozen), allowing them to adapt to the new computation before the model proceeds to full-parameter training in subsequent stages.

---

Architectural Enhancements: QK-Normalization, Output Gates, and HyPE

Three architectural modifications are applied uniformly across all attention layers (both sparse and linear) to address specific training and performance challenges that arise in the hybrid setting.

QK-Normalization. This technique, introduced by Henry et al. (2020), applies LayerNorm (or a similar normalization) to the query and key vectors before they enter the attention computation. The paper states two motivations: (1) "to prevent the activation spikes that often occur in long-context training"—when processing very long sequences, attention logits can grow large due to accumulating dot products, causing numerical instability during training; normalizing queries and keys keeps the dot-product magnitudes controlled regardless of sequence length—and (2) "to further improve and boost the expressivity of linear attention modules"—linear attention in particular benefits from normalized inputs because its recurrent state updates are sensitive to the magnitude of incoming key and value vectors; normalization prevents any single token from dominating the state update.

Operationally, QK-Normalization adds a small computational overhead per attention layer but does not change the asymptotic complexity. It is applied to both sparse and linear layers for uniformity.

Output gates. Following recent work on gated attention mechanisms (Qiu et al., 2025), MiniCPM-SALA incorporates an output gate after each attention block—both sparse and linear. The output gate is a learned scalar (or vector, per dimension) that multiplies the attention output before it is added to the residual stream. The paper's rationale: "the output gate has been shown to effectively mitigate issues such as attention sink. By regulating the information flow, the output gate prevents excessive focus on specific tokens and ensures a more flexible distribution of attention weights."

The attention sink problem, well-documented in the literature, refers to the phenomenon where the first few tokens of a sequence (especially the initial token) receive disproportionately high attention weights regardless of their relevance, effectively "sinking" attention mass into uninformative positions. In long-context processing, this wastes attention budget on tokens that are not useful for the current computation. The output gate can learn to suppress the contribution from attention outputs that are dominated by sink tokens, effectively gating out poor attention distributions.

The paper reports empirically that "integrating output gates into both linear and sparse attention significantly improves model stability and performance," though no ablation study is presented for this specific component.

HyPE (Hybrid Positional Encoding). This is perhaps the most subtle and architecturally significant enhancement. The problem HyPE solves is that sparse attention and linear attention have conflicting requirements for positional encoding.

Standard Transformers use Rotary Positional Embedding (RoPE) (Su et al., 2023), which encodes relative position by rotating query and key vectors before computing attention. RoPE has a distance-based decay property: tokens that are far apart in the sequence have attention weights that naturally decay, which is desirable for local coherence but harmful for long-range retrieval—the model may struggle to attend strongly to a token 100K positions away because RoPE's rotation effectively reduces the dot-product magnitude at large distances.

For linear attention layers, this decay property is actually beneficial: the recurrent state naturally emphasizes recent tokens (since they are added last to the running aggregate), and RoPE's distance-sensitivity helps the model preserve relative order within the fixed-size state.

For sparse attention layers, however, RoPE's decay property is counterproductive. The whole point of having sparse attention layers is to enable precise retrieval of any past token regardless of distance. If RoPE penalizes long-distance attention, the sparse layers cannot fulfill their role. The paper states this explicitly: "we remove RoPE in the sparse attention layers. This strategic omission prevents the decay of long-distance information often associated with RoPE, thereby enabling more precise recall over extended contexts."

The HyPE strategy, adopted from Chen et al. (2026), therefore applies RoPE only to linear attention layers (where position-sensitive memory aids the recurrent state) and uses NoPE (no positional encoding) for sparse attention layers (where absolute position is less important than content-based retrieval and distance-based decay must be avoided). The paper attributes MiniCPM-SALA's impressive length extrapolation capabilities (maintaining 81.6 RULER score at 2048K tokens despite being trained only up to 520K) partly to this design: "The length extrapolation capabilities of MiniCPM-SALA can be attributed to the NoPE configuration within the sparse attention layers. In this design, the stored KV-Cache does not require combination with positional information, which can otherwise hinder the capture of long-range dependencies" (Section 3.1, end of ultra-long context discussion).

This is a crucial design insight: in a hybrid architecture, different attention types should not necessarily use the same positional encoding. The choice of positional encoding should align with the specific role each attention type plays in the overall system.

---

Training Pipeline: Five-Stage Conversion from Transformer to Hybrid

The training of MiniCPM-SALA is a multi-stage process that starts from an intermediate checkpoint of MiniCPM-4.0 (which has already been trained on 7 trillion tokens) and transforms it into the final hybrid model. Table 1 in the paper provides the complete specification. The total training consumes approximately 2 trillion tokens, which the paper claims is "roughly 25% of the data volume required to train MiniCPM-4.0 from scratch (8T tokens)."

The five stages, in order, with their precise configurations:

---

Stage 1: Architecture Conversion (HALO)

Purpose: Transform selected full-attention layers into linear attention layers while keeping other components frozen.

Configuration (from Table 1):

• Trainable parameters: Linear attention layers only (all other parameters frozen).
• Sparse attention: Disabled (all layers operating in dense/full-attention mode).
• Sequence length: 0.5K tokens (512 tokens).
• Total tokens processed: 1.3 billion.

What happens mechanically. The HALO framework (Chen et al., 2026) is applied with two modifications from the standard procedure. First, the first and last layers of the Transformer are kept unconverted (remaining as full softmax attention) to improve training stability—these layers handle input embedding integration and final output projection, and preserving their full-attention capability prevents disruption to the model's fundamental input-output mapping. Second, the standard HALO process includes a final fine-tuning step that the authors skip; instead, they substitute the more extensive continual pre-training and post-training stages that follow.

The layer selection algorithm within HALO determines which of the remaining layers (all except first and last) become linear attention and which are preserved as softmax attention (to later become sparse attention in stage 4). The selected layers' attention computation is modified from standard softmax attention to Lightning Attention, and only these converted layers' parameters are trained at this stage.

The choice of 512-token sequences at this stage is deliberate: at short context lengths, full attention is cheap enough that the model can focus on learning the linear attention computation pattern without being overwhelmed by long-sequence training dynamics. The tiny token budget (1.3B) reflects that this is essentially an initialization stage—the model is learning the basic mechanics of the new attention formulation rather than acquiring new knowledge.

Why freeze most parameters. By training only the linear attention layers, the conversion is surgical: the vast majority of the model's knowledge (stored in the FFN weights, embedding layers, and preserved attention layers) remains exactly as it was in the pre-trained checkpoint. Only the layers whose computation has changed need to adapt. This minimizes the risk of catastrophic forgetting during the architectural transition.

---

Stage 2: Continual Stable-Training

Purpose: Enable the converted linear attention layers to coordinate with the rest of the model (preserved attention layers, FFN layers, embeddings) that were frozen in stage 1.

Configuration (from Table 1):

• Trainable parameters: All parameters.
• Sparse attention: Disabled (attention layers operate in dense mode).
• Sequence length: 4K tokens.
• Total tokens processed: 314.6 billion.
• Learning rate: $7.5 \times 10^{-3}$, held constant after a 2,000-step warmup period.
• Global batch size: 7.8 million tokens (adjusted for sequence length and GPU count).

What happens mechanically. The checkpoint from stage 1 is now trained with all parameters unfrozen, on the full MiniCPM-4.0 pre-training dataset, at 4K sequence length. Sparse attention remains disabled because 4K tokens is short enough that full attention is computationally feasible—there is no benefit to sparsity yet, and full attention provides more precise gradients for training.

The key shift from stage 1 is that the FFN layers, embeddings, and preserved attention layers are now also updating. This allows them to adapt to the presence of linear attention layers in the stack. For example, an FFN layer that previously received its input from a standard attention output now receives input from a linear attention output with potentially different statistical properties (since linear attention's compressed state introduces a different noise/distortion profile). The FFN weights can adjust to accommodate these differences.

The learning rate is relatively high ($7.5 \times 10^{-3}$, constant after warmup), indicating that this stage is about substantial re-coordination rather than fine-tuning. The data volume (314.6B tokens) is substantial—about 16% of the total 2T training budget—reflecting the importance of this integration phase.

The "stable" in stable-training. The paper names this stage "continual stable-training," suggesting that the goal is to reach a stable equilibrium where the heterogeneous layer types work together smoothly, before moving on to the more aggressive training schedules in subsequent stages.

---

Stage 3: Short-Decay Training

Purpose: Compress and internalize large amounts of knowledge using high-quality, high-information-density data while the context length is still short.

Configuration (from Table 1):

• Trainable parameters: All parameters.
• Sparse attention: Disabled.
• Sequence length: 4K tokens.
• Total tokens processed: 1,006.6 billion (approximately 1 trillion).
• Learning rate: Exponential decay from $7.5 \times 10^{-3}$ to $3.75 \times 10^{-4}$.
• Global batch size: 7.8 million tokens.

What happens mechanically. This is the most data-intensive stage in the pipeline, consuming roughly half of the total 2T training budget. The sequence length remains at 4K, so sparse attention stays disabled. The learning rate decays exponentially rather than remaining constant, which is a standard technique for transitioning the model from rapid adaptation (high LR) to careful refinement (low LR) as it approaches convergence.

Data strategy shift. The paper describes a significant change in data composition at this stage: "Building on the MiniCPM-4.0 decay strategy, we significantly increase the weight of L2 high-quality selection data (Wang et al., 2026) and introduce a large volume of PDF corpora and L3 synthetic data. This approach aims to enhance general capabilities and logical reasoning using high-information-density training data, achieving the efficient compression and internalization of massive amounts of knowledge."

Operationally, this means the training data in stage 3 is not a uniform sample of internet text. Instead, it is deliberately curated:

• L2 high-quality selection data: Pre-filtered, high-quality text from the MiniCPM data pipeline (Wang et al., 2026). This likely includes textbooks, academic papers, technical documentation, and carefully curated web content.
• PDF corpora: Structured and semi-structured documents in PDF format, which the model needs to handle for long-context document understanding tasks.
• L3 synthetic data: Data generated by other language models (likely larger or more capable models) that provides high-quality reasoning examples. Synthetic data is commonly used to teach models reasoning patterns that are underrepresented in natural text.

The phrase "efficient compression and internalization of massive amounts of knowledge" is key: the model is not just being exposed to more text—it is being exposed to dense text where each token carries more information (concepts, reasoning steps, factual relationships) than an average web token. At 4K sequence length, the model can process this efficiently, and the decaying learning rate helps it solidify the acquired knowledge.

Why keep sequence length at 4K for a trillion tokens. This might seem counterintuitive for a model whose purpose is long-context processing. The rationale is that long-context capabilities are largely about mechanism (the attention architecture and its training to handle long-range dependencies), while general capabilities (knowledge, reasoning, code, math) are largely about data quality and volume. Stage 3 uses the bulk of the training budget to maximize general capabilities using high-quality data at a sequence length where training is fastest (since full attention at 4K is cheap). The long-context mechanism training is deferred to stage 4, when the model's general capabilities are already strong.

---

Stage 4: Long-Decay Training

Purpose: Teach the model to handle long sequences by progressively extending the context length and enabling sparse attention.

Configuration (from Table 1):

• Trainable parameters: All parameters.
• Sparse attention: Enabled (this is the key change from stages 2–3).
• Sequence length and data volume: Three sub-stages:
  • 32K context: 102.2 billion tokens
  • 160K context: 62.9 billion tokens
  • 520K context: 50.6 billion tokens
• Learning rate decay across sub-stages: $3 \times 10^{-4} \to 2 \times 10^{-4}$ (32K), then $\to 1 \times 10^{-4}$ (160K), then $\to 3.75 \times 10^{-5}$ (520K).
• Global batch size adjustments: 7.8M tokens (32K), 9.8M tokens (160K), 10.1M tokens (520K).

What happens mechanically. This is the stage where MiniCPM-SALA becomes a long-context model. Three things happen simultaneously:

1. Context length increases progressively: 4K → 32K → 160K → 520K. The model is trained on progressively longer sequences, learning to handle the extended attention ranges. This progressive extension is a standard technique in long-context training (also used by models like Llama 3 and Qwen3) because jumping directly from 4K to 520K would be a severe distribution shift that could destabilize training.

2. Sparse attention is enabled: At each sub-stage, the layers designated as sparse attention layers switch from dense (full-attention) mode to sparse mode. This is when the model learns the synergy between sparse and linear attention—sparse layers handle precise long-range retrieval while linear layers handle efficient global context processing. The paper states: "Given the growing computational advantages of sparse attention at longer sequences, we enable the sparse attention mechanism at this stage and maintain full-parameter training, thereby allowing the model to effectively learn the synergy between sparse attention and linear attention."

3. Long-context data is upsampled: The data mixture shifts to include a higher proportion of naturally long documents and synthetic long-context tasks, "to better align the model with long-sequence distributions." This likely includes concatenation of related documents, long-form QA with context spanning hundreds of thousands of tokens, and retrieval tasks where the target information is deliberately placed at varying positions in long contexts.

Why the data volume decreases as sequence length increases. The total token count drops from 102.2B (at 32K) to 50.6B (at 520K). This is because training on longer sequences is computationally more expensive per token (even with the efficiency of the hybrid architecture, longer sequences mean more attention computation). The decreasing volume reflects a practical tradeoff: expose the model to enough long-context training to learn the mechanism, but not so much that training costs explode. The batch size also increases slightly to compensate for the reduced number of sequences per batch (since each sequence is longer).

Learning rate strategy. The learning rate decays across sub-stages, reflecting that the model is transitioning from adapting to a new context length (initially higher LR) to refining its long-context capabilities (lower LR). The final LR of $3.75 \times 10^{-5}$ is very low, indicating that by the end of this stage the model is close to convergence on the long-context training objective.

---

Stage 5: Supervised Fine-Tuning (SFT)

Purpose: Align the model to follow instructions, handle reasoning-intensive tasks, and perform information retrieval in extended contexts.

Configuration (from Table 1):

• Trainable parameters: All parameters.
• Sparse attention: Enabled (continuing from stage 4).
• Sequence length and data volume: Two sub-stages:
  • 64K context: 204.5 billion tokens
  • 140K context: 213.3 billion tokens
• Learning rate: Warmup to peak $1 \times 10^{-3}$ over 1,000 steps, then decay to $1 \times 10^{-4}$.
• Global batch size: 15.7M tokens (64K), 17.8M tokens (140K).

What happens mechanically. This stage trains the model on supervised instruction-following data rather than the unsupervised next-token prediction of pre-training. The SFT corpus is composed of "high-quality reasoning-intensive data, encompassing code, mathematics, knowledge, function calls, and general dialogue. This selection is designed to fully catalyze the reasoning and task-execution capabilities under complex logic."

Two types of long-context data are specifically synthesized for this stage:

• Long-context information retrieval data: Questions that require finding specific pieces of information scattered across long documents, training the model to use its hybrid attention effectively for precise retrieval.
• Cross-document comprehension data: Tasks that require synthesizing information from multiple documents within a single long context, training the model to maintain coherence across document boundaries.

The two sub-stages (64K then 140K) "bridge shorter and longer contexts, allowing the model to better balance general capabilities with long-context proficiency." The SFT data volume is substantial (over 400B tokens total), indicating that instruction-following and reasoning are not superficial additions but are deeply trained into the model.

The relationship between SFT and pre-training in this pipeline. Most LLM training pipelines have a sharp separation between pre-training (unsupervised, massive data, learning language and knowledge) and fine-tuning (supervised, smaller data, learning instruction-following). MiniCPM-SALA blurs this boundary somewhat: the SFT stage uses 417.8B tokens, which is roughly 20% of the total 2T training budget. This is a very large SFT budget by typical standards, reflecting the paper's emphasis on making the model practically useful for complex reasoning and long-context tasks rather than just demonstrating architectural feasibility.

---

Training Data Composition and Strategy

The paper provides a high-level description of the data strategy across stages, with specific details worth noting:

Data sources referenced:

• MiniCPM-4.0 pre-training dataset: The base data used in stages 2 and 3, inherited from the MiniCPM project. This is a large-scale curated web corpus.
• L2 high-quality selection data (Wang et al., 2026): A tiered data management system where "L2" represents high-quality, curated data above general web-crawl quality. Upweighted in stage 3.
• PDF corpora: Documents in PDF format, introduced in stage 3 to prepare the model for document understanding tasks.
• L3 synthetic data: Machine-generated data, likely from larger teacher models, used in stage 3 for reasoning enhancement.
• Synthesized long-context SFT data: Task-specific data created for stage 5, covering information retrieval and cross-document comprehension.

The data strategy philosophy. The paper's data strategy reflects a staged approach: early stages use broad, high-quality data to build general capabilities; middle stages shift to dense, curated data to compress knowledge efficiently; later stages introduce long-context data to teach the mechanism; and the final stage fine-tunes on task-specific reasoning and long-context data. This staged approach ensures that the model learns long-context processing on top of a solid foundation of general language understanding, rather than trying to learn everything simultaneously.

---

Design Choices and Their Justifications: A Summary

The paper makes a series of interconnected design choices, each with explicit or implicit justification:

• 1:3 sparse-to-linear ratio over 50:50 or pure linear: inspired by prior work (Qwen3-Next, Kimi-Linear) and validated by internal experiments; balances precision (sparse) and memory efficiency (linear).
• Non-uniform layer placement over uniform interleaving: uses HALO layer selection algorithm because different layers have different roles, and sparse attention is most beneficial at layers responsible for long-range dependency modeling.
• InfLLM-V2 for sparse attention over other sparse mechanisms: introduces no new parameters (critical for weight inheritance in continual training) and supports dense-sparse switching (compatible with staged training where sparsity is enabled mid-pipeline).
• Lightning Attention for linear attention over Mamba/RWKV: functionally closer to softmax attention (easier parameter adaptation from pre-trained weights) and has better length generalization (important for extrapolating to 1M+ tokens from 520K training length).
• HALO conversion with frozen non-linear layers over full-parameter conversion: surgical approach minimizes catastrophic forgetting by preserving FFN and embedding weights during the architectural transition.
• Progressive context extension (512 → 4K → 32K → 160K → 520K) over direct long-context training: standard technique that avoids destabilizing the model with extreme distribution shifts; allows short-context general capabilities to solidify before long-context mechanism training begins.
• Disabling sparse attention until long-decay stage over enabling it earlier: at short context lengths (≤4K), full attention is both feasible and provides better gradient signals; sparsity only becomes beneficial (and necessary) at 32K+.
• NoPE for sparse layers, RoPE for linear layers over uniform RoPE or uniform NoPE: addresses the conflicting positional encoding requirements—sparse layers need distance-agnostic retrieval, linear layers benefit from position-sensitive recurrent state.
• QK-Normalization and output gates as universal additions: prevent activation spikes and attention sink (documented failure modes) across both attention types, improving training stability.
• Large SFT budget (417.8B tokens) over minimal SFT: the model is intended for practical deployment on complex reasoning and long-context tasks; superficial instruction tuning would not suffice for the claimed capabilities.

These choices collectively define MiniCPM-SALA not as a novel attention mechanism but as a carefully engineered system where each component (attention type, positional encoding, normalization, training stage, data mixture) is chosen to solve a specific sub-problem that arises at the intersection of heterogeneous attention types and continual training from a pre-trained Transformer.

4. Key Insights and Innovations

Innovation 1: The Hybrid Architecture Is Not Just a Compromise — It Is a Principled Division of Labor Between Precision and Efficiency

The dominant assumption in the long-context efficiency literature has been that sparse attention and linear attention are competing alternatives — that you pick one paradigm and accept its characteristic weakness. Sparse attention gives you faithful long-range modeling but retains the dense KV-cache memory bottleneck; linear attention gives you true O(N) memory scaling but loses precision through state compression. The field has largely treated this as a tradeoff to be optimized within a single paradigm (e.g., designing better sparsity patterns to reduce the number of tokens stored, or designing better state-update rules to improve linear attention fidelity).

MiniCPM-SALA reframes this from a choice to a division of labor. The core conceptual move is recognizing that sparse and linear attention are not substitutes — they solve different sub-problems that both arise in long-context processing. Linear attention is the efficiency engine: its constant-memory state makes 1M-token inference possible on consumer GPUs where full-attention models OOM. Sparse attention is the precision patch: distributed sparingly (25% of layers) at positions selected by an algorithm, it compensates for the lossy compression of the linear layers specifically where long-range dependencies are most critical. Each mechanism is deployed where its strengths are needed and its weaknesses are tolerable.

This is not an incremental refinement of either paradigm. It is a fundamental reframing that converts a perceived dichotomy (sparse OR linear) into an architectural design principle (sparse AND linear, in a specific ratio and placement). The paper's evidence that this hybrid matches full-attention models on standard benchmarks (Table 2: 76.53 average vs. Qwen3-8B's 73.45) while achieving 3.5× speedup at 256K tokens and enabling 1M-token inference where full-attention models fail (Figures 2–3) validates that the division-of-labor framing is not merely conceptually elegant — it works at scale.

The prior work that the paper cites as "beginning to explore" hybrid sparse-linear integration (Hu et al., 2025; Hou et al., 2025; He & Garner, 2025) had not demonstrated competitive performance against full-attention baselines. MiniCPM-SALA's claim to be "the first to demonstrate through large-scale experimentation that these hybrids can match the performance of full-attention baselines" (Section 1) marks a transition from feasibility to competitiveness, enabled by the principled layer-selection and ratio-choice rather than naive uniform interleaving.

---

Innovation 2: Attention-Type-Specific Positional Encoding Is Necessary for Hybrid Architectures to Work

The standard practice in Transformer training is to apply the same positional encoding scheme (invariably RoPE in modern models) uniformly across all layers. The assumption is that all attention layers benefit from the same relative-position signal. MiniCPM-SALA challenges this assumption directly through its HyPE (Hybrid Positional Encoding) strategy: RoPE is applied only to linear attention layers, while sparse attention layers use NoPE (no positional encoding).

The conceptual insight here is that different attention mechanisms have different relationships to position, and forcing a uniform positional encoding creates a conflict that degrades the hybrid's effectiveness. RoPE's distance-based decay — where attention weights naturally diminish for tokens that are far apart — is beneficial for linear attention layers because their recurrent state already emphasizes recency; RoPE's position sensitivity helps the fixed-size state preserve the temporal structure of the context. But for sparse attention layers, RoPE's distance penalty is counterproductive: the entire purpose of having sparse attention in the architecture is to enable precise retrieval of any past token regardless of how far back it appears. If RoPE penalizes long-distance attention, the sparse layers cannot fulfill their designated role in the division of labor.

This insight is not just a practical trick — it is a diagnostic contribution that explains a potential failure mode in hybrid architectures. If one were to apply RoPE uniformly to a sparse-linear hybrid (as one might unthinkingly do following standard Transformer practice), the sparse attention layers would be handicapped at long range, undermining the very capability they were included to provide. The paper's attribution of MiniCPM-SALA's length extrapolation capability — maintaining an 81.6 RULER score at 2048K tokens despite being trained only to 520K (Table 4) — to the NoPE configuration in sparse layers supports this diagnosis: "the stored KV-Cache does not require combination with positional information, which can otherwise hinder the capture of long-range dependencies."

Compared to prior work, this is a fundamental shift rather than an incremental improvement. Prior sparse attention methods (including InfLLM and InfLLM-V2) and linear attention methods (including Lightning Attention) had each been studied with their own positional encoding choices, but the interaction between attention type and positional encoding in a hybrid stack had not been systematically identified as a design dimension. HyPE introduces a new principle: positional encoding should be attention-type-conditional, chosen to align with the specific role each attention mechanism plays in the overall architecture.

---

Innovation 3: The Transformer-to-Hybrid Conversion Paradigm Demonstrates That Architectural Transitions Can Be More Compute-Efficient Than Training From Scratch — With Competitive Results

The conventional wisdom in training large language models with non-standard architectures has been that architectural innovation requires training from scratch. Models like Falcon-H1 (Zuo et al., 2025), Qwen3-Next (Qwen Team, 2025), Kimi-Linear (Kimi Team et al., 2025), and Nemotron 3 Nano (NVIDIA et al., 2025b) all train their hybrid architectures de novo. The alternative — cross-architecture distillation, where a full-attention teacher trains a hybrid student (Wang et al., 2024a; Hoshino et al., 2025; Gu et al., 2025) — reduces training cost but introduces the complexity of training an auxiliary model and typically does not fully close the performance gap to the teacher.

MiniCPM-SALA introduces a third paradigm: direct architectural conversion via continual training with weight inheritance. Instead of training a hybrid model from random initialization or distilling from a separate teacher, the method takes a pre-trained full-attention Transformer (MiniCPM-4.0) and surgically converts selected layers to linear attention using the HALO framework, then trains the entire model through a staged pipeline to adapt to the new architecture while preserving acquired knowledge.

The conceptual contribution is not the HALO conversion technique itself — that is from Chen et al. (2026) — but rather the demonstration at scale that this paradigm can produce a competitive model. The paper claims approximately 75% training cost reduction relative to training a comparable model from scratch (2T tokens for conversion vs. 8T tokens for MiniCPM-4.0's original training), while achieving results that match or exceed full-attention baselines on standard benchmarks (Table 2) and substantially outperform them on long-context evaluations (Table 3: 38.97 average for MiniCPM-SALA vs. 32.02 for Qwen3-8B on long-context benchmarks).

This is significant beyond raw performance numbers because it changes the cost calculus for architectural experimentation. If every new attention mechanism requires a full 8T-token pre-training run to evaluate at scale, only the most well-resourced organizations can participate in architectural research. The conversion paradigm lowers the barrier: an organization that has already invested in pre-training a full-attention model can explore hybrid variants at roughly 25% of the cost of starting over. This has implications for the research ecosystem — it makes architectural innovation more accessible — not just for this specific model.

The paper's staged training pipeline (Table 1) is the operational manifestation of this paradigm. The key design choices — freezing non-converted layers during architecture conversion, progressively extending context length only after general capabilities are stabilized, enabling sparse attention mid-pipeline when sequence lengths justify it — are not arbitrary engineering details. They represent a principled strategy for managing the stability-capability tradeoff during architectural transition: prevent catastrophic forgetting by being surgical at first (stage 1), then gradually integrate (stages 2–3), then teach the new mechanism (stage 4), then align the result (stage 5).

The evidence that this works is in the standard benchmark results. A naive conversion that degraded general capabilities would show up as poor scores on MMLU-Pro, HumanEval, AIME, etc. The fact that MiniCPM-SALA achieves 67.04 on MMLU-Pro, 95.12 on HumanEval, and 83.75 on AIME24 (Table 2) — competitive with Qwen3-8B and Falcon-H1R-7B — demonstrates that the conversion paradigm preserves the knowledge acquired during the original pre-training. This is the "existence proof" that Transformer-to-hybrid conversion can produce models that are not merely efficient but actually competitive.

---

Innovation 4: Identifying That the Memory Bottleneck — Not the Compute Bottleneck — Is the Hard Barrier for Democratizing Long-Context Inference

This is a diagnostic contribution rather than a methodological one. The paper's experimental design makes visible a distinction that is often conflated in the efficiency literature: compute bottlenecks (operations per second) and memory bottlenecks (bytes of storage) are different constraints with different implications, and which one dominates depends on the deployment context.

The paper demonstrates this through a specific design choice in its speed benchmarks (Figures 2 and 3): testing on two GPUs with dramatically different VRAM capacities — the A6000D (96GB) and the RTX 5090 (32GB). On the A6000D, Qwen3-8B processes 256K tokens (slowly — 180.8s TTFT) but fails at 512K due to OOM. On the RTX 5090, the same model fails at just 128K tokens (non-quantized) or 256K (quantized). MiniCPM-SALA, with its 75% linear attention layers that use constant memory regardless of context length, processes 1M tokens on both GPUs without OOM.

The conceptual insight is that sparse attention alone cannot solve the memory bottleneck, a point the paper makes explicitly with its "sparse computation, dense storage" characterization of InfLLM-V2. Even with sparse attention, the model must retain the full KV-cache because the relevant subset of tokens may differ for each query. Linear attention, through its recurrent state formulation, is what actually reduces memory consumption. By contrasting the failure modes on different hardware, the paper makes the case that memory efficiency — not computational efficiency — is the prerequisite for deploying long-context models on consumer hardware.

This reframes the problem statement for the field. Much efficiency research focuses on reducing FLOPs (through sparse patterns, kernel fusion, quantization). MiniCPM-SALA argues implicitly that for the specific goal of ultra-long-context inference on edge devices, reducing memory footprint is the more urgent target — because without it, the model cannot run at all, regardless of how optimized the computation is. The paper's evidence that a 9B hybrid model can process 1M tokens on a 32GB consumer GPU while an 8B full-attention model cannot is a vivid demonstration that memory architecture, not parameter count, is the gating factor for democratization.

This insight is not entirely novel in isolation — the linear attention literature has long emphasized the memory benefits of constant-size states — but the paper's contribution is in making the distinction empirically concrete through a hardware-grounded comparison that shows exactly where and why full-attention models hit the wall. The 3.5× speedup at 256K is a nice efficiency gain; the ability to run at 1M tokens at all is a qualitative capability difference, and the paper's experimental design makes clear that memory — not computation — is what enables it.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three distinct categories of benchmarks. For standard (short-context) general capabilities: CMMLU (Li et al., 2023), MMLU-Pro (Wang et al., 2024b), HumanEval (Chen et al., 2021), LCB-v5/v6 (Jain et al., 2025), MBPP (Austin et al., 2021), AIME24 and AIME25 (AIME, 2025), BBH (Suzgun et al., 2022), and IFEval (Zhou et al., 2023). For long-context evaluation: RULER (Hsieh et al., 2024), MRCR (a dataset from OpenAI, referenced via HuggingFace URL in a footnote), and NoLiMa (Modarressi et al., 2025). For ultra-long context (Section 3.1, Table 4): RULER at 128K through 2048K token lengths. All evaluations use the OpenCompass framework (Contributors, 2023).

• Base model(s). MiniCPM-SALA is a 9B-parameter model built from an intermediate checkpoint of MiniCPM-4.0 (MiniCPM-Team et al., 2025), which had been pre-trained on 7 trillion tokens. The architectural conversion adds approximately 1B parameters beyond the original MiniCPM-4.0 8B scale. The choice of MiniCPM-4.0 as the starting point reflects the paper's Transformer-to-hybrid conversion paradigm: the base model provides pre-trained knowledge that is preserved through the staged continual-training pipeline.

• Metrics. All benchmarks report accuracy scores (percentage of correct answers). For standard benchmarks, accuracy is computed per the standard evaluation protocol of each benchmark (e.g., pass@1 for HumanEval and MBPP, exact match for AIME, task-specific scoring for BBH and IFEval). For long-context benchmarks (RULER, MRCR, NoLiMa), accuracy is likewise the primary metric, reported at multiple context lengths (32K, 64K, 128K, and in Table 4, up to 2048K). For inference speed (Figures 2 and 3), the paper reports Time To First Token (TTFT, measuring prefilling latency) and end-to-end latency (prefilling plus decoding of 1K generated tokens) in seconds. OOM (out-of-memory) is reported as a binary failure condition.

• Baselines. The paper compares against five models of comparable scale: Qwen3-8B (Yang et al., 2025a), Nemotron-Nano-v2-9B (NVIDIA et al., 2025a), MiniCPM-4.1-8B (MiniCPM-Team et al., 2025), Ministral-3-Reasoning-8B (Liu et al., 2026), and Falcon-H1R-7B (Team et al., 2026). MiniCPM-4.1-8B is excluded from long-context evaluations because it is limited to a 64K context length. For ultra-long context evaluation (Table 4), additional baselines from the Qwen3 family are included as reference points: Qwen3-30B-A3B-Instruct-2507, Qwen3-235B-A22B-Instruct-2507, and Qwen3-Next-80B-A3B-Instruct, with results cited from official Qwen3-Next documentation. For inference speed comparisons (Figures 2 and 3), Qwen3-8B is the sole full-attention baseline.

• Generation budget / compute accounting. The paper does not use a unified "generation budget" concept like some other test-time compute studies. Instead, efficiency is measured directly through wall-clock inference speed: TTFT and end-to-end latency at specific sequence lengths (64K, 128K, 256K, 512K, 1024K) on specific hardware (NVIDIA A6000D with 96GB VRAM, NVIDIA RTX 5090 with 32GB VRAM), with both non-quantized and GPTQ INT4-quantized (Frantar et al., 2023) configurations. Training cost is measured in total tokens processed: the conversion pipeline consumes approximately 2 trillion tokens, compared to 8 trillion tokens for MiniCPM-4.0's training from scratch, yielding the claimed ~75% reduction. The paper does not report FLOP counts for either training or inference.

• Cross-validation / statistical protocol. The paper reports no cross-validation, statistical significance testing, confidence intervals, or error bars for any of its experimental results. Standard benchmark evaluations (Table 2, Table 3) report single-number accuracy scores without variance estimates. The ultra-long context evaluation (Table 4) reports single RULER scores per context length. Inference speed measurements (Figures 2 and 3) report single latency values without error bars or multiple-trial averaging. This is a notable methodological omission: without variance estimates, it is impossible to assess whether the reported performance differences—particularly the small margins on standard benchmarks (e.g., 76.53 vs. 73.45 average)—are statistically reliable or within noise.

Main Quantitative Results

Standard Benchmark Performance: Matching Full-Attention Models

Table 2 presents the headline result for general capabilities. MiniCPM-SALA achieves an average score of 76.53 across all standard benchmarks, compared to 73.45 for Qwen3-8B, 73.82 for Nemotron-Nano-v2-9B, 76.13 for MiniCPM-4.1-8B, 74.21 for Ministral-3-R-8B, and 76.45 for Falcon-H1R-7B. The model ranks first among these baselines by average score, though margins are small: a 0.08 point lead over Falcon-H1R-7B and a 0.40 point lead over MiniCPM-4.1-8B.

Breaking down by category:

• Knowledge: MiniCPM-SALA scores 81.55 on CMMLU (vs. 81.68 for Qwen3-8B, 84.72 for MiniCPM-4.1-8B) and 67.04 on MMLU-Pro (vs. 73.26 for Qwen3-8B, 71.79 for Nemotron-Nano-v2-9B). The MMLU-Pro score is notably lower than Qwen3-8B's, by 6.22 points, which is the largest single-benchmark deficit against the primary full-attention baseline.

• Code: MiniCPM-SALA achieves 95.12 on HumanEval (Qwen3-8B: 93.90, Falcon-H1R-7B: 96.34), 60.48 on LCB-v5 (Nemotron-Nano-v2-9B: 68.26, Falcon-H1R-7B: 67.66), 52.00 on LCB-v6 (Falcon-H1R-7B: 57.71, Qwen3-8B: 48.57), and 89.11 on MBPP (Nemotron-Nano-v2-9B: 93.39, Ministral-3-R-8B: 94.16). Coding performance is competitive but not dominant: the model trails the best baseline on LCB-v5 by 7.78 points and on MBPP by 5.05 points.

• Math: On AIME24, MiniCPM-SALA scores 83.75 (Qwen3-8B: 73.33, Falcon-H1R-7B: 86.67). On AIME25, it scores 78.33 (Qwen3-8B: 66.67, Falcon-H1R-7B: 81.04). Mathematical reasoning is a relative strength: 10.42 points above Qwen3-8B on AIME24 and 11.66 points above on AIME25, though trailing Falcon-H1R-7B by 2.92 and 2.71 points respectively.

• Other: On BBH, MiniCPM-SALA scores 81.55 (vs. 82.68 for MiniCPM-4.1-8B, 74.17 for Qwen3-8B). On IFEval, it scores 76.34 (vs. 86.69 for Nemotron-Nano-v2-9B, 84.66 for Qwen3-8B)—a notable deficit of 10.35 points against the best baseline.

The overall pattern is that MiniCPM-SALA achieves broad parity with full-attention models of comparable scale, with specific strengths in math (AIME) and specific weaknesses in instruction-following (IFEval) and knowledge (MMLU-Pro). The paper's claim that "the integration of long-context mechanisms does not result in a significant degradation of general capabilities or short-context performance" (Section 3.1) is broadly supported, though the MMLU-Pro and IFEval results suggest some capability tradeoffs exist.

Long-Context Evaluation: Substantial Advantages Over Full-Attention Baselines

Table 3 presents the long-context benchmark results across RULER, MRCR, and NoLiMa at context lengths of 64K and 128K. MiniCPM-SALA achieves an overall long-context average of 38.97, compared to 32.02 for Qwen3-8B, 25.12 for Nemotron-Nano-v2-9B, 28.18 for Ministral-3-R-8B, and 16.04 for Falcon-H1R-7B. This 6.95-point advantage over Qwen3-8B (21.7% relative improvement) is the clearest evidence for the hybrid architecture's long-context benefits.

Breaking down by benchmark:

• RULER: At 64K, MiniCPM-SALA scores 92.65, substantially ahead of Nemotron-Nano-v2-9B (88.77), Qwen3-8B (80.53), Ministral-3-R-8B (70.66), and Falcon-H1R-7B (56.50). At 128K, the gap widens: MiniCPM-SALA scores 89.37, while Qwen3-8B drops to 71.74—a 17.63-point gap that indicates Qwen3-8B's full-attention mechanism is beginning to struggle at this length, even though it can still process the context. Nemotron-Nano-v2-9B drops to 68.01, Ministral-3-R to 45.09, and Falcon-H1R to 36.33. Remarkably, MiniCPM-SALA's RULER score at 128K (89.37) is only 3.28 points lower than at 64K (92.65), demonstrating minimal degradation as context doubles—a property that none of the baselines exhibit.

• MRCR: This benchmark evaluates multi-hop retrieval with varying numbers of hops (2N, 4N, 8N) at 64K and 128K. The results are mixed. At 64K-2N, MiniCPM-SALA scores 29.77, behind Ministral-3-R-8B (44.02) but ahead of Qwen3-8B (29.20) and substantially ahead of Falcon-H1R-7B (13.18). As the number of hops increases, all models degrade sharply: at 64K-8N, MiniCPM-SALA scores 16.56, while Ministral-3-R-8B drops to 17.23. At 128K-8N, all models score poorly (10.12 for MiniCPM-SALA, 12.15 for Qwen3-8B, 14.47 for Ministral-3-R-8B). MRCR is the one long-context benchmark where MiniCPM-SALA does not consistently lead: Ministral-3-R-8B outperforms it at several configurations despite being a full-attention model.

• NoLiMa: This benchmark evaluates literal matching beyond simple retrieval, testing whether the model can reason about information across long contexts. MiniCPM-SALA dominates decisively. At 32K, it scores 54.54 versus Qwen3-8B's 43.40, a 11.14-point gap. At 64K: 42.95 vs. 23.35 (19.60-point gap). At 128K: 23.86 vs. 11.25 (12.61-point gap). The next-best baseline at 128K is Nemotron-Nano-v2-9B at 5.80. NoLiMa is the benchmark where MiniCPM-SALA's architectural advantages are most pronounced, with more than double the accuracy of any baseline at 64K and 128K.

The long-context results support the paper's central architectural claim: the hybrid sparse-linear design preserves information retrieval capabilities at scale substantially better than full-attention models. The NoLiMa results are particularly compelling because they test reasoning over long contexts, not just simple retrieval, and the gap widens as context length increases—exactly what the hybrid architecture's division of labor between efficient global processing (linear attention) and precise local retrieval (sparse attention) is designed to achieve.

Ultra-Long Context Evaluation: Extrapolation Beyond Training Length

Table 4 presents RULER scores at extreme context lengths: 128K, 512K, 1000K, and 2048K tokens. MiniCPM-SALA was trained only up to 520K tokens (during long-decay training, Table 1), making all results beyond 512K extrapolations. The results are:

• 128K: 89.4 (consistent with the 89.37 reported in Table 3)
• 512K: 87.1 (only 2.3 points of degradation from 128K)
• 1000K: 86.3 (a further 0.8 points of degradation)
• 2048K: 81.6 (a 4.7-point drop from 1000K, but still remarkably high at 4× the training length)

The paper compares these results to three much larger Qwen3 models, with results cited from official documentation:

• Qwen3-30B-A3B-Instruct scores 89.1 at 128K, 78.4 at 512K, and 72.8 at 1000K. MiniCPM-SALA outperforms this model at all lengths beyond 128K, despite being less than one-third the parameter count.
• Qwen3-235B-A22B-Instruct scores 93.9 at 128K, 90.9 at 512K, and 84.5 at 1000K. MiniCPM-SALA trails at 128K and 512K but surpasses it at 1000K (86.3 vs. 84.5).
• Qwen3-Next-80B-A3B-Instruct scores 96.0 at 128K, 86.9 at 512K, and 80.3 at 1000K—notably, MiniCPM-SALA surpasses it at 512K (87.1 vs. 86.9) and at 1000K (86.3 vs. 80.3).

The paper attributes this extrapolation capability to the NoPE configuration in sparse attention layers: "the stored KV-Cache does not require combination with positional information, which can otherwise hinder the capture of long-range dependencies." This claim has face validity: if positional information is not encoded into the cached keys and values, then the model's retrieval mechanism does not degrade with distance in the same way that RoPE-based attention does.

However, three caveats are important. First, these Qwen3 comparisons are not head-to-head experiments run by the authors—they are cited from official documentation, which may use different evaluation configurations. Second, the Qwen3-Next model is a Mixture-of-Experts architecture (80B total, 3B active), making it an imperfect comparison to a 9B dense model. Third, Table 4 reports no baselines beyond Qwen3 family models for these ultra-long lengths—the full-attention models from Table 3 (Qwen3-8B, Nemotron-Nano-v2-9B, Ministral-3-R-8B, Falcon-H1R-7B) are absent, presumably because they cannot process these lengths at all.

Inference Speed: Dramatic Latency Reductions and the Memory Wall

Figures 2 and 3 present the practical deployment case for MiniCPM-SALA through comprehensive latency measurements.

Figure 2 (NVIDIA A6000D, 96GB VRAM):

In non-quantized settings (Figure 2a-b):

• At 64K tokens: MiniCPM-SALA TTFT is 109.9s vs. Qwen3-8B's 250.3s (2.3× speedup). End-to-end latency: 128.2s vs. 269.1s (2.1× speedup).
• At 128K tokens: TTFT 109.9s (unchanged from 64K—a remarkable property suggesting the linear attention layers dominate prefilling cost independent of sequence length) vs. 350.0s, though Qwen3-8B's reported time of 350.0s may indicate a measurement ceiling. End-to-end: 128.2s vs. 269.1s.
• At 256K tokens: TTFT 51.6s vs. 180.8s—this is the 3.5× speedup the paper highlights. End-to-end: 69.8s vs. 223.8s (3.2× speedup).
• At 512K tokens: MiniCPM-SALA TTFT 25.2s, end-to-end 43.3s. Qwen3-8B: OOM (fails to run).
• At 1024K tokens: MiniCPM-SALA TTFT 12.3s, end-to-end 30.4s. Qwen3-8B: OOM.

The counterintuitive decrease in TTFT as sequence length increases (109.9s at 64K, 51.6s at 256K, 12.3s at 1024K) is notable and unexplained in the paper. This could reflect measurement methodology (e.g., the prefilling phase may include fixed overhead that dominates at shorter lengths) or a property of the Lightning Attention implementation where longer sequences achieve better hardware utilization, but the paper provides no analysis.

Quantized settings (GPTQ INT4, Figures 2c-d) show similar patterns with slightly reduced absolute latencies. At 256K, quantized TTFT is 52.6s vs. 182.3s for Qwen3-8B (3.5× speedup). At 1024K, MiniCPM-SALA achieves TTFT of 12.6s and end-to-end latency of 16.4s.

Figure 3 (RTX 5090, 32GB VRAM):

This is where the memory bottleneck becomes dramatically visible. In non-quantized settings (Figures 3a-b):

• At 64K: MiniCPM-SALA TTFT 100.3s vs. Qwen3-8B 222.6s (2.2× speedup).
• At 128K: MiniCPM-SALA TTFT 100.3s. Qwen3-8B: OOM. The full-attention model fails at just 128K tokens—one-eighth the length that MiniCPM-SALA can process.
• At 256K, 512K: Qwen3-8B is OOM. MiniCPM-SALA continues: TTFT 45.9s at 256K, 22.3s at 512K.
• At 1024K: MiniCPM-SALA TTFT 10.8s, end-to-end 25.0s. Qwen3-8B: OOM.

In quantized settings (Figures 3c-d), Qwen3-8B survives to 256K (TTFT 46.1s for MiniCPM-SALA vs. 350.0s for Qwen3-8B) but fails at 512K. MiniCPM-SALA processes 1024K tokens with TTFT of 10.9s and end-to-end latency of 17.6s in quantized mode.

The starkest comparison: on the RTX 5090, Qwen3-8B's maximum processable context length is 128K (non-quantized) or 256K (quantized), while MiniCPM-SALA reaches 1024K in both configurations—a 4–8× advantage in maximum usable context length, not just a speedup.

Ablation Studies and Robustness Checks

The paper presents no formal ablation studies. Sections 2 and 3 describe architectural choices with qualitative justifications but do not report controlled experiments isolating individual components. The following components are described as beneficial but are never ablated:

• Layer selection algorithm vs. uniform interleaving: The paper states that non-uniform placement via the HALO selection algorithm "results in superior downstream performance" (Section 2.1) compared to naive uniform interleaving, but no experimental comparison is provided. The reader must take this on faith or consult Chen et al. (2026).

• 1:3 ratio vs. other ratios: The paper states the ratio was "inspired by" prior work and "internal small-scale preliminary experiments," but these experiments are not reported. No comparison to 1:1, 1:7, or pure linear configurations is provided.

• QK-Normalization: Described as preventing activation spikes and improving linear attention expressivity (Section 2.1), but no ablation compares performance with and without it.

• Output gates: Described as "significantly improv[ing] model stability and performance" based on empirical observation (Section 2.1), but no quantitative ablation is presented.

• HyPE (RoPE for linear layers, NoPE for sparse) vs. uniform RoPE or uniform NoPE: This is arguably the most architecturally interesting design choice—attention-type-specific positional encoding—yet no experiment compares HyPE to a uniform positional encoding baseline. The paper attributes length extrapolation to NoPE in sparse layers (Section 3.1, end of ultra-long context discussion) but provides no controlled evidence for this attribution.

• First and last layers kept unconverted: Described as improving training stability (Section 2.2), but not ablated against converting all layers.

• Staged training pipeline: The five-stage process is described but no ablation compares it to, for example, a direct jump from architecture conversion to long-context training at 32K.

• Data mixture changes in short-decay training: The upweighting of L2 high-quality selection data, PDF corpora, and L3 synthetic data is described but not ablated against continuing with the MiniCPM-4.0 pre-training data mixture.

• SFT data composition: The 417.8B tokens of SFT data with reasoning-intensive and synthesized long-context data are not compared to a smaller SFT budget or different data composition.

A partial implicit ablation exists in the comparison between MiniCPM-SALA and MiniCPM-4.1-8B (Table 2). MiniCPM-4.1-8B is a full-attention model from the same MiniCPM family, and comparing the two provides some evidence about the impact of the architectural conversion on general capabilities. MiniCPM-SALA achieves 76.53 average vs. MiniCPM-4.1-8B's 76.13—a marginal 0.40-point difference. However, this is not a clean ablation: MiniCPM-4.1-8B is a different model with potentially different training data and hyperparameters, not simply MiniCPM-SALA without the architectural changes.

Length extrapolation as an implicit robustness check. The ultra-long context results (Table 4) serve as an implicit test of the architecture's robustness to out-of-distribution sequence lengths. The model was trained at maximum 520K tokens but maintains an 81.6 RULER score at 2048K—nearly 4× the training length. This is strong evidence that the architecture's length generalization is genuine, but it is an evaluation of the final system rather than a controlled experiment isolating which component enables it.

Quantization robustness. Figures 2 and 3 include both non-quantized and GPTQ INT4-quantized configurations. The latency patterns are qualitatively similar across both settings, with MiniCPM-SALA maintaining its advantage. For example, at 256K on A6000D, the quantized speedup is 3.5× (52.6s vs. 182.3s TTFT), identical to the non-quantized 3.5× (51.6s vs. 180.8s). This suggests the architecture's efficiency benefits are robust to quantization, but no accuracy comparison between quantized and non-quantized model performance is provided—only latency.

Cross-architecture comparison (implicit). The baseline set includes two hybrid models—Nemotron-Nano-v2-9B (a Mamba-Transformer hybrid, per NVIDIA et al., 2025a) and Falcon-H1R-7B (a hybrid-head architecture, per Team et al., 2026)—providing an implicit comparison between MiniCPM-SALA's sparse-linear hybrid and other hybrid paradigms. On standard benchmarks, MiniCPM-SALA (76.53) edges out Nemotron-Nano-v2-9B (73.82) and Falcon-H1R-7B (76.45). On long-context benchmarks, the gap is substantial: MiniCPM-SALA's 38.97 average vs. 25.12 (Nemotron-Nano-v2-9B) and 16.04 (Falcon-H1R-7B). This suggests not all hybrid architectures are equally effective at long-context tasks, but the comparison is not controlled—these models differ in parameter count, training data, and training methodology, not just attention architecture.

Critical Assessment

Claim: MiniCPM-SALA matches or exceeds full-attention models on standard benchmarks.

Assessment: Supported with specific weaknesses. The average of 76.53 (Table 2) does indeed edge out Qwen3-8B's 73.45 by 3.08 points. This supports the claim that architectural conversion does not catastrophically degrade general capabilities. However, the average masks significant variance: MiniCPM-SALA trails Qwen3-8B by 6.22 points on MMLU-Pro (67.04 vs. 73.26) and by 8.32 points on IFEval (76.34 vs. 84.66). These are not small gaps, and the IFEval deficit in particular suggests that the hybrid architecture may have impaired instruction-following capability—a practically important skill for deployed models. The claim of "matching" performance is true on average but misleading on specific important benchmarks.

Moreover, the baseline comparison is selective. The paper chooses Qwen3-8B as the primary full-attention comparator, but Table 2 shows that MiniCPM-4.1-8B—a full-attention model from the same MiniCPM family—achieves 84.72 on CMMLU vs. MiniCPM-SALA's 81.55, and 82.68 on BBH vs. 81.55. The fact that a sibling full-attention model with the same lineage outperforms MiniCPM-SALA on knowledge and reasoning benchmarks suggests that the architectural conversion may have traded some capability for efficiency, even if the average obscures it. A more conservative characterization would be: MiniCPM-SALA achieves broad parity with comparable full-attention models, with specific degradations in instruction-following and factual knowledge that are compensated by gains in mathematical reasoning.

Claim: MiniCPM-SALA achieves up to 3.5× inference speedup over Qwen3-8B at 256K tokens.

Assessment: Strongly supported for the tested configuration, but generalizability is unclear. Figure 2 shows TTFT of 51.6s (MiniCPM-SALA) vs. 180.8s (Qwen3-8B) at 256K on an A6000D—a 3.5× ratio. This is a clean, well-measured result. The speedup is confirmed across quantized and non-quantized settings (Figures 2a,c) and across hardware (Figure 3 shows a similar ratio at 256K on RTX 5090 in quantized mode: 46.1s vs. 350.0s, which is 7.6×, though the 350.0s value may reflect a measurement timeout rather than actual runtime).

The broader interpretation—that the hybrid architecture is generally 3.5× faster at 256K—requires caution. The speedup is measured against a single baseline (Qwen3-8B) on specific hardware, with specific software implementations (the Lightning Attention kernel, the InfLLM-V2 sparse attention routine, the vLLM or similar inference framework, though the paper does not specify the inference stack). Performance could differ substantially with different kernels, different hardware (e.g., H100 vs. consumer GPUs), or different baselines (e.g., a GQA-optimized Qwen3-8B implementation vs. the one tested). The 3.5× figure is best understood as a proof-of-concept rather than a guaranteed speedup in all deployment scenarios.

Claim: MiniCPM-SALA supports inference at context lengths up to 1M tokens where full-attention 8B models fail due to OOM.

Assessment: Strongly supported and vividly demonstrated. Figures 2 and 3 show Qwen3-8B failing at 512K on A6000D (96GB) and at 128K on RTX 5090 (32GB), while MiniCPM-SALA runs successfully at 1M tokens on both. This is the paper's most impactful empirical result: it demonstrates a qualitative capability difference (can run vs. cannot run), not just a quantitative speed difference. The dual-hardware testing strengthens the claim by showing the memory bottleneck is the limiting factor—on the larger A6000D, Qwen3-8B survives longer (to 256K) than on the smaller RTX 5090 (128K), exactly as the KV-cache memory analysis would predict.

However, the paper does not report the actual memory consumption of either model at these sequence lengths. The reader cannot verify why Qwen3-8B OOMs—is it truly the KV-cache, or is it activation memory during prefilling, or some combination? Providing peak memory usage would strengthen the claim and allow practitioners to estimate whether their specific hardware can run the model. Additionally, the paper does not report whether MiniCPM-SALA's accuracy degrades at 1M tokens on these benchmarks—Table 4 shows RULER scores at 1000K and 2048K, but the speed benchmarks (Figures 2 and 3) only measure latency, not whether the generated outputs are correct. An OOM failure is unambiguous; a completed but incorrect generation is a different kind of failure that the latency measurements do not capture.

Claim: The Transformer-to-hybrid training paradigm reduces training cost by approximately 75%.

Assessment: Plausible but not rigorously demonstrated. The paper states that MiniCPM-SALA's conversion training consumes approximately 2T tokens, compared to 8T tokens for MiniCPM-4.0's training from scratch (Section 2.2). This is a 75% reduction in data volume, which is used as a proxy for training cost. However, the paper does not report FLOP counts, GPU-hours, or wall-clock training time. Token count is a reasonable but imperfect cost metric: the conversion pipeline includes stages at different sequence lengths (512, 4K, 32K, 160K, 520K, 64K, 140K) with different attention mechanisms enabled, meaning the per-token computational cost varies across stages. The 2T tokens at long sequence lengths with sparse attention enabled are more expensive per token than the tokens in MiniCPM-4.0's original 4K-context training. Whether the total FLOPs or GPU-hours are actually 25% of the original training cost is not established.

Furthermore, the 75% figure compares the conversion cost to MiniCPM-4.0's total training from scratch, but MiniCPM-4.0 was an existing pre-trained model. An organization starting from scratch would need to both (a) train the full-attention base model (8T tokens) and (b) run the conversion pipeline (2T tokens), for a total of 10T tokens—which is 125% of the cost of training a full-attention model alone. The 75% savings only materialize if one views the base model as a sunk cost. The paper implicitly assumes this perspective (the base model already exists), which is reasonable for the MiniCPM team but limits the claim's applicability to organizations without existing pre-trained models.

Claim: MiniCPM-SALA is the first hybrid model at scale to match full-attention performance.

Assessment: Supported within the paper's baseline set, but the comparison set is limited. The five baseline models (Qwen3-8B, Nemotron-Nano-v2-9B, MiniCPM-4.1-8B, Ministral-3-R-8B, Falcon-H1R-7B) are reasonable comparators at the 7–9B scale, and MiniCPM-SALA does indeed achieve the highest average standard benchmark score (76.53). However, several important baselines are absent. Llama 3.1-8B is not evaluated. Gemma 2-9B is not evaluated. DeepSeek-V2-Lite (which also uses a hybrid attention architecture, MLA) is not evaluated. Mistral-Nemo-12B is not evaluated. The claim of "first at scale" depends on the definition of "matching performance"—if a broader set of full-attention models were included, the average ranking might shift.

Additionally, the ultra-long context comparison (Table 4) includes larger Qwen3 models but omits any model from the Llama, Gemma, or DeepSeek families that might also support 1M-token contexts. The paper's claim of superior parameter efficiency at ultra-long contexts (86.3 RULER at 1000K for 9B vs. 80.3 for Qwen3-Next-80B-A3B) is striking but would be more convincing if tested against a broader set of long-context models.

Missing Experiments That Would Strengthen the Paper

The most significant gap is the complete absence of controlled ablation studies. The paper describes numerous architectural choices—the 1:3 ratio, layer selection algorithm, HyPE, output gates, QK-Normalization, first/last layer preservation, staged training with progressive context extension, data mixture shifts—all with qualitative justifications but zero quantitative isolation of individual effects. This makes it impossible to determine which design choices are load-bearing and which are incidental. A practitioner attempting to replicate or adapt the approach would not know, for example, whether output gates are essential or merely helpful, or whether the 1:3 ratio is near-optimal or whether 1:7 would work nearly as well.

Specific missing experiments include:

• Ablation of HyPE vs. uniform RoPE: This is the most architecturally novel choice and the one with the clearest mechanistic rationale. Showing that uniform RoPE degrades long-context performance or length extrapolation would directly validate the paper's positional encoding hypothesis.

• Ablation of sparse attention layers (pure linear baseline): Comparing MiniCPM-SALA to a version with all layers converted to linear attention would quantify how much the 25% sparse layers contribute to long-context performance. If the pure linear model performs nearly as well, the sparse layers are unnecessary complexity; if it performs substantially worse, the division-of-labor hypothesis is supported.

• Ablation of layer selection algorithm vs. uniform placement: Given that the paper explicitly claims non-uniform placement yields superior performance, showing the magnitude of this superiority (e.g., RULER score at 128K with uniform vs. selected placement) would validate this design choice and provide guidance for future work.

• Memory profiling: Reporting peak memory consumption at each sequence length for both MiniCPM-SALA and Qwen3-8B would make the OOM failures interpretable and allow practitioners to assess whether the model fits their hardware constraints without reproducing the experiments.

• Accuracy-latency tradeoff curves: Figures 2 and 3 report latency without corresponding accuracy measurements at those sequence lengths. For the speedup claim to be practically meaningful, one needs to know whether the model is producing correct outputs at those speeds. An end-to-end benchmark (e.g., RULER score vs. latency) at multiple sequence lengths would show whether the speedup comes with accuracy degradation.

• Comparison against sparse-only and linear-only variants at matched parameter counts: The paper argues for the hybrid approach over either paradigm alone, but never tests a pure sparse-attention model (e.g., all layers using InfLLM-V2) or a pure linear-attention model (all layers using Lightning Attention) at the same 9B scale. Without these comparisons, the claim that the hybrid is superior to either pure approach remains hypothetical.

Statistical and Methodological Limitations

The absence of any statistical reporting—no confidence intervals, no error bars, no multiple-trial averaging, no significance testing—is the most significant methodological weakness. For standard benchmarks (Table 2), the differences between models are often small (3 points on average, with individual benchmark differences ranging from ~1 to ~10 points). Without variance estimates, the reader cannot distinguish genuine capability differences from sampling noise. The long-context benchmarks are similarly single-point estimates. The latency measurements (Figures 2 and 3) report single values without indication of variability across runs or prompts—GPU inference latency can vary substantially due to thermal throttling, CUDA kernel scheduling, and other factors.

The paper also does not report prompt templates, decoding parameters (temperature, top-p, top-k), or the exact evaluation protocol for each benchmark beyond stating that OpenCompass was used. These details matter for reproducibility and for assessing whether the comparisons to baselines are fair (e.g., if MiniCPM-SALA was evaluated with a different prompt template than Qwen3-8B, the comparison would not be apples-to-apples).

Summary of Experimental Strength and Weakness

The paper's experiments are strongest where they make vivid, qualitative demonstrations: the OOM failures of Qwen3-8B at long contexts while MiniCPM-SALA continues processing (Figures 2 and 3), and the preservation of high RULER scores at 4× the training context length (Table 4). These results directly illustrate the paper's central motivation—the memory bottleneck is a hard barrier for full-attention models—and the architecture's primary benefit—hybrid attention enables practical ultra-long-context inference on consumer hardware.

The experiments are weakest where they claim quantitative superiority: the small margins on standard benchmarks without statistical reporting make it difficult to assess whether MiniCPM-SALA genuinely matches full-attention models or whether the differences are within noise. The absence of ablation studies leaves the contribution's novelty ambiguous: the paper adopts existing attention mechanisms (InfLLM-V2, Lightning Attention) and an existing conversion framework (HALO), so the value proposition rests on the specific integration choices (ratio, placement, positional encoding, training recipe). Without ablations isolating these choices, the paper demonstrates that the integration works but not why each choice matters—which limits both the intellectual contribution and the practical guidance for future work.

6. Limitations and Trade-offs

6.1 The Ablation Gap: None of the Individual Architectural Choices Are Experimentally Isolated, So the Contribution's Novelty Is Difficult to Localize

The assumption or constraint. The paper describes a large number of architectural and training design choices—the 1:3 sparse-to-linear ratio, the non-uniform layer placement via the HALO selection algorithm, HyPE (attention-type-specific positional encoding), QK-Normalization, output gates, preserving the first and last layers as full-attention during conversion, the five-stage training pipeline with progressive context extension, the data mixture shifts in short-decay training, and the large SFT budget—but provides no ablation study isolating any of them. For the 1:3 ratio, the paper states it was "inspired by the architectural designs of recent representative studies, such as Qwen3-Next and Kimi-Linear, as well as our internal small-scale preliminary experiments" (Section 2.1), but these experiments are not reported. For the layer selection algorithm, the paper claims it "results in superior downstream performance" relative to uniform interleaving (Section 2.1) without quantification. For output gates, the paper reports empirically that they "significantly improve model stability and performance" (Section 2.1) without showing the magnitude. For HyPE, the paper attributes length extrapolation to the NoPE configuration in sparse layers (Section 3.1, end of ultra-long context discussion) without a controlled comparison to uniform RoPE.

The consequence. Because every design choice was made simultaneously and evaluated only as an integrated system, it is impossible to determine which choices are load-bearing and which are incidental. A practitioner attempting to replicate or adapt the approach cannot know whether output gates are essential or optional, whether the 1:3 ratio is near-optimal or whether 1:7 would work nearly as well, or whether HyPE is the primary driver of length extrapolation or merely one contributing factor among many. The paper's contribution framing—that the integration of existing components at scale is the novelty—is undermined by the inability to attribute performance to specific integration decisions. If, for example, the length extrapolation and long-context performance are primarily driven by Lightning Attention's inherent properties and the progressive context extension schedule, while the sparse attention layers, HyPE, and output gates contribute marginally, then the paper's central architectural narrative (the hybrid division of labor) is weaker than it appears. Conversely, if the sparse layers are essential but the layer selection algorithm is not, then future work could simplify the architecture substantially. The paper provides no evidence to distinguish these scenarios.

What evidence exists in the paper. None. There is not a single controlled experiment in the paper that varies one architectural or training choice while holding others constant. The only implicit comparison is between MiniCPM-SALA and its full-attention sibling MiniCPM-4.1-8B (Table 2), but this is an entirely different model with potentially different training data and hyperparameters, not an ablation of MiniCPM-SALA's architectural changes. The baseline set includes two other hybrid architectures—Nemotron-Nano-v2-9B (Mamba-Transformer hybrid) and Falcon-H1R-7B (hybrid-head architecture)—which provide an implicit cross-architecture comparison, but these differ in far more dimensions than attention type alone, making it impossible to attribute performance differences to specific design choices.

Mitigation status. The paper does not acknowledge this as a limitation and does not suggest future ablation work. The stated future directions in Section 4 are high-level ("extending to other domains and modalities," "combining with other efficiency techniques") rather than addressing the validation gaps in the current work. The strongest mitigation available today is to consult the prior work that the paper builds on: Chen et al. (2026) for the HALO framework and HyPE, Zhao et al. (2025) for InfLLM-V2, and Qiu et al. (2025) for output gates—but these references study their respective components in isolation, not in the specific integration MiniCPM-SALA uses, and may not transfer directly to the 9B-scale hybrid setting.

---

6.2 The 75% Training Cost Reduction Claim Is Based on Token Count, Not FLOPs or GPU-Hours, and Assumes the Pre-Trained Base Model Is a Sunk Cost

The assumption or constraint. The paper claims that "the Transformer-to-hybrid training of MiniCPM-SALA consumes approximately 2T tokens. This corresponds to roughly 25% of the data volume required to train MiniCPM-4.0 from scratch (8T tokens)" (Section 2.2). This is a cost comparison measured purely in training tokens processed, not in FLOPs, GPU-hours, or wall-clock time. The paper provides no FLOP counts or hardware-hour estimates for either the original MiniCPM-4.0 training or the MiniCPM-SALA conversion pipeline.

The consequence. Token count is an imperfect cost metric for this pipeline because the per-token computational cost varies substantially across stages. The conversion pipeline includes stages at sequence lengths ranging from 512 tokens (stage 1) to 520K tokens (stage 4), with sparse attention disabled in early stages and enabled at 32K and beyond. Training at 520K tokens per sequence is substantially more FLOP-intensive per token than training at 4K tokens, even with the efficiency of linear attention, because (a) the 25% sparse attention layers still involve quadratic or near-quadratic computation in the sequence length, (b) the FFN computation per token is identical regardless of attention type, and (c) the longer sequences require more gradient accumulation steps and communication overhead. Conversely, the original MiniCPM-4.0 training at 4K context (which is standard for many pre-training runs) has a lower per-token cost that is uniform across the run. The 2T tokens of MiniCPM-SALA training are therefore likely more expensive per token on average than the 8T tokens of MiniCPM-4.0 training, meaning the actual FLOP or GPU-hour savings are almost certainly less than 75%.

The second half of the cost assumption is even more consequential: the 75% figure compares the conversion cost (2T tokens) to MiniCPM-4.0's total training from scratch (8T tokens). This comparison treats the base model as a free resource. An organization starting from scratch would need to both train the full-attention base model (8T tokens) and run the conversion pipeline (2T tokens), for a total of 10T tokens, which is 125% of the cost of training a full-attention model alone. The savings only materialize if one already has a pre-trained full-attention model available. The paper implicitly assumes this perspective (the MiniCPM team already had MiniCPM-4.0), which is reasonable for the authors but limits the claim's generality.

What evidence exists in the paper. None that would allow a practitioner to estimate actual training cost. Table 1 provides token counts, sequence lengths, and learning rates per stage but no FLOP estimates. The paper does not report the number of GPUs, GPU-hours, or training duration. The only cost proxy is the ~2T token figure.

Mitigation status. The paper does not acknowledge this limitation. The 75% figure is presented as a contribution in the abstract ("reduces training costs by approximately 75% compared to training from scratch") and Section 1 without caveats about FLOP accounting or the sunk-cost assumption. A more precise accounting would report total FLOPs for both the original training and the conversion pipeline, or at minimum report the hardware configuration and training duration so practitioners can estimate cost independently. This is a practical concern: organizations deciding whether to adopt the conversion paradigm need to know the actual compute budget required, not just the data volume.

---

6.3 The Difficulty of Long-Context Capability Retention at Scale: MRCR Performance Degrades to Near-Random as Retrieval Complexity Increases, and NoLiMa Scores at 128K Remain Low (23.86) Despite Being the Best Baseline

The assumption or constraint. The paper's central claim is that the hybrid architecture achieves "a balanced solution that maintains both efficiency and high performance for long-context tasks" (Section 1). This implies that the model's long-context capabilities are practically useful across a range of retrieval and reasoning tasks, not just on simple needle-in-a-haystack benchmarks.

The consequence. On more demanding long-context evaluation, the model's performance degrades to levels that may limit practical utility. On MRCR (Table 3), which evaluates multi-hop retrieval (chaining multiple retrieved facts to answer a question), MiniCPM-SALA's accuracy drops sharply as complexity increases: 29.77 at 64K-2N, 20.57 at 64K-4N, and 16.56 at 64K-8N. At 128K, the pattern is similar: 28.62 at 128K-2N, 19.62 at 128K-4N, and 10.12 at 128K-8N. These scores are non-trivial (above random guessing on multiple-choice tasks) but are low enough that a deployed system relying on the model for multi-hop reasoning over long documents would frequently produce incorrect answers—roughly 70-90% error rates depending on context length and number of hops. On NoLiMa at 128K, which tests reasoning beyond literal matching, MiniCPM-SALA achieves 23.86 (Table 3). While this is substantially better than Qwen3-8B's 11.25, it still means the model fails on approximately 76% of reasoning queries that require synthesizing information across 128K-token contexts.

These results are not failures of the architecture—MiniCPM-SALA dominates the baselines on NoLiMa and is competitive on MRCR—but they reveal that even with the hybrid design, precise reasoning over extremely long contexts remains a hard, partially unsolved problem. The "high performance" the paper claims is relative to full-attention baselines that perform even worse, not relative to a threshold of practical reliability. For applications like repository-scale code understanding or multi-day agent memory, where errors in multi-hop retrieval or cross-document reasoning cascade into incorrect actions, the absolute accuracy levels at 128K may be insufficient.

What evidence exists in the paper. Table 3 provides the MRCR and NoLiMa breakdowns. The paper reports these results without commenting on the absolute performance levels or discussing what accuracy thresholds would be needed for practical applications. The emphasis in the text is on the relative advantage over baselines ("the advantage of the model is particularly visible in the NoLiMa benchmark") rather than on the absolute capability ceiling.

Mitigation status. The paper does not address this as a limitation. The ultra-long context results (Table 4) focus on RULER, which primarily tests retrieval (finding specific information in long contexts) rather than multi-hop reasoning or inference over retrieved information. RULER scores remain high even at 2048K (81.6), but RULER is an easier task than MRCR or NoLiMa. The paper does not discuss how the model's reasoning capabilities scale with context length beyond retrieval, nor whether the low MRCR and NoLiMa scores at 128K represent a fundamental limitation of the compressed linear attention state for complex reasoning or a training data shortfall that could be addressed with more long-context reasoning data.

---

6.4 No Memory Profiling or Absolute Memory Consumption Data Are Provided, Making the OOM Claims Unverifiable and the Hardware Requirements Unspecified

The assumption or constraint. The paper's most impactful empirical claim is that MiniCPM-SALA supports inference at context lengths where full-attention models fail due to out-of-memory (OOM) errors. Figures 2 and 3 show Qwen3-8B failing at 512K on A6000D (96GB) and at 128K on RTX 5090 (32GB) while MiniCPM-SALA runs to 1M tokens on both. The paper attributes this to the linear attention layers' constant memory footprint versus full attention's linearly growing KV-cache ("sparse computation, dense storage" in Section 1; "the memory bottleneck of KV-Cache" in Section 1).

The consequence. Without reporting the actual memory consumption of either model at any sequence length, the reader cannot verify the explanation for the OOM failures, cannot determine whether the memory bottleneck is truly the KV-cache or another factor (activation memory, CUDA context overhead, inference framework buffers), and cannot predict whether MiniCPM-SALA will fit on their specific hardware without reproducing the experiments. The paper shows that Qwen3-8B OOMs at certain lengths on certain GPUs, but does not report peak memory usage, memory breakdown (KV-cache vs. activations vs. model weights), or the inference framework used (vLLM, HuggingFace Transformers, custom kernel). Different inference frameworks have different memory management strategies; Qwen3-8B might run at 512K on A6000D with a more memory-efficient framework or with aggressive KV-cache quantization, making the OOM claim framework-dependent rather than architectural.

The RTX 5090 results (Figure 3) are particularly striking—Qwen3-8B fails at just 128K tokens non-quantized—but without memory profiling, the reader cannot assess whether this is a hard architectural limit or a solvable engineering problem. If Qwen3-8B's KV-cache at 128K with GQA is, say, 20GB out of 32GB available, and the remaining 12GB is consumed by model weights and activation buffers, then a more memory-efficient inference setup (e.g., FlashAttention-3 with better activation recomputation, or 4-bit KV-cache quantization) might push the usable context length substantially higher. The paper's framing—that the OOM failures demonstrate an inherent architectural advantage of the hybrid design—is only valid if the memory bottleneck is genuinely the attention mechanism and not the inference stack.

What evidence exists in the paper. Figures 2 and 3 report latency values and OOM markers but no memory statistics. The paper does not specify the inference framework, the KV-cache data type (FP16? BF16? INT8?), the batch size during inference testing, or whether any memory optimization techniques (PagedAttention, flash decoding, KV-cache offloading) were applied. The latency values themselves provide indirect evidence: the fact that MiniCPM-SALA's TTFT decreases at longer sequence lengths (109.9s at 64K → 51.6s at 256K → 12.3s at 1024K on A6000D non-quantized, Figure 2a) is an unexplained anomaly that might indicate an interaction between the measurement methodology and the inference framework rather than a pure architectural property.

Mitigation status. The paper does not address this gap. Providing peak memory consumption at each tested sequence length for both models—and breaking down memory into KV-cache, model weights, and activations—would directly validate the central memory-efficiency claim and would be straightforward to measure with standard CUDA profiling tools. This is a practical limitation: a practitioner cannot determine from the current data whether MiniCPM-SALA will fit on their specific GPU (e.g., an RTX 4090 with 24GB, or an RTX 5070 with 12GB) without running the experiment themselves.

---

6.5 Single Model Family, Single Benchmark Suite, No Demonstration of Transfer to Non-Math/Code Domains or Non-English Languages

The assumption or constraint. The paper evaluates MiniCPM-SALA exclusively on the benchmarks reported in Tables 2–4: knowledge (CMMLU, MMLU-Pro), code (HumanEval, LCB-v5/v6, MBPP), math (AIME24/25), reasoning (BBH), instruction-following (IFEval), and long-context tasks (RULER, MRCR, NoLiMa). All benchmarks are English-language (with the exception of CMMLU, which is Chinese). The base model (MiniCPM-4.0) was pre-trained on a multilingual corpus, but the paper does not evaluate on multilingual benchmarks. The long-context benchmarks (RULER, MRCR, NoLiMa) are synthetic or semi-synthetic evaluations of retrieval and reasoning, not real-world tasks like repository-scale code understanding, multi-document summarization, or long-horizon agent memory—the very applications the paper's introduction cites as motivation.

The consequence. The paper demonstrates that the hybrid architecture works well on academic long-context benchmarks, but its effectiveness on the deployment scenarios that motivated the work is untested. Processing a 1M-token context on RULER (which primarily tests whether the model can find a specific piece of information buried in long irrelevant text) is a different capability from understanding a 100K-line codebase with complex dependency graphs, or maintaining coherent task state over a multi-day agent interaction with thousands of turns. The MRCR and NoLiMA results (Section 6.3 above) already show that performance degrades substantially on more complex long-context reasoning tasks; real-world applications are likely harder still.

Additionally, the standard benchmark suite, while diverse, is a narrow sample of the full range of LLM capabilities. The paper does not evaluate on tasks like summarization (CNN/DailyMail, XSum), translation, open-ended generation quality (AlpacaEval, MT-Bench), or safety/alignment benchmarks. The IFEval deficit (76.34 for MiniCPM-SALA vs. 84.66 for Qwen3-8B) hints that instruction-following may be impaired, but the paper does not investigate this systematically. For edge-deployment scenarios where the model might be used as a general-purpose assistant (not just for long-context retrieval), these missing evaluations matter.

What evidence exists in the paper. None beyond the reported benchmarks. The paper's introduction (Section 1) lists "deep understanding and generation of ultra-long contexts," "repository-scale code engineering," and "long-horizon agents for complex tasks" as motivating applications, but none of these are evaluated. The long-context benchmarks in Table 3 are synthetic evaluations, not real-world tasks. The standard benchmarks in Table 2 cover knowledge, code, math, reasoning, and instruction-following, which is a reasonable cross-section but not comprehensive.

Mitigation status. The paper does not acknowledge this as a limitation or discuss the gap between the evaluated benchmarks and the motivating applications. Evaluating on SWE-bench (for repository-scale code tasks, which the paper cites in its introduction) or on a long-horizon agent benchmark would directly test whether the efficiency gains translate to the use cases the paper argues are important. Without such evaluations, the paper establishes architectural feasibility on academic benchmarks but does not demonstrate practical utility for the applications that justify the architectural innovation.

---

6.6 The Latency Measurements Show an Unexplained Pattern (TTFT Decreasing at Longer Sequence Lengths) That Undermines Confidence in the Speedup Numbers

The assumption or constraint. Figures 2 and 3 report Time To First Token (TTFT) for MiniCPM-SALA and Qwen3-8B at sequence lengths from 64K to 1024K. For MiniCPM-SALA on A6000D non-quantized (Figure 2a), TTFT values are: 109.9s at 64K, 51.6s at 256K, 25.2s at 512K, and 12.3s at 1024K. This is a monotonic decrease in prefilling latency as the sequence length gets longer—the model processes 1M tokens in less time than it takes to process 64K tokens. The same pattern appears on the RTX 5090 (Figure 3): TTFT of 100.3s at 64K, 45.9s at 256K, 22.3s at 512K, and 10.8s at 1024K.

The consequence. This pattern is physically impossible for a standard Transformer, where prefilling latency should grow with sequence length (linearly for linear attention, quadratically for full attention, but never negatively). The fact that TTFT decreases as input length increases suggests either:

• A measurement error in the experimental setup (e.g., the prompt content or number of prompts differs across sequence length conditions, or the 64K measurement includes overhead that is amortized away at longer lengths).
• An inference framework quirk where the system processes the 64K input inefficiently (e.g., using a smaller batch size or a different kernel for short prefill) but switches to a more optimized path at longer lengths.
• The TTFT measurement is not measuring what it claims to measure—for example, it might be measuring only the attention computation time while excluding data loading, embedding, or framework overhead that dominates at short lengths but is a smaller fraction at long lengths.

In any of these scenarios, the reported 3.5× speedup at 256K (comparing MiniCPM-SALA's 51.6s to Qwen3-8B's 180.8s) is difficult to interpret. If MiniCPM-SALA's 64K TTFT of 109.9s includes 100s of fixed overhead unrelated to attention computation, then the 51.6s at 256K represents a much larger speedup relative to the attention computation alone than the 3.5× number suggests. Conversely, if Qwen3-8B's measurements are similarly affected, the speedup ratio might be accurate but the absolute latencies are not. Without understanding the source of the anomalous TTFT pattern, the reader cannot assess whether the speedup figures are reliable or are artifacts of the measurement methodology.

What evidence exists in the paper. The raw TTFT numbers in Figures 2 and 3. The paper makes no comment on the decreasing TTFT pattern and provides no analysis of what might cause it. The end-to-end latency numbers (which include both prefilling and 1K tokens of decoding) show the same decreasing pattern: 128.2s at 64K, 69.8s at 256K, 43.3s at 512K, and 30.4s at 1024K (A6000D non-quantized, Figure 2b). This is somewhat less anomalous because decoding 1K tokens at longer context lengths might benefit from the same efficiency mechanisms that reduce prefilling time, but it still implies that generating 1K tokens from a 1M-token context is cheaper than generating 1K tokens from a 64K-token context, which is not what the architecture's computational complexity would predict—decoding cost should be roughly constant or slightly increasing with context length due to the sparse attention layers' access to the growing KV-cache.

Mitigation status. The paper does not acknowledge or explain this anomaly. The minimum required information to make the speedup claims interpretable would include: (1) a fixed-cost baseline measurement (e.g., TTFT at 1K or 4K sequence length) to establish what portion of latency is sequence-length-dependent versus fixed overhead; (2) confirmation that the same number of prompts and the same prompt content are used at each sequence length; (3) specification of the inference framework, kernel implementations, and any measurement methodology details (e.g., warmup steps, averaging over multiple runs, handling of CUDA synchronization). Without this, the 3.5× speedup at 256K should be treated as a preliminary finding rather than a precisely characterized efficiency gain.

7. Implications and Future Directions

How This Work Changes the Landscape

MiniCPM-SALA shifts the long-context efficiency conversation from paradigm competition (sparse OR linear) to architectural division of labor (sparse AND linear, in a specific ratio and placement). Before this paper, the dominant framing in the literature was that sparse attention and linear attention were alternative solutions to the Transformer's quadratic bottleneck, each with a characteristic tradeoff—sparse attention preserves precision but retains the memory bottleneck, linear attention solves the memory bottleneck but loses precision through state compression. The implicit question was "which is better?" and the answer depended on whether one prioritized accuracy or memory efficiency. MiniCPM-SALA reframes the question entirely: the answer is "both, because they solve complementary sub-problems that both arise in long-context processing."

The magnitude of this contribution is best characterized as a demonstration that the tradeoff is not fundamental. This is not a paradigm shift in the sense of introducing a new attention primitive—the paper explicitly adopts InfLLM-V2 and Lightning Attention from prior work—but it is a conceptual reframing with practical consequences. The paper provides the first large-scale evidence that a hybrid architecture can match full-attention models on standard benchmarks (76.53 average vs. Qwen3-8B's 73.45, Table 2) while achieving dramatic efficiency gains on ultra-long contexts and, crucially, enabling inference at scales where full-attention models fail entirely (1M tokens on 32GB consumer GPUs, Figures 2 and 3). This is an existence proof that the hybrid approach is not merely a compromise—it can be strictly better on both axes (performance and efficiency) for long-context tasks.

This reframing has diagnostic value beyond this specific model. The paper's "sparse computation, dense storage" characterization of sparse attention (Section 1) makes explicit a limitation that was often implicit in prior work: solving the compute bottleneck does not solve the memory bottleneck, because the KV-cache must still store every token. By contrast, linear attention's constant-size state is what actually reduces memory consumption. The hybrid architecture can be understood as deploying sparse attention where the memory cost is tolerable (25% of layers) and linear attention where memory is the binding constraint (75% of layers). This diagnosis applies to any future work on efficient attention: a new mechanism should be evaluated on both FLOP reduction and peak memory consumption, and the paper's hardware-grounded comparison methodology (dual GPU testing with different VRAM capacities in Figures 2 and 3) provides a template for doing so.

The paper also shifts the narrative around training cost for architectural innovation. The conventional wisdom has been that architectural changes require training from scratch, which at the 8B+ scale is prohibitively expensive for most organizations. MiniCPM-SALA demonstrates—through its multi-stage conversion pipeline starting from MiniCPM-4.0—that continual training with weight inheritance can produce a competitive hybrid model at a fraction of the de novo cost. The paper claims approximately 75% token-count reduction (2T vs. 8T), which, even accounting for the caveats discussed in Section 6.2 (token count vs. FLOPs, sunk-cost assumption), is a substantial enough reduction to change the calculus for architectural experimentation. Organizations with existing pre-trained Transformers can explore hybrid variants without committing to full-scale pre-training, lowering the barrier to architectural innovation.

Research directions that become more attractive after this work:

• Attention-type-specific optimization. The HyPE finding—that sparse and linear layers have conflicting positional encoding requirements, and that using NoPE for sparse layers and RoPE for linear layers improves performance—opens a broader design space: what other architectural hyperparameters should be attention-type-conditional? Learning rates? Normalization strategies? Initialization schemes?
• Memory-first architecture design. The paper's demonstration that memory, not compute, is the hard barrier for consumer-grade hardware suggests that future efficiency research should prioritize peak memory consumption over FLOP counts as the primary optimization target for long-context deployment.
• Conversion-based training for other architectural families. The Transformer-to-hybrid paradigm could extend to other efficient architectures (state-space models, linear RNNs, mixture-of-experts) that currently require training from scratch.

Research directions that become less attractive (or at least, require stronger justification):

• Pure sparse attention for consumer deployment. The paper's vivid demonstration that Qwen3-8B OOMs at 128K tokens on a 32GB GPU while MiniCPM-SALA reaches 1M tokens makes clear that sparse attention alone cannot solve the memory bottleneck for edge devices. Future work on sparse attention for consumer hardware must either incorporate a linear-attention-like memory reduction or target a different deployment scenario (e.g., datacenter GPUs with abundant VRAM).
• Naive uniform interleaving of attention types. The paper explicitly claims—though without experimental evidence—that the HALO layer selection algorithm yields superior performance relative to uniform interleaving (Section 2.1). If this claim holds under ablation (a key future work direction identified in Section 6.1), it means that simply mixing attention types is not enough; the placement matters, and arbitrary placement may leave performance on the table.

The paper also implicitly reconciles a tension in the prior literature. Several earlier hybrid models (Nemotron-Nano-v2-9B, Falcon-H1R-7B) demonstrated that hybrids could be trained, but did not achieve competitive performance against full-attention baselines—Falcon-H1R-7B's 16.04 long-context average versus MiniCPM-SALA's 38.97 (Table 3) makes this gap stark. The paper's results suggest that specific integration choices (the 1:3 ratio, non-uniform placement, HyPE, progressive context extension, large SFT budget with synthesized long-context data) are what bridge the gap from "feasible" to "competitive." The field now has a blueprint, not just a proof of concept.

---

Follow-Up Research This Work Enables

Ablation of HyPE vs. uniform RoPE to determine whether attention-type-specific positional encoding is load-bearing. The paper attributes MiniCPM-SALA's length extrapolation capability—maintaining an 81.6 RULER score at 2048K tokens despite training only to 520K (Table 4)—to the NoPE configuration in sparse attention layers, reasoning that "the stored KV-Cache does not require combination with positional information, which can otherwise hinder the capture of long-range dependencies" (Section 3.1). This is a testable hypothesis. A controlled experiment would train two variants of MiniCPM-SALA at smaller scale (e.g., 1B parameters) with identical architecture, data, and training recipe, differing only in whether sparse layers use NoPE or RoPE. The primary metrics would be RULER score and NoLiMa accuracy at lengths beyond the training context (e.g., train at 32K, evaluate at 128K and 256K). If the NoPE variant shows substantially better length extrapolation, this validates the paper's mechanistic claim and establishes HyPE as a general principle for hybrid architectures. If the uniform-RoPE variant performs comparably, the length extrapolation is driven by other factors (Lightning Attention's inherent properties, the progressive context extension schedule) and the HyPE design choice is less critical than the paper implies. A strong follow-up would also test the reverse configuration—RoPE on sparse layers and NoPE on linear layers—to map the full design space.

Ablation of sparse attention layers (pure linear baseline) to quantify the contribution of the hybrid design to long-context precision. The paper's central architectural claim is that 25% sparse attention layers compensate for the lossy compression of linear attention, enabling precise long-range retrieval. This claim can be tested by training a pure linear variant—all layers using Lightning Attention, with the same total parameter count (9B), same training pipeline, and same data—and comparing long-context benchmark performance. The paper's Table 3 results already show that MiniCPM-SALA substantially outperforms the Mamba-Transformer hybrid Nemotron-Nano-v2-9B on long-context benchmarks (38.97 vs. 25.12 average), but this is not a controlled comparison because Nemotron-Nano differs in multiple architectural dimensions. A head-to-head MiniCPM-SALA vs. MiniCPM-SALA-minus-sparse comparison would answer: does the NoLiMa gap over Qwen3-8B (42.95 vs. 23.35 at 64K) come primarily from the linear attention layers (which Qwen3-8B lacks) or from the sparse attention layers (which provide precise retrieval)? If the pure linear model approaches MiniCPM-SALA's performance, the sparse layers are unnecessary complexity; if it collapses to near-Qwen3-8B levels, the hybrid division of labor is validated and the field should invest in optimizing layer ratios and placement algorithms.

Evaluation on repository-scale code tasks (SWE-bench) and long-horizon agent benchmarks (AgencyBench) to test whether efficiency gains translate to the motivating applications. The paper's introduction cites "repository-scale code engineering" and "long-horizon agents for complex tasks" as key motivating applications (Section 1), but evaluates only on synthetic long-context benchmarks (RULER, MRCR, NoLiMa). These synthetic benchmarks test retrieval and reasoning over long contrived texts; they do not capture the complexity of real codebases with intricate dependency graphs (SWE-bench, SWE-bench Multimodal) or multi-day agent interactions with evolving state (AgencyBench, GAIA). A strong follow-up would benchmark MiniCPM-SALA on SWE-bench Verified, where the model must identify and fix bugs across multiple files in a repository, with the entire repository loaded into context. The hypothesis is that MiniCPM-SALA's 25% sparse attention layers enable precise code retrieval across files (e.g., finding the definition of a function called on line 15,437) while the linear layers efficiently process the bulk of the codebase, leading to higher resolve rates than a full-attention model that OOMs at full repository scale or requires chunking. Similarly, testing on AgencyBench (Li et al., 2026), which the paper itself cites, would evaluate whether the model can maintain coherent state over 1M-token agent trajectories—a direct test of the paper's claim that "maintaining coherent task states and memory over multi-day human-AI collaborations" (Section 1) is within reach. The key metric would be whether MiniCPM-SALA's task completion rate at 500K+ context lengths exceeds that of full-attention models limited to shorter contexts by OOM.

Memory profiling and hardware requirement specification to make the OOM claims verifiable and the deployment envelope predictable. The paper's most impactful result—that MiniCPM-SALA processes 1M tokens on consumer GPUs where full-attention models fail—is reported without any memory consumption data (Section 6.4). A follow-up study would profile MiniCPM-SALA and Qwen3-8B at sequence lengths from 1K to 1M tokens, reporting peak GPU memory, memory breakdown (KV-cache vs. model weights vs. activations vs. framework overhead), and the inference framework and configuration used. This would serve three purposes: (1) validate that the KV-cache is indeed the dominant memory cost and that linear attention reduces it as claimed; (2) establish the minimum VRAM required to run MiniCPM-SALA at various context lengths, enabling practitioners to predict deployability on their hardware without experimentation; (3) determine whether Qwen3-8B's OOM failures are genuinely architectural or could be mitigated with more memory-efficient inference frameworks (e.g., KV-cache quantization, FlashAttention-3, PagedAttention), which would contextualize the paper's efficiency claims relative to engineering optimizations that do not require architectural change. A strong study would also compare MiniCPM-SALA's memory consumption to other hybrid architectures (Nemotron-Nano-v2-9B, Falcon-H1R-7B) to assess whether the sparse-linear combination is more memory-efficient than Mamba-Transformer or hybrid-head alternatives.

Scaling the hybrid ratio and the layer selection algorithm to determine whether the 1:3 ratio and non-uniform placement are near-optimal or merely sufficient. The paper's choice of 75% linear / 25% sparse is informed by prior work and internal preliminary experiments (Section 2.1), but these experiments are not reported. A systematic follow-up would train multiple variants at a smaller scale (e.g., 1B–3B parameters) with ratios ranging from 100% linear (pure Lightning Attention) through 25% sparse (MiniCPM-SALA's choice) to 50% sparse and 100% sparse (pure InfLLM-V2), using the same training pipeline and data. The evaluation would measure both standard benchmark averages (to assess general capability retention) and long-context benchmarks at multiple lengths (to assess the precision-efficiency tradeoff curve). The key question: is there a "sweet spot" ratio where performance saturates, or does long-context performance improve monotonically with more sparse layers (at the cost of memory)? If the sweet spot is sharp, the 1:3 ratio is load-bearing; if performance is flat across a wide range (e.g., 10%–40% sparse), the architectural choice is less sensitive than the paper implies. A parallel experiment would compare the HALO layer selection algorithm against uniform interleaving at the same 1:3 ratio, measuring the same benchmarks. The paper claims non-uniform placement yields "superior downstream performance" (Section 2.1) without quantification; quantifying this superiority—e.g., a 5-point RULER difference vs. a 0.5-point difference—would inform whether future hybrid architectures need principled layer selection or can use simple interleaving.

Training on long-context reasoning data to test whether the MRCR and NoLiMa performance ceilings can be raised. Section 6.3 identified that MiniCPM-SALA's absolute performance on complex long-context reasoning tasks remains low (10.12 MRCR at 128K-8N, 23.86 NoLiMa at 128K), even though it dominates baselines. The paper's SFT stage includes "specifically synthesized long-context data to enhance the precision of information retrieval and cross-document comprehension" (Section 2.2, SFT paragraph), but the synthesis methodology is not described, and the volume dedicated specifically to complex multi-hop reasoning over long contexts is not reported. A targeted follow-up would generate a large corpus of synthetic multi-hop reasoning data at context lengths from 64K to 512K—questions that require chaining 2–8 facts retrieved from different positions in the context, with controlled difficulty (number of hops, distance between relevant facts, distractors). Fine-tuning MiniCPM-SALA on this corpus and re-evaluating on MRCR and NoLiMa would test whether the current performance ceiling is a fundamental limitation of linear attention's compressed state (i.e., the state cannot preserve enough detail for multi-hop reasoning regardless of training) or a training data shortfall (i.e., the model has the architectural capacity but was not trained on enough complex long-context reasoning examples). If performance improves substantially, the paper's architecture is more capable than the current benchmarks suggest; if it plateaus, the compression ceiling is real and future work should focus on improving linear attention fidelity rather than training recipes.

---

Practical Applications and Downstream Use Cases

On-device document assistants for consumer hardware. The paper's demonstration that MiniCPM-SALA processes 1M tokens on an RTX 5090 (32GB VRAM) with TTFT of 10.8s and end-to-end latency of 25.0s (quantized, Figure 3d) enables a deployment scenario that was previously infeasible: a fully local, privacy-preserving assistant that can ingest entire technical manuals, legal contracts, or research paper collections (hundreds of thousands of words) and answer questions, summarize, or cross-reference—all on a single consumer GPU. In this setting, the key benefit is not just speed (the 3.5× advantage over Qwen3-8B at 256K, Figure 2a) but feasibility: the full-attention alternative cannot run at all beyond ~256K tokens on the same hardware, meaning the choice is between MiniCPM-SALA with 1M-token context and a full-attention model with a 4× smaller context window. For applications like legal document review (where missing a relevant clause buried 200 pages in can change the analysis), the difference between seeing everything and seeing only a quarter of the context is qualitative, not just a latency improvement. The RULER score of 89.4 at 128K and 87.1 at 512K (Table 4) suggests the model retains high retrieval accuracy at these scales—sufficient for many practical document QA tasks.

Cost-efficient long-context API backends with reduced hardware requirements. For organizations deploying long-context LLMs behind an API, the memory wall has direct cost implications: serving a full-attention 8B model at 512K context requires high-VRAM GPUs (A100 80GB, H100 80GB), which are expensive and supply-constrained. MiniCPM-SALA's ability to process 1M tokens on an A6000D (96GB, Figure 2) means that a GPU class typically considered insufficient for ultra-long-context inference can now serve these workloads. Concretely: the A6000D has an MSRP roughly 40–50% lower than an A100 80GB, and is more readily available in workstation configurations. An API provider could deploy MiniCPM-SALA on A6000D instances and offer 1M-token context windows at a per-query cost that undercuts providers using full-attention models on A100s—or offer ultra-long context as a premium feature that was previously technically impossible on non-datacenter hardware. The 75% training cost reduction (with the caveats of Section 6.2) further lowers the barrier to entry: organizations that have already pre-trained a full-attention model can convert to MiniCPM-SALA at a fraction of the cost of training a new long-context model from scratch.

Batch processing pipelines for document corpora and code repositories. In offline batch settings—evaluating a corpus of documents, analyzing a large codebase, or generating summaries for a collection of long texts—the latency of individual queries matters less than total throughput and memory capacity. MiniCPM-SALA's memory efficiency enables higher batch sizes on fixed hardware compared to full-attention models. For example, on an A6000D with 96GB, a full-attention 8B model at 256K context might support a batch size of 1 before OOM (the KV-cache consumes the majority of VRAM), while MiniCPM-SALA's 75% linear layers reduce per-sequence memory, potentially enabling batch size 4–8 for higher total throughput. The paper does not report batch inference results, but the memory architecture directly implies this benefit. A concrete use case: a legal tech company processing 100,000 contracts for due diligence could run MiniCPM-SALA on a single A6000D workstation overnight, whereas a full-attention model would require multi-GPU parallelism or aggressive context truncation, increasing both cost and the risk of missing cross-document references.

---

When to Prefer This Method

The paper does not articulate an explicit decision rule for choosing between MiniCPM-SALA (hybrid sparse-linear) and alternatives like training a larger full-attention model or using a pure linear architecture. The experimental comparisons are against specific named baselines (Qwen3-8B, Nemotron-Nano-v2-9B, etc.) rather than against general paradigms, and the paper does not provide the controlled ablations (pure linear, pure sparse, different ratios) that would enable a principled "prefer A when X, prefer B when Y" framework. The pragmatic implication from the results is that the hybrid architecture is preferable when long-context inference on memory-constrained hardware is the primary requirement and the base capabilities of a ~8B-parameter model are sufficient, but this is an implicit takeaway from the efficiency benchmarks rather than an articulated tradeoff studied in the paper.