MiniCPM4: Ultra-Efficient LLMs on End Devices

ArXiv: 2506.07900

🎯 Pitch

MiniCPM4's sparse attention is trained end-to-end, not bolted on after—enabling a 7× decode speedup for 128K contexts over Qwen3-8B on an edge GPU, while matching its benchmark scores using just 22% of the training tokens.

---

1. Executive Summary

This paper introduces MiniCPM4, a family of ultra-efficient large language models explicitly designed for end-side devices, available in 0.5B and 8B parameter versions with both general and hybrid reasoning variants (MiniCPM4.1). The work achieves efficiency through systematic innovation across four dimensions: a trainable sparse attention mechanism called InfLLM v2 (enabling token-level block selection with semantic kernels rather than training-free heuristics), a data filtering and generation pipeline called UltraClean (using efficient two-stage annealing verification on a nearly-trained LLM to assess data quality without full retraining), and a load-balanced reinforcement learning strategy called chunk-wise rollout (breaking long reasoning trajectories into fixed-length chunks across iterations to prevent GPU idle time from straggler sequences). In a FLOPs-matched comparison on standard knowledge and reasoning benchmarks, MiniCPM4-8B achieves comparable performance to Qwen3-8B while using only 22% of the training data (8T tokens versus 36T), and on end-side devices demonstrates a 7× decoding speedup over Qwen3-8B when processing 128K-length documents on Jetson AGX Orin — establishing that trainable sparse attention combined with aggressive data curation can match state-of-the-art dense models at a fraction of the pretraining cost, but only when the sparse attention mechanisms are trained end-to-end rather than applied post-hoc.

2. Context and Motivation

The Core Problem: LLMs Are Stuck in the Cloud

The fundamental tension this paper tackles is straightforward: large language models keep getting bigger, but the devices people actually use keep getting smaller. Since the release of GPT-3 (Brown et al., 2020), the dominant paradigm has been to scale models up — more parameters, more training data, more compute — and deploy them behind API interfaces on cloud servers. This works well for applications with reliable internet connectivity and generous latency budgets, but it completely forecloses a vast range of deployment scenarios that the paper's authors argue are both practically important and technically underserved.

The paper opens with an observation that has become increasingly hard to ignore: as models grow from billions to hundreds of billions of parameters, the computational requirements grow even faster. The scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) has established that performance improves predictably with compute investment, but this very predictability creates a trap — the field optimizes for benchmark scores rather than deployability, producing models that are technically impressive but practically unusable on the hardware available to most users. A model that requires multiple A100 GPUs simply to run inference cannot be deployed on a laptop, a phone, or an embedded system.

This gap matters for several concrete reasons that the paper highlights in Section 1:

• Privacy and data sovereignty: When users have strict confidentiality requirements — medical records, legal documents, proprietary research — uploading data to a cloud API is unacceptable. The model must run locally, which means it must fit within the computational envelope of the device the user actually has.
• Latency and connectivity: End-side applications in robotics, automotive systems, and mobile assistants cannot tolerate the round-trip latency of cloud inference, nor can they assume persistent network connectivity. A model that stalls when the WiFi drops is useless in these contexts.
• Cost at scale: Cloud inference incurs per-token costs that become substantial for high-volume applications like document processing, survey generation, or agent-based tool use. Running inference locally eliminates these recurring costs, shifting the economic calculus toward one-time hardware investment.
• Energy efficiency: Datacenter inference consumes enormous power; moving computation to the edge distributes this load and can reduce total energy consumption when the alternative is transmitting large amounts of data over networks.

Why This Problem Has Resisted Solution

Making LLMs efficient enough for end-side deployment is not simply a matter of "make the model smaller." The paper identifies four interconnected bottlenecks that must be addressed simultaneously, and prior work has typically addressed them in isolation, achieving gains in one dimension at the expense of others.

Bottleneck 1: Attention mechanisms scale quadratically with sequence length. The standard self-attention operation (Vaswani et al., 2017) requires each token to attend to all preceding tokens, producing $O(l^2)$ computational and memory complexity for a sequence of length $l$. This is merely inconvenient for short sequences but becomes prohibitive for the long-context processing that modern applications demand — 128K-token documents, multi-turn agent interactions, and deep reasoning chains that routinely exceed 16K tokens. The paper points out that "the computational and memory demands of self-attention mechanisms pose significant challenges for efficiently processing lengthy documents on end-side devices" (Section 1). On a Jetson AGX Orin — a representative end-side chip used in automotive and robotics applications — a dense 8B-parameter model processing a 128K document is simply not viable at interactive speeds.

Bottleneck 2: Training data quality determines capability density, but filtering it efficiently is unsolved. The relationship between data quality and model performance is well-established: higher-quality data yields better models for a given parameter count and training budget (Xiao et al., 2024a). However, as the paper notes in Section 2.2, current data filtering approaches face two major challenges: (1) there is no efficient way to verify whether a filtering strategy actually improves model quality without training a full model from scratch (requiring ~1,200 GPU-hours for a 1B-parameter model on 100B tokens, per Table 1), and (2) the selection of positive seed samples for training data classifiers relies heavily on human intuition rather than empirical validation, introducing subjective biases. The consequence is that most open-source models compensate for mediocre data quality by training on enormous volumes — Qwen3-8B uses 36 trillion tokens — which is exactly the opposite of what you want for an efficient model that should train on as little data as possible.

Bottleneck 3: Reinforcement learning for reasoning suffers from load imbalance. The RL pipelines that produce strong reasoning capabilities (as in DeepSeek-R1 and OpenAI o1) require generating long chain-of-thought trajectories during the rollout phase. But these trajectories have wildly variable lengths — a simple arithmetic problem might finish in 100 tokens, while a competition math problem might require 8,000. In a synchronous training setup, the entire GPU batch waits for the longest trajectory to complete, leaving most compute units idle. The paper identifies this as a critical efficiency bottleneck in Section 3.2, noting that "directly applying RL to an end-side base model often leads to unstable training and slow convergence." Existing solutions either pad everything to maximum length (wasting compute) or truncate long trajectories (losing training signal).

Bottleneck 4: Inference systems for end-side devices are fragmented and under-optimized. Even if you have an efficient model architecture, actually running it efficiently on diverse end-side hardware (NVIDIA Jetson, Qualcomm, MediaTek, Rockchip) requires platform-specific optimization. The paper observes that "the fragmentation of end-side chips presents another significant hurdle. This fragmentation necessitates adapting models to multiple platforms and chip types for each new model release, leading to complex adaptation and deployment" (Section 4.2). Each platform has its own inference framework, its own quantization scheme, its own memory constraints. Without a unified deployment system, the engineering effort of porting a model to all platforms becomes prohibitive.

Where Prior Approaches Fall Short

The paper positions its contributions against several lines of prior work, each of which addresses one piece of the puzzle but leaves the others unsolved.

Training-free sparse attention is fast but inaccurate. A substantial body of work has explored sparse attention mechanisms that dynamically select relevant context tokens without modifying model weights — StreamingLLM (Xiao et al., 2023), MInference (Jiang et al., 2024), XAttention (Xu et al., 2025), and SpargeAttn (Zhang et al., 2025a). The paper's critique is pointed: "These models can only be applied in prefilling acceleration due to their unsatisfactory sparsity" (Section 2.1). The problem is that training-free methods use heuristic relevance scoring — often based on simple dot products with representative tokens — that degrades when the attention pattern becomes sparse enough to matter for decoding speed. If you need to attend to 30-50% of tokens to maintain accuracy, you haven't actually solved the quadratic scaling problem.

MoBA's query blocking prevents decoding acceleration. Lu et al. (2025) propose Mixture of Block Attention (MoBA), which applies sparse attention during pretraining but groups query tokens into blocks that share the same key-value context. The paper identifies a critical flaw: "MoBA utilizes the design of query blocks, which prevents it from achieving acceleration during the decoding phase" (Section 2.1). During decoding, tokens are generated one at a time — you can't form a block from a single token — so the training-time block structure creates a training-inference mismatch that degrades performance during autoregressive generation.

NSA introduces parameter overhead and triple key-value storage. Yuan et al. (2025) propose Native Sparse Attention (NSA), which uses three separate attention components (compressed attention, selected attention, and sliding window attention) to capture long-distance dependencies. The paper's criticism is practical: "The three attention components introduce additional parameters, which will lead to increased computational overhead for short sequences and threefold key-value storage costs for pre-training" (Section 2.1). For an efficiency-focused model, adding parameters and memory overhead defeats the purpose — short sequences (which are the common case in many applications) become slower than dense attention.

Existing data filtering lacks efficient verification. The paper describes two common approaches to model-based data filtering (illustrated in Figure 3). In approach (a), classifiers are trained without verification — the quality of the filtered data is never directly measured, only inferred from downstream model performance after a full training run. In approach (b), verification happens through full-scale LLM training, which the paper quantifies: "training 100B tokens on an LLM with 1B parameters requires approximately 1,200 GPU hours, equivalent to running 64 GPUs continuously for nearly 19 hours" (Section 2.2.1). This cost makes iterative refinement of filtering strategies impractical. The consequence is that seed data selection remains dominated by heuristics — LLM scoring above some threshold, manual curation, or simple domain matching — without rigorous evidence that the chosen seeds actually produce better models.

Standard RL rollouts waste compute on length imbalance. In a typical GRPO-style RL setup (as used in DeepSeek-R1 and subsequent work), the policy model generates complete responses for each prompt in the batch, then rewards are computed and gradients are applied. The paper points out that this is computationally wasteful because "the rollout process of RL usually suffers from the unbalanced load challenge, which can lead to very inefficient computations" (Section 3). A batch of 256 prompts might contain a mix of short problems (generating 500 tokens) and long problems (generating 8,000 tokens), and the entire batch waits for the longest one. Prior work has not directly addressed this load balancing problem in the context of LLM reinforcement learning.

Standard quantization methods miss prefix-related artifacts. GPTQ (Frantar et al., 2023) is the dominant post-training quantization method, computing a Hessian matrix from calibration data to guide weight rounding. But the paper identifies a specific, previously undocumented failure mode: "when computing the covariance matrix for down-projection layers, particularly in those deeper Transformer blocks, the beginning of sentence token and some initial tokens consistently introduce significant statistical bias" (Section 4.1.2). These initial tokens exhibit activation magnitudes 10× larger than subsequent tokens, dominating the Hessian computation and producing suboptimal quantization parameters. Prior methods like PrefixQuant (Chen et al., 2024) addressed activation outliers at initial positions but did not account for how these same outliers corrupt the weight quantization calibration process.

How This Paper Positions Itself

The paper frames its contribution not as a single breakthrough method but as a systematic integration of efficiency innovations across the full LLM pipeline. The key claim is that only by optimizing simultaneously across architecture, data, training algorithms, and inference systems can you achieve the combination of competitive accuracy and dramatic speedup that makes end-side deployment viable.

The paper's approach to sparse attention is the clearest example of this philosophy. Rather than accepting the training-free vs. trainable dichotomy, the authors propose InfLLM v2 as a trainable sparse attention that introduces no additional parameters for attention output and degrades gracefully to dense attention for short sequences (Section 2.1). This directly addresses the critiques of prior work: it supports both prefilling and decoding acceleration because queries operate at token granularity while key-values operate at block granularity (avoiding MoBA's decoding problem); it uses only standard key-value representations with mean pooling for block relevance scoring (avoiding NSA's parameter and storage overhead); and it is trained end-to-end, allowing the model to learn which attention patterns are sparsifiable rather than relying on hand-crafted heuristics.

The paper positions its data filtering work as a practical solution to the verification bottleneck. The "efficient verification strategy" — fine-tuning a nearly-trained model on candidate data during the annealing phase — reduces verification cost from 1,200 GPU-hours to approximately 110 GPU-hours (Table 1), a 10× improvement. This is not presented as a theoretical advance in data quality assessment but as an engineering contribution that makes iterative data curation practically feasible for teams without industrial-scale compute budgets.

The chunk-wise rollout strategy for RL is positioned as a direct response to the load imbalance problem that prior work ignored. By breaking trajectories into fixed-length chunks and resuming incomplete ones in subsequent iterations, the method ensures that GPU utilization remains high regardless of trajectory length variance. The paper is explicit that this introduces new stability challenges (distributional shift from partially-sampled trajectories, off-policy data from previous model versions), and the bulk of Section 3.2.3 is devoted to the stabilization techniques (chunk-level importance sampling, dual-clip, KL regularization, garble filtering) needed to make it work reliably.

Finally, the paper positions its inference system work — CPM.cu and ArkInfer — as addressing the deployment fragmentation that makes end-side LLMs impractical regardless of model quality. The approach is pragmatic: build a lightweight CUDA framework for NVIDIA chips (where most development happens), then layer a cross-platform abstraction (ArkInfer) that adapts to diverse backends through standardized APIs. This two-tier approach acknowledges that performance matters most on common platforms while ensuring that portability doesn't become an afterthought.

A subtle but important aspect of the paper's positioning is its relationship to knowledge distillation. Several of the baseline models it compares against — Qwen3, Llama3.2, Gemma3 — employ distillation from larger teacher models during training. The paper explicitly notes that MiniCPM4 uses only ground-truth supervision signals (Section 5.2) and still achieves competitive or superior performance. This is significant because distillation itself requires deploying and running the larger teacher model, which consumes substantial compute that the paper's efficiency-focused approach is trying to avoid. The implication is that high-quality data curation can substitute for expensive teacher supervision, which aligns with the paper's broader thesis that data quality is the most leverageable efficiency lever.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

MiniCPM4 is a family of small language models (0.5B and 8B parameters) designed from the ground up to run efficiently on devices like laptops, phones, and embedded systems rather than on cloud servers. The system solves the problem that standard large language models are too computationally expensive and memory-hungry for end-side hardware by rethinking four interconnected parts of the model pipeline simultaneously: the attention mechanism that processes long sequences, the data used for training, the algorithms that teach the model to reason, and the inference software that actually runs the model on diverse hardware.

3.2 Big-Picture Architecture (Diagram in Words)

The MiniCPM4 system is not a single model but a complete pipeline with five major stages:

1. Pre-training data pipeline (UltraClean): Takes raw web data (FineWeb, Chinese FineWeb) and produces a high-quality, filtered dataset (UltraFineWeb) using a fastText classifier trained on empirically-verified seed data. Also generates synthetic reasoning-intensive data for mathematics and code. The output is 8.3 trillion tokens of curated pre-training data.

2. Architecture with trainable sparse attention (InfLLM v2): A modified Transformer where each attention layer selects only the most relevant blocks of key-value pairs for each query token, using learned semantic kernels rather than heuristic scoring. This reduces the quadratic attention cost to approximately linear in practice, operating at ~95% sparsity on 128K sequences.

3. Pre-training with hyperparameter search (ModelTunnel v2): Uses small-scale experiments on million-parameter models with $\mu$P parameterization to find optimal learning rates, batch sizes, and initialization schemes, then transfers these to the full 8B model. Training uses multi-token prediction objectives and FP8 mixed-precision.

4. Post-training (UltraChat v2 + chunk-wise RL): First fine-tunes the pre-trained model on a diverse supervised dataset covering knowledge, reasoning, instruction following, long context, and tool use. Then applies reinforcement learning with a chunk-wise rollout strategy that breaks long reasoning trajectories into fixed-length segments to prevent GPU idle time, using GRPO with stabilization techniques.

5. Inference deployment (CPM.cu + ArkInfer): A lightweight CUDA inference framework optimized for NVIDIA edge chips, integrating the sparse attention kernel, speculative decoding with frequency-ranked vocabulary pruning (FR-Spec), and prefix-aware quantization (P-GPTQ). A cross-platform layer (ArkInfer) adapts the model to non-NVIDIA hardware through standardized backend interfaces.

The information flow: raw web data → UltraClean filtering → 8.3T curated tokens → pre-training with InfLLM v2 architecture → SFT on UltraChat v2 → RL with chunk-wise rollout for reasoning → quantization + deployment through CPM.cu or ArkInfer.

3.3 Roadmap for the Deep Dive

• First, InfLLM v2 (Section 2.1): The trainable sparse attention mechanism because it is the architectural foundation that makes long-context processing efficient and is tightly coupled with every other component (training, inference, deployment).
• Second, UltraClean data pipeline (Section 2.2): The data filtering and generation strategy because data quality is what enables the model to match larger models with fewer training tokens, and the efficient verification strategy is a key practical innovation.
• Third, ModelTunnel v2 (Section 2.3): The hyperparameter search methodology and pre-training engineering because it determines how the architecture and data are combined into an effective training run.
• Fourth, post-training pipeline (Section 3): UltraChat v2 for SFT, chunk-wise rollout for RL, and BitCPM4 for ternary quantization, because these build on the pre-trained model to add instruction following, reasoning, and extreme compression capabilities.
• Fifth, inference and deployment systems (Section 4): CPM.cu for efficient CUDA inference, FR-Spec for accelerated speculative decoding, P-GPTQ for prefix-aware quantization, and ArkInfer for cross-platform deployment, because these are what make the efficiency gains real on actual hardware.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems engineering paper whose core thesis is that simultaneously optimizing architecture, data curation, training algorithms, and inference systems enables small models to match much larger models at a fraction of the computational cost, and that trainable sparse attention — rather than training-free heuristics — is the key architectural enabler for efficient long-context processing on end-side devices.

---

InfLLM v2: Trainable Sparse Attention for Both Prefilling and Decoding

The standard Transformer attention mechanism computes a weighted sum over all previous tokens for each position in the sequence. For a sequence of length $l$, each token performs $l$ dot-product computations and $l$ memory accesses, producing $O(l^2)$ total complexity. This is manageable for short sequences but becomes prohibitive for the 128K-token contexts that modern applications require. InfLLM v2 reduces this by having each query token attend to only a small, adaptively selected subset of the available context tokens, organized into coarse-grained blocks.

Block-level key-value partitioning. The key innovation of InfLLM v2 over its predecessor (InfLLM) is how it computes relevance scores between queries and context blocks. Both approaches partition the key-value cache into equal-sized blocks of $m$ tokens each. Formally, given key and value sequences $\mathbf{K} = \{\mathbf{k}_1, \ldots, \mathbf{k}_l\}$ and $\mathbf{V} = \{\mathbf{v}_1, \ldots, \mathbf{v}_l\}$, the cache is partitioned into blocks:

$\mathcal{B} = \{\mathbf{B}_0, \ldots, \mathbf{B}_{\lfloor l/m \rfloor - 1}\}, \quad \mathbf{B}_j = (\mathbf{K}_{jm:(j+1)m}, \mathbf{V}_{jm:(j+1)m})$

where $m$ is the block size (the number of key-value pairs in each block), and $\lfloor l/m \rfloor$ is the total number of blocks.

What this does: Instead of storing key-value pairs as individual vectors, they are grouped into contiguous chunks. This is critical because modern GPU memory systems are optimized for accessing contiguous regions — reading one block of $m$ tokens is much faster than reading $m$ individual tokens scattered across memory.

Why this granularity: Token-level sparse attention would require computing and sorting relevance scores for every individual key vector, which is itself $O(l^2)$ in the scoring step alone — defeating the purpose of sparsification. Block-level selection amortizes the relevance computation cost: you compute one score per block (of $m$ tokens) rather than per token, reducing the selection overhead by a factor of $m$.

---

Semantic kernels for block relevance scoring. The original InfLLM selected a few "representative tokens" from each block and used their dot products with the query as the block relevance score. This required token-level memory access to extract those representatives, creating an efficiency bottleneck. InfLLM v2 replaces this with semantic kernels — fine-grained overlapping spans of key vectors that capture local semantics without requiring token-level selection.

The input sequence is partitioned at a finer granularity than the blocks. Let $p$ be the kernel size (number of tokens per kernel) and $s$ be the stride (number of tokens between kernel starts). The key sequence is partitioned into overlapping kernels:

$\mathcal{S} = \{\mathbf{S}_0, \ldots, \mathbf{S}_{\lfloor l/s \rfloor - 1}\}, \quad \mathbf{S}_{\hat{j}} = \mathbf{K}_{\hat{j}s : \hat{j}s + p}$

where each kernel $\mathbf{S}_{\hat{j}}$ covers $p$ consecutive key vectors starting at position $\hat{j}s$, and adjacent kernels overlap by $p - s$ tokens. This overlap ensures that no semantic span in the input falls on a kernel boundary — every subsequence of length $p$ is contained within at least one complete kernel.

What this computes: For each query token $\mathbf{q}_i$, InfLLM v2 computes a relevance score against each semantic kernel using the mean-pooled representation:

$r_{\text{kernel}}(\mathbf{q}_i, \mathbf{S}_{\hat{j}}) = \text{softmax}(\mathbf{q}_i \cdot \text{Mean}(\mathbf{K}_{\hat{j}s : \hat{j}s + p}))$

The softmax is applied across all kernels for a given query, producing a normalized distribution over kernel positions. The block relevance score is then the maximum kernel relevance among all kernels that overlap with that block:

$r_{\text{block}}(\mathbf{q}_i, \mathbf{B}_j) = \max r_{\text{kernel}}(\mathbf{q}_i, \mathbf{S}_{\hat{j}}), \quad \forall \mathbf{S}_{\hat{j}} \in \mathcal{S} \text{ where } \mathbf{B}_j \cap \mathbf{S}_{\hat{j}} \neq \emptyset$

Why mean pooling: The selection operation (Top-K over block scores) is non-differentiable, so the kernel representations cannot be directly optimized through the sparse attention computation. Mean pooling is a parameter-free operation that keeps the kernel representation in the same vector space as the original key vectors — specifically, $\text{Mean}(\mathbf{K}_{a:b})$ is just the average of key vectors in that span. This means that optimizing the token-level key vectors during training indirectly optimizes the kernel representations, even though no gradient flows through the Top-K selection itself.

Why overlapping kernels: If kernels were non-overlapping (stride = kernel size), a semantically important span that straddles two kernels would be split across them, and neither kernel would fully capture it. The overlap ensures that every $p$-token span falls entirely within at least one kernel. The recommended configuration sets kernel size $p = 32$ and stride $s = 16$, giving 50% overlap — each token appears in two adjacent kernels.

---

Query group sharing for memory efficiency. Modern LLMs use Grouped Query Attention (GQA), where multiple query heads share a single set of key-value heads. InfLLM v2 exploits this by requiring all query heads within the same group to attend to the same Top-K blocks. This means the relevance scores from individual heads are averaged within the group:

$r_{\text{group}}(\mathbf{Q}_{\text{group}}, \mathbf{S}_{\hat{j}}) = \frac{1}{|\text{group}|} \sum_{\mathbf{q} \in \text{group}} r_{\text{kernel}}(\mathbf{q}, \mathbf{S}_{\hat{j}})$

and the Top-K selection is performed once per group rather than once per head.

Why this matters: In standard GQA, even though multiple query heads share the same key-value tensors, they can theoretically attend to different positions (the attention weights are computed independently per head). This means the memory access pattern is still per-head — each head fetches its own subset of key-value pairs. By forcing heads in a group to share the same selected blocks, InfLLM v2 ensures that the key-value data for a group is loaded from memory exactly once and reused across all heads in that group. The paper notes that a query group must contain at least 16 heads "to ensure hardware is fully utilized" (Section 2.1.3), matching the requirements of GPU tensor core matrix multiplication instructions.

---

Top-K implementation and the LSE bottleneck. The Top-K selection requires three steps: (1) compute query-kernel dot products and apply softmax, (2) aggregate scores across the query group dimension, and (3) select the K blocks with the highest aggregated scores. The bottleneck is step (1), specifically computing the softmax normalization, which requires:

• Pass 1: Compute the LogSumExp (LSE) across all kernels for each query group — this requires iterating over all kernels to find the maximum dot product and sum of exponentials.
• Pass 2: Use the LSE to normalize each kernel score and compute the final softmax values.

Each pass requires reading all kernel representations from memory, making this an $O(l/s)$ memory-bound operation where $l/s$ is the number of semantic kernels (which grows with sequence length).

Efficient LSE approximation. To reduce this cost, InfLLM v2 introduces coarse-grained kernels with size $s_c \gg s$. Instead of computing exact LSE over all fine-grained kernels, it computes LSE using only the coarse-grained kernels (of which there are $s/s_c$ times fewer). The coarse-grained LSE serves as an approximation for the fine-grained LSE, reducing the memory access and computation in the first pass by a factor of $s_c/s$.

Why this works: The LSE is a smooth function of its inputs — replacing some fine-grained kernels with coarser ones changes the denominator of the softmax slightly but preserves the relative ordering of block scores. Since we only care about which blocks have the highest scores (for Top-K selection), not their exact values, this approximation has minimal impact on selection accuracy while substantially reducing cost.

---

Initial and local tokens always selected. InfLLM v2 ensures that two special token groups are always attended to, regardless of relevance scores. The initial blocks $\mathbf{B}_0$ (containing the first tokens of the sequence) are assigned infinite relevance scores, forcing their selection. Similarly, blocks within a local window around the current token position are always selected. The paper justifies this based on prior findings that initial tokens act as "attention sinks" (Xiao et al., 2023) and that local context is disproportionately important for coherence.

When the total sequence length is shorter than $k \times m$ (the total tokens in $k$ selected blocks), InfLLM v2 degenerates exactly to dense attention — every token attends to every other token. This is a crucial property: it means the sparse attention mechanism imposes no degradation for short sequences, which is important because many real-world inputs are short even if the model supports long contexts.

---

Complexity analysis. The computational and memory access costs break down as follows:

• Stage 1 (relevance scoring): Each query token computes dot products with $\lfloor l/s \rfloor$ semantic kernels. With the LSE approximation using coarse kernels, this reduces to $\lfloor l/s_c \rfloor$ for the first pass and $\lfloor l/s \rfloor$ for the second pass. This stage remains $O(l^2)$ in the worst case because the number of kernels grows with $l$.

• Stage 2 (attention computation): Each query token attends to $k \times m$ tokens total (across the $k$ selected blocks). Since $k$ and $m$ are constants independent of $l$, this stage is $O(l)$ — linear in sequence length.

• Overall: The $O(l^2)$ factor in stage 1 dominates asymptotically, but with a much smaller constant than dense attention because kernel-level operations are $1/s$ of token-level operations. The paper reports that when $l \gg m$, the total overhead is reduced to approximately $1/s$ of dense attention. For the recommended $s = 16$, this means roughly 6.25% of the computational cost of dense attention for the relevance scoring, plus the constant cost of attending to $k \times m$ tokens.

On 128K sequences, the paper states that InfLLM v2 achieves 95% sparsity — each token attends to approximately 6K context tokens rather than 128K.

---

Hyperparameter configuration. The recommended settings balance the tradeoff between selection accuracy and computational cost:

• Block size $m$: Not explicitly specified but implicitly determined by hardware constraints. The paper states that "the block size $m$ typically being a relatively large value" (Section 2.1.2).
• Semantic kernel size $p = 32$: Smaller kernels would give more precise relevance scoring but increase the number of kernels (and thus computational cost). Larger kernels would be cheaper but risk information loss from mean pooling over diverse tokens.
• Kernel stride $s = 16$: Gives 50% overlap between adjacent kernels. This was chosen "to achieve a good balance between effectiveness and efficiency" (Section 2.1.3).
• Query group minimum size: 16 heads, matching tensor core matrix multiply-accumulate instruction requirements.

---

Training-time considerations. The Top-K selection operation is non-differentiable — the model cannot learn to improve which blocks are selected because gradients cannot flow through the argmax. This is a fundamental challenge for any sparse attention mechanism. InfLLM v2 addresses this indirectly:

1. Mean pooling uses the same key vectors that participate in the attention computation. During training, the attention computation in stage 2 produces gradients that flow back through the key vectors. Since the same key vectors are used (via mean pooling) in stage 1, optimizing for better attention in stage 2 indirectly optimizes the representations used for block selection.

2. The model learns to structure its key representations so that relevant blocks score higher. Even though the model cannot directly learn "select this block," it can learn to produce key vectors that, when mean-pooled, yield high dot products with query vectors for tokens that should attend to those blocks.

3. No additional parameters are introduced for the selection mechanism. This contrasts with NSA (Yuan et al., 2025), which adds compressed attention parameters that are optimized through a separate loss term. InfLLM v2's parameter-free design means the model's representational capacity is fully devoted to the attention computation itself, which the paper argues is more parameter-efficient.

---

Design principles summary. Section 2.1.3 codifies several architectural decisions as explicit design principles:

• Different granularity for queries vs. key-values: Queries operate at token level (each token selects its own blocks), while key-values operate at block level (selected as contiguous groups). This enables decoding acceleration because decoding generates tokens one at a time — block-level query grouping (as in MoBA) would create a training-inference mismatch during autoregressive generation.

• Trainable context selection via indirect optimization: The paper explicitly acknowledges that Top-K is non-differentiable and that mean pooling is the mechanism for indirect optimization through key vector gradients. This is described as a deliberate choice to keep the architecture simple while still allowing the model to learn sparse attention patterns.

• No overhead for short sequences: When the input is shorter than $k \times m$ tokens, InfLLM v2 is mathematically identical to dense attention. This property is important for practical deployment because models with sparse attention should not be worse than dense models on short inputs, which are the common case in many applications.

---

UltraClean: High-Quality Pre-Training Data Filtering and Generation

The UltraClean pipeline addresses a fundamental efficiency problem: given a fixed training budget of ~8 trillion tokens, how do you select the subset of available web data that will produce the best model? The core insight is that verifying data quality typically requires training a full model from scratch, which costs thousands of GPU-hours and makes iterative refinement impractical. UltraClean's efficient verification strategy reduces this cost by a factor of ~10, enabling rapid experimentation with data filtering strategies.

---

Efficient verification via two-stage annealing. The standard approach to verifying data quality is straightforward but expensive: train a model on the candidate data from scratch and measure downstream performance. For a 1B-parameter model on 100B tokens, this requires approximately 1,200 GPU-hours (64 GPUs for ~19 hours). The paper's efficient verification strategy exploits the observation that a nearly-trained model is highly sensitive to data quality changes during the final annealing phase.

The procedure works as follows:

1. Pre-train a 1B-parameter model to near-convergence using the WSD (warmup-stable-decay) learning rate scheduler, covering 1.1 trillion tokens total: 1T tokens in the stable phase and 0.1T tokens in the initial decay phase.

2. Introduce candidate data during a second annealing phase: Take the nearly-trained model and fine-tune it on just 10B tokens, where 30% of the data comes from the candidate corpus being evaluated and the remaining 70% follows the default mixed-data ratio. This fine-tuning uses the WSD decay schedule to make the model particularly sensitive to data quality changes.

3. Measure the improvement in validation loss and downstream metrics relative to a baseline that uses the default data mix for the second annealing phase. The performance delta directly reflects the quality of the candidate data.

Why this works: During the decay phase of WSD training, the learning rate is decreasing, and the model is settling into a local minimum. Introducing high-quality data at this stage provides a stronger training signal that pulls the model toward a better minimum; low-quality data either provides weak signal or actively pulls the model away. The effect is amplified because the model's parameters are already near-optimal, so even small changes in the data distribution produce measurable differences in loss. This is analogous to how fine-tuning a pre-trained model on a small high-quality dataset can produce large improvements, while training from scratch on the same dataset would not.

Cost comparison (Table 1):

The efficient strategy uses approximately 9% of the cost of a 100B-token from-scratch verification. The paper uses a 1B-parameter proxy model for verification, not the full 8B model — the assumption is that data quality rankings transfer across model scales, which is consistent with the $\mu$P framework used throughout.

---

Classifier training with empirically-verified seeds. The paper's data classifier is a fastText model (a simple but fast linear classifier over bag-of-n-grams features) that assigns a quality score to each document. Training this classifier requires positive examples (high-quality documents) and negative examples (low-quality documents). The key innovation is how the positive examples are selected.

The core hypothesis: "High-quality seed data that can improve LLM performance should also be beneficial for training classifiers capable of identifying high-quality training samples" (Section 2.2.1). This means you don't need human judgment to select seeds — you can empirically verify which candidate seeds actually improve model performance using the efficient verification strategy, then use those verified seeds to train the classifier.

The process:

1. Start with a pool of candidate seed data from multiple sources: documents with LLM scores above 4 (on a 1-5 quality scale), instruction-formatted datasets (OH-2.5, ELI5), real-world educational materials, LLM-synthesized textbook-style content, and curated high-quality web data.

2. For each candidate seed, run the efficient verification strategy — fine-tune the 1B proxy model on 10B tokens with 30% seed data, measure the delta in loss and benchmark scores.

3. Select seeds that produce statistically significant improvements as positive training examples for the classifier.

4. Construct negative examples by randomly sampling from diverse raw web corpora: FineWeb, C4, Dolma, The Pile, and RedPajama for English; CCI3, ChineseWebtext, and other mainstream corpora for Chinese. The paper notes that "diversified data sources for negative samples significantly improves classifier generalization and cross-domain adaptability" (Section 2.2.1).

5. Iteratively refine: After initial training, use the current classifier to score unlabeled data. Documents classified as high-quality become additional positive seeds for the next round; documents classified as low-quality become additional negative seeds. This bootstrapping process progressively improves classifier precision.

Why fastText over LLM-based classifiers: The choice is purely practical. The paper quantifies the difference: "processing 15 trillion tokens using an LLM-based classifier requires approximately 6,000 hours on GPUs, whereas fastText can complete the same task on a non-GPU server using 80 CPUs in under 1,000 hours" (Section 2.2.1). For the scale of data being processed (filtering from web-scale corpora down to 8.3T tokens), LLM-based classification would dominate the total computational budget. fastText is fast enough that classification cost is negligible compared to the model training it enables.

fastText hyperparameters (Section 2.2.1):

• Vector dimension: 256
• Learning rate: 0.1
• Maximum n-gram length: 3
• Minimum word frequency: 5
• Training epochs: 3
• Classification threshold: 0.5 (default, no tuning)

Preprocessing before classification:

• Removal of redundant blank lines and excessive whitespace
• Stripping of diacritics
• Normalization of English text to lowercase
• Tokenization using the DeepSeek-V2 tokenizer (which outperforms space-based tokenization for English and Jieba for Chinese)
• Preservation of structural tokens (\n, \t, \r)

Distributed processing is done on a Spark cluster for scalability.

---

Results of UltraClean filtering (Table 2). The paper evaluates the filtered data by training 1.2B-parameter models (MiniCPM-1.2B architecture with MiniCPM3-4B tokenizer) on approximately 100B tokens each and evaluating on standard benchmarks using Lighteval in a zero-shot setting.

English results (UltraFineWeb-en vs. baselines):

• Average score: 45.89, compared to 42.28 for FineWeb (+3.61 points) and 44.56 for FineWeb-edu (+1.33 points)
• Largest gains on ARC-C (+10.50 over FineWeb), ARC-E (+11.44), and OpenBookQA (+4.00)
• Small regression on HellaSwag (-0.15 vs. FineWeb), though UltraFineWeb still outperforms FineWeb-edu on this metric (+0.59)

Chinese results (UltraFineWeb-zh vs. baselines):

• Average: 35.16, compared to 33.18 for Chinese-FineWeb (+1.98) and 34.55 for Chinese-FineWeb-edu-v2 (+0.61)
• Largest gain on CMMLU (+3.65 over Chinese-FineWeb)

The key takeaway is that UltraClean filtering consistently improves over both raw web data and the existing FineWeb-edu filtered data, with gains concentrated on knowledge-intensive tasks (MMLU, ARC, C-Eval, CMMLU) rather than surface-level pattern matching tasks (HellaSwag). This aligns with the design goal of filtering for knowledge density rather than just fluency.

---

Reasoning-intensive data generation. Beyond filtering existing web data, the paper synthesizes new pre-training data specifically designed to improve reasoning capabilities. The motivation is that even high-quality web data lacks the structured, step-by-step reasoning chains that models need to learn mathematical and logical inference.

The generation pipeline has two components:

1. Seed data selection: Use the UltraFineWeb classifier to identify knowledge-intensive, logically complete content from web corpora as general-domain seeds. Additionally, manually curate domain-specific seeds from open-source textbooks, QA materials, and academic content in mathematics, programming, and natural sciences.

2. Structured generation with two paradigms:

   • Textbook paradigm: For each knowledge point, generate layered content consisting of: (a) a clear statement of the concept, (b) multi-round explanations at increasing depth, (c) a summary, and (d) practice questions. This systematically builds from simple to complex, enforcing hierarchical knowledge organization.

   • Forum paradigm: Simulate authentic discussions by generating multi-turn QA exchanges and viewpoint debates around a central topic. This introduces diverse reasoning paths and alternative problem-solving approaches, preventing the model from learning a single "school solution" style.

The data generation uses open-source LLMs under 10B parameters (to keep costs manageable) and operates iteratively: generated data is fed back into the seed pool for subsequent rounds of evolution, progressively enriching the training corpus with deeper reasoning examples.

Why this matters for efficiency: Reasoning ability is the capability that scales most strongly with model size in standard scaling laws — small models typically perform dramatically worse than large models on reasoning benchmarks. By synthesizing high-quality reasoning-intensive pre-training data, the paper aims to shift this tradeoff, giving a small model access to the kind of structured reasoning examples that are naturally rare in web corpora. This is a direct application of the "data quality over data quantity" principle.

---

Discussion of limitations (Section 2.2.3). The paper is candid about open challenges: the filtering and generation pipelines still rely on human-designed heuristics for seed selection; balancing corpus diversity with task relevance in evolutionary generation remains difficult (risk of semantic mode collapse); and extending the approach to multilingual, cross-task, and multimodal settings is an open problem. These are presented as directions for future work rather than solved problems.

---

ModelTunnel v2: Efficient Pre-Training Strategy Search

Training a large language model requires choosing numerous hyperparameters — learning rate, batch size, model width and depth, initialization scheme, data mixture ratios — that dramatically affect final performance. The standard approach (grid search) is infeasible at the 8B scale, where a single training run consumes hundreds of thousands of GPU-hours. ModelTunnel v2 is a methodology for determining optimal hyperparameters using small-scale experiments and then transferring them to the full model, based on two key innovations over the original ModelTunnel (Hu et al., 2024).

---

ScalingBench: A better performance indicator. In the original ModelTunnel, the performance indicator used to compare hyperparameter configurations was the language modeling loss on open-source pre-training corpora. The assumption was that lower loss implies better downstream performance. The paper identifies a flaw in this assumption: "loss on open-source pretraining datasets cannot accurately reflect model performance on downstream tasks" (Section 2.3.1).

ScalingBench is a constructed evaluation dataset where the loss does correlate with downstream task performance through a known functional relationship. The construction process:

1. Start with validation sets from standard downstream benchmarks (the paper does not enumerate exactly which, but they include knowledge and reasoning tasks consistent with the evaluation suite in Section 5).

2. For each test instance, use GPT-4o to generate step-by-step reasoning chains that lead to the correct answer. The original instance consists of a user instruction and a short human-annotated label (typically a few words).

3. Compute the conditional loss on the concatenation of reasoning steps and label, conditioned on the task input. Specifically:

   $\mathcal{L}_{\text{ScalingBench}} = -\frac{1}{|\text{reasoning} + \text{label}|} \sum_{t} \log p(y_t | \text{input}, y_{<t})$

   where $y_t$ are the tokens of the GPT-4o-generated reasoning chain followed by the ground-truth label, and the model generates them autoregressively given the task input.

Why this works: For a small model (millions of parameters) that cannot achieve above-random accuracy on downstream tasks, directly measuring accuracy is uninformative — all configurations look equally bad. But the loss on the reasoning chain is sensitive to whether the model assigns higher probability to correct reasoning steps versus incorrect ones, even if the model never selects the right final answer. The paper demonstrates that the relationship between ScalingBench loss and downstream accuracy follows a sigmoid function across models ranging from 0.36B to 80B parameters (Figure 4), with 7B and 80B models serving as held-out test points that fall on the same curve.

The sigmoid relationship means there is a "sensitive region" of loss values where small improvements in loss translate to large improvements in accuracy, and saturated regions where further loss reduction yields diminishing returns. This makes ScalingBench loss a more informative signal for hyperparameter search than raw language modeling loss, which may improve without corresponding downstream gains.

---

Comparison between $\mu$P and vanilla architecture. The paper evaluates two approaches to hyperparameter transfer from small to large models:

1. $\mu$P (maximal update parameterization, Yang et al., 2022): Modifies the model architecture (specifically, the scaling of initialization and learning rates with width) so that the optimal hyperparameters are invariant to model size. Under $\mu$P, the learning rate found optimal for a small model can be directly used for a large model.

2. StepLaw (Li et al., 2025a): A data-driven method that predicts optimal hyperparameters for a target model size based on scaling trends observed across multiple model scales.

The comparison (Table 3) uses small-scale experiments at 150M, 360M, and 700M parameters, trained on varying token budgets (4B to 100B tokens). The paper finds that both methods produce comparable results, with StepLaw showing "slightly more instances of advantage" but the differences being "minimal" and "neither approach exhibiting consistently stable superiority" (Section 2.3.1).

The paper attributes this equivalence to several factors:

• Hardware constraints prevent strict adherence to StepLaw's prescribed batch size (must be divisible by 16 for GPU efficiency).
• The paper uses WSD learning rate schedules with different data allocations for stable and decay phases, while StepLaw assumes cosine decay.
• Randomness in training and evaluation introduces variance that can mask small systematic differences.

Why the paper chooses $\mu$P: The pragmatic reason is cost. The paper states that "reproducing steplaw's work incurs significant expense, whereas $\mu$P search requires minimal GPU hours. This makes it accessible to common researchers" (Section 2.3.1). The paper uses $\mu$P as the base architecture and conducts hyperparameter search on million-parameter models (the "Wind Tunnel" phase), transferring the found hyperparameters to the 8B model.

---

Pre-training engineering: Multi-token prediction. The standard language modeling objective is next-token prediction (NTP): given preceding tokens, predict the next one. Multi-token prediction (MTP, Gloeckle et al., 2024) extends this to predict multiple future tokens simultaneously using additional prediction heads.

The architecture works as follows:

1. Main model produces hidden states: Given input tokens $\{x_0, x_1, \ldots, x_{l-1}\}$, the main Transformer produces hidden vectors $\mathbf{H} = \{\mathbf{h}_0, \mathbf{h}_1, \ldots, \mathbf{h}_{l-1}\}$.

2. Next-token prediction (NTP) loss:

   $\mathcal{L}_{\text{NTP}} = \frac{1}{l} \sum_i \text{CrossEntropy}(\text{OutputHead}(\mathbf{h}_i), x_{i+1})$

   This is the standard objective: predict $x_{i+1}$ from $\mathbf{h}_i$.

3. Additional prediction head input construction: For each position $i$, the input to the MTP head is the concatenation of the normalized hidden state and the normalized embedding of the next token:

   $\hat{\mathbf{h}}_i = \text{Concat}(\text{Norm}(\mathbf{h}_i), \text{Norm}(\text{Emb}(x_{i+1})))$

   This gives the MTP head access to both the contextual representation and the identity of the token that would have been predicted under NTP.

4. MTP head processing: The concatenated vectors are linearly projected and passed through a single Transformer layer:

   $\{\mathbf{h}^{\text{MTP}}_0, \ldots, \mathbf{h}^{\text{MTP}}_{l-2}\} = \text{Transformer}(\text{Linear}(\{\hat{\mathbf{h}}_0, \ldots, \hat{\mathbf{h}}_{l-2}\}))$

5. MTP loss: The MTP head predicts the token two positions ahead:

   $\mathcal{L}_{\text{MTP}} = \frac{1}{l-1} \sum_i \text{CrossEntropy}(\text{OutputHead}(\mathbf{h}^{\text{MTP}}_i), x_{i+2})$

6. Combined objective:

   $\mathcal{L} = \mathcal{L}_{\text{NTP}} + \lambda \mathcal{L}_{\text{MTP}}$

   where $\lambda$ is a weighting hyperparameter (the paper does not specify its value explicitly in Section 2.3.2).

Why MTP helps: The paper identifies two benefits. First, it "introduces denser supervision signals" — instead of one prediction per token, the model gets two, which improves data efficiency by extracting more learning signal from each training example. Second, the additional prediction heads can be repurposed during inference for speculative decoding: the MTP head can serve as a draft model that predicts upcoming tokens, and the main model verifies them. Training the MTP head jointly with the main model ensures they share representations, which the paper claims leads to "higher acceptance length in speculative sampling" (Section 2.3.2).

---

FP8 mixed-precision training. The paper implements FP8 training following the approach of DeepSeek-V3 (DeepSeek et al., 2024). The key design decisions:

• Quantization granularity: Use block-wise online quantization with block size 128×128 for parameters (weight matrices) and 128×1 for activations. "Online" means quantization parameters are computed dynamically for each batch rather than using pre-computed statistics.

• Precision allocation: Apply FP8 only to linear projection layers during the forward pass (for activations) and backward pass (for input gradients). Parameter gradients are computed in BF16 because "parameters are extremely sensitive to precision" (Section 2.3.2).

• Accumulation: Use FP8 Matrix Multiply Accumulate (MMA) instructions on tensor cores, with FP32 as the accumulator precision to prevent numerical overflow during summation.

• Hardware motivation: The paper explicitly ties this to "NVIDIA's Tensor Core GPUs [which] have powerful FP8 computing capabilities" (Section 2.3.2). FP8 operations are approximately 2× faster than BF16 on H100 GPUs, making this a significant throughput improvement for the training infrastructure.

---

UltraChat v2: Foundational Capability Enhanced SFT Data Generation

After pre-training, the model undergoes supervised fine-tuning (SFT) to learn instruction following and activate the knowledge acquired during pre-training. UltraChat v2 is a multi-track SFT dataset covering five capability dimensions: knowledge application, reasoning, instruction following, long-context processing, and tool use. The data generation framework is "task-oriented" — for each capability, the data construction process is tailored to the specific cognitive demands of that skill.

---

Knowledge-intensive data (Section 3.1.1). The construction process:

1. Knowledge framework extraction: Extract and organize knowledge points from domain-specific corpora, exam syllabi, and textbook materials across disciplines. This creates a structured taxonomy of what the model should know.

2. Initial QA generation: Use LLMs to generate practice question-answer pairs targeting individual knowledge points. These are straightforward "test your knowledge" style questions.

3. Diversity evolution: Apply two transformation strategies to the initial QA pairs:

   • Instruction evolution: Rewrite prompts in diverse ways — different questioning styles, task formulations, and persona settings — to simulate variety in how users might ask about the same knowledge.
   • Answer diversity evolution: Guide the model to generate plausible but stylistically varied answers to the same question, improving robustness to different expression styles.

---

Reasoning-intensive data (Section 3.1.2). Two specialized tracks:

Math reasoning data:

• Organize mathematical knowledge hierarchically by domain (linear algebra, calculus, probability, statistics, differential equations, discrete mathematics, differential geometry) and by educational level (elementary through university).
• Generate problems using curated seed data or direct LLM prompting.
• For each problem, generate multiple valid solution paths to expand the reasoning space — the model learns that there isn't just one "correct" way to solve a problem.
• Apply difficulty-based stratification: "actively reducing the proportion of easy questions during training" to focus the model's learning on medium-to-high-difficulty problems where reasoning chains are longer and more complex.
• Use heuristic rules (instruction adherence, response length control) to ensure mathematical validity and answer verifiability.

Code reasoning data:

• Define coding contexts, problem categories (semantic completion, bug localization, complex logic understanding), and difficulty levels.
• Extract high-quality code snippets from real GitHub repositories, coding challenge libraries, and open-source scripts.
• Use LLMs to generate contextually relevant reasoning problems from these snippets — e.g., "what does this function return when given input X?", "identify the bug in this code," or "rewrite this function to handle edge case Y."
• Generate accompanying unit tests and input-output examples so that model outputs can be automatically verified.
• Diversify through format conversion (output prediction, logical diagnosis, code rewriting) and cross-language translation (Python ↔ Java, C++ ↔ Rust) to teach language-agnostic reasoning patterns.

---

Instruction following data (Section 3.1.3). Four construction strategies:

1. Progressive complexity: Start with simple base instructions and iteratively add constraints (style, format, content requirements) to build a curriculum from easy to hard.

2. Verifiable constraints: Construct instructions with explicitly checkable requirements — length limits, required keywords, structural specifications. Generate multiple outputs with varied decoding parameters and automatically filter to keep only responses that satisfy all constraints. This enables scalable data generation without manual verification.

3. Domain-persona diversity: Combine domain-specific knowledge with various persona settings (e.g., "as a physicist, explain X to a high school student" vs. "as a journalist, summarize X for a general audience") to cover a broader spectrum of real-world instruction patterns.

4. Reverse instruction generation: Treat existing high-quality text as the target output and prompt an LLM to generate plausible instructions that would produce that output. This leverages unlabeled corpora to augment the instruction dataset without requiring human-written prompts.

---

Long-context data (Section 3.1.4). Inspired by LongAlign (Bai et al., 2024):

1. Sampling documents: Draw documents from diverse pre-training sources — web pages, source code, mathematical content, encyclopedic texts.

2. Query generation: For each document $d$, use an LLM to generate $n$ task-oriented queries (extraction, summarization, reasoning, open-domain QA).

3. Distractor retrieval: For each query $q_j$, retrieve $k$ related but potentially irrelevant documents from an indexed corpus.

4. Context construction: Concatenate the original document with the retrieved documents, inserting the original document at a random position:

   $C_j = \text{Concat}(d_{j,1}, \ldots, d_{j,m-1}, d, d_{j,m}, \ldots, d_{j,k})$

   where $m \in \{1, \ldots, k+1\}$ is randomly selected.

5. Answer generation: An LLM is prompted with query $q_j$ and context $C_j$ to generate answer $a_j$.

6. Length control: Total token count of $C_j$ is uniformly distributed between 8K and 64K tokens to ensure coverage across different context lengths.

The random insertion position and distractor documents teach the model to locate relevant information within long, mostly-irrelevant contexts — a skill that is critical for real-world long-document processing.

---

Tool use data (Section 3.1.5). Two sub-tracks:

Function calling:

• Combine publicly available datasets (xlam-function-calling-60k, glaive-function-calling-v2) with in-house data generated via in-context learning.
• Apply strict filtering: remove samples where the ground-truth tool is not in the available tool set, or where parameter names/types are inconsistent with the tool schema.
• Prepend a chain-of-thought reasoning step before the tool invocation to help the model understand the task and select appropriate tools.

Code interpreter:

• Curate open-source datasets (CodeAct, Code-Feedback) and preprocess by analyzing ASTs to filter out code that imports external packages or requires user interaction.
• For in-house data: collect various file types (CSV, PDF, images, videos), prompt an LLM to generate code-solvable problems related to file content, and have the model solve them using a code interpreter in a sandboxed environment.
• Allow up to 10 attempts with execution feedback; discard data points where the model fails to solve within this limit. This creates examples that teach the model to use code execution iteratively for problem-solving.

---

Chunk-wise Rollout: Load-Balanced Reinforcement Learning for Deep Reasoning

After SFT provides basic instruction-following and reasoning capabilities, the paper applies reinforcement learning specifically to enhance deep reasoning (long chain-of-thought problem solving on mathematics and code). The central challenge is computational efficiency during the rollout phase, where the model generates responses that are then scored by a reward function.

---

The load imbalance problem. In standard RL for language models (using algorithms like GRPO), each training step involves:

1. Sampling a batch of prompts.
2. Having the current policy model generate complete responses for each prompt.
3. Computing rewards for each response.
4. Computing policy gradients and updating the model.

The problem: response lengths can vary dramatically. A simple arithmetic problem might complete in 100 tokens; an AIME-level competition math problem might require 8,000 tokens of chain-of-thought reasoning. In a synchronous training setup (which is standard for GPU efficiency), the entire batch waits for the longest response to finish generating before rewards can be computed and gradients applied. This means most GPUs sit idle while a few straggler sequences finish.

---

Chunk-wise rollout strategy (Algorithm 1). The core idea: instead of generating complete responses in one go, generate fixed-length chunks and resume incomplete responses in subsequent iterations.

The algorithm proceeds as follows:

1. Initialization: Maintain a replay buffer $\mathcal{R}$ for unfinished trajectories, a dynamic sampling buffer $\mathcal{B}$ for completed trajectories awaiting training, and a log-prob buffer $\mathcal{L}$ for storing computed log-probabilities.

2. For each training step:

   • Sample a batch $\mathcal{D}_b$ of prompts from the training set.
   • Append any unfinished trajectories from the replay buffer $\mathcal{R}$ to the batch.
   • For each prompt $q$ in the batch, generate $G$ outputs, but each output is generated up to a fixed chunk length (controlled by a token budget, e.g., 4K or 8K tokens).
   • If an output reaches the chunk length without producing an end-of-sequence token, store it in the replay buffer $\mathcal{R}$ as "unfinished." It will be resumed in a future step.
   • If all $G$ outputs for a prompt are completed (either naturally terminated or reached maximum generation length), compute rewards and add the completed trajectories to the dynamic sampling buffer $\mathcal{B}$.
   • Once the sampling buffer contains at least $N$ complete trajectories, sample a training batch of size $N$ and compute the policy gradient update.

3. Stabilization mechanisms: Because trajectories now span multiple model versions (the policy model is updated between chunk generation and resumption), several techniques are needed to maintain training stability.

Why this improves efficiency (Table 5): The paper benchmarks the chunk-wise strategy against a vanilla rollout baseline using DeepSeek-R1-Distill-Qwen-1.5B trained on the DAPO dataset for 150 steps. Key results:

(All timing values normalized to the vanilla baseline; lower is better.)

The chunk-wise strategy reduces total step time by 26-42% and sampling time by 36-62%, with minimal impact on accuracy. Chunk-8K achieves the best accuracy-time tradeoff: 34.79 on AIME 2024 (vs. 32.91 vanilla) while reducing sampling time by 56%.

Importantly, the paper observes that reducing chunk size from 8K to 4K further decreases sampling time but does not further decrease total step time. The reason: smaller chunks mean more frequent log-probability computations for chunk-level importance sampling (explained below), which adds overhead that offsets the sampling speedup. The paper identifies this as a tradeoff for future optimization.

---

Stabilization techniques for chunk-wise rollout (Section 3.2.3). The chunk-wise strategy introduces several sources of training instability:

1. Distributional shift: When a trajectory is paused mid-generation and resumed after the policy model has been updated, the second half is generated by a different policy than the first half. This violates the on-policy assumption of standard policy gradient methods.

2. Off-policy correction: The importance sampling ratio used in GRPO assumes trajectories were generated by the current policy. Partial trajectories from previous model versions require correction.

3. Garbled outputs: Reusing incomplete trajectories across iterations increases the risk of degenerate outputs (repetition, incoherence) that can destabilize training.

Chunk-level importance sampling. Standard GRPO uses the importance sampling ratio:

$r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}$

where $\pi_\theta$ is the current policy, $\pi_{\theta_{\text{old}}}$ is the behavior policy that generated the trajectory, and $o_{i,t}$ is the $t$-th token of the $i$-th output.

In chunk-wise rollout, different chunks of the same trajectory are generated by different behavior policies. The paper applies importance sampling at the chunk level: for each chunk, the denominator uses the specific policy version that generated that chunk, not a single $\pi_{\theta_{\text{old}}}$. This is expressed in Equation 8:

$r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{c(t)}}(o_{i,t} \mid q, o_{i,<t})}$

where $\theta_{c(t)}$ is the policy parameters at the time chunk containing token $t$ was generated. The log-probability buffer $\mathcal{L}$ stores $\log \pi_\theta(o_{i,t})$ for all cached chunks under the current policy, which is needed for the numerator; the denominator values are stored when each chunk is originally generated.

Dual-clip. The chunk-wise strategy introduces partial off-policy rollouts that can cause "spikes in training loss due to high variance in sampled trajectories" (Section 3.2.3). Dual-clip (Ye et al., 2020) constrains the policy update from both directions:

Standard PPO-style clipping restricts the importance sampling ratio to $[1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}]$ when the advantage $\hat{A}_{i,t} > 0$. Dual-clip additionally applies a lower-bound clip when the advantage is negative, preventing the policy from moving too far away from the behavior policy for actions that received negative rewards. This is formalized in Equation 7:

For $\hat{A}_{i,t} > 0$ (positive advantage — the action was good): $\min\left(r_{i,t}(\theta) \cdot \hat{A}_{i,t},\; \text{clip}(r_{i,t}(\theta), 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}) \cdot \hat{A}_{i,t}\right)$

For $\hat{A}_{i,t} \leq 0$ (negative advantage — the action was bad): $\max\left(\min\left(r_{i,t}(\theta) \cdot \hat{A}_{i,t},\; \text{clip}(r_{i,t}(\theta), 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}) \cdot \hat{A}_{i,t}\right),\; c \cdot \hat{A}_{i,t}\right)$

The additional $c \cdot \hat{A}_{i,t}$ term (where $c$ is a constant) acts as a floor — even if the importance ratio is very large, the update is bounded below by $c$ times the advantage, preventing extremely large negative updates for off-policy actions.

KL regularization with dynamic reference updates. Unlike recent work that removes the KL penalty from the GRPO objective (Yu et al., 2025a; Xia et al., 2025), the paper retains it as essential for chunk-wise stability:

$\mathcal{J}_{\text{KL}}(\theta) = D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$

where $\pi_{\text{ref}}$ is a reference policy. To avoid the KL penalty becoming too restrictive (which would prevent the model from improving), the reference model is periodically updated to the current policy — this balances stability (by penalizing large jumps) with progress (by allowing the reference to drift over time).

The full objective (Equation 6):

$\mathcal{J}(\theta) = \mathcal{J}_{\text{clip}}(\theta) - \beta \cdot \mathcal{J}_{\text{KL}}(\theta)$

where $\beta = 0.001$ (Section 3.2.5).

Garble filter. The chunk-wise strategy increases the risk of generating corrupted text (garbled output, excessive repetition) because incomplete trajectories from earlier policy versions may diverge when resumed. The garble filter detects and excludes such samples from loss computation, preventing them from destabilizing training. The paper does not provide detailed heuristics for what constitutes "garble," but typical approaches check for excessive repetition (n-gram overlap above a threshold), low perplexity collapse, or Unicode artifact patterns.

Dynamic sampling (additional GRPO improvement). Beyond the chunk-wise strategy, the paper modifies the standard GRPO algorithm in several ways:

• Prompt filtering: During the rollout phase, filter out prompts for which all generated responses are correct or all are incorrect. These prompts contribute zero-variance gradient estimates (the advantage is identical across all samples) and therefore provide no useful learning signal. Removing them maintains a consistent effective batch size and reduces gradient variance.

• Clip-higher: Raise the upper clipping threshold $\varepsilon_{\text{high}}$ to alleviate entropy collapse in later training stages, allowing the policy to explore more diverse responses.

• Token-level loss: Instead of averaging the policy gradient loss at the sample level (each response contributes equally regardless of length), compute it at the token level:

  $\mathcal{J}_{\text{clip}}(\theta) = \frac{1}{\sum_i |o_i|} \sum_i \sum_t \min\left(r_{i,t}(\theta) \hat{A}_{i,t}, \ldots\right)$

  This gives longer sequences proportionally more weight in the gradient update, encouraging the model to learn complex multi-step reasoning rather than converging to short, simple responses.

• Overlong sample filtering: Exclude responses that are truncated due to length constraints from the loss computation, preventing the model from being penalized for valid reasoning that was cut off by the generation limit.

---

RL implementation details (Section 3.2.5).

• Training batch size: 256
• Mini-batch size: 128
• Learning rate: constant $1 \times 10^{-5}$
• $\mu$P learning rate strategy (matching the pre-training architecture)
• KL penalty coefficient: 0.001
• No entropy constraint (unlike some recent work)
• Maximum response length: 32,768 tokens
• Rollout temperature: 1.0
• Rollout top-p: 1.0
• Number of rollouts per query: 16
• $\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$: not explicitly specified in the text but standard values would be around 0.2-0.3

---

RL data curation (Section 3.2.1).

Mathematics data sources: DAPO, Deepscaler, Numina, Prime, and other verifiable mathematical datasets. Rewards are computed using a combination of rule-based matching and symbolic verification via SymPy. Outputs that don't conform to the expected reasoning format receive zero reward.

Code data sources: LeetCode, TACO, Kodcode, Codeforces, and similar platforms. Code is executed in a Firejail sandbox environment for safety. For problems with multiple test cases, the reward is the proportion of test cases passed, with a full reward of 1.0 assigned when all test cases pass.

Data filtering: After collection, deduplication is performed on both RL and SFT data using Semhash. To retain challenging samples, DeepSeek-R1-Distill-Qwen-1.5B is used to generate four predictions per training example. Samples for which all four predictions are correct are filtered out as too easy. Code data is upsampled multiple times because its quantity is significantly smaller than math data.

---

BitCPM4: Quantization-Aware Training for Ternary LLMs

For extremely resource-constrained devices, even 4-bit quantization may not be sufficient. BitCPM4 pushes this to the extreme: ternary quantization, where each weight takes one of three values ($\{-1, 0, +1\}$ scaled by a per-channel factor). The paper proposes an efficient quantization-aware training (QAT) approach that converts a pre-trained high-precision model to ternary rather than training a ternary model from scratch (as BitNet does).

---

Why activations are not quantized. Standard QAT typically quantizes both weights and activations. The paper's preliminary experiments indicate that for extremely low-bit models, "quantizing activations increases the QAT overhead without reducing too much inference costs on end-side devices" (Section 3.3.1). The reasoning: on end-side devices, the memory bottleneck is typically model weights (which must be stored and loaded), not activations (which are transient and can be recomputed). Therefore, they apply ternary quantization only to weights.

---

Two-stage training with learning rate re-warmup. The process:

1. Stage 1: Train an FP8 model normally (this is the base MiniCPM4 model).

2. Stage 2: Convert the FP8 model to ternary by replacing each weight with its ternary approximation, then continue training (QAT) to recover performance. Critically, the learning rate is re-warmed up at the beginning of stage 2: after the sudden degradation from weight quantization, a higher learning rate is needed to re-adapt the model. The paper uses a learning rate of $1 \times 10^{-2}$ in stage 1 and $5 \times 10^{-3}$ in stage 2.

3. Token allocation: The paper sweeps the proportion of total training tokens allocated to the QAT stage. Results (Figure 5) show that when the QAT proportion exceeds 40% of total tokens (equivalent to twice the decay-phase tokens), the final loss closely approaches that of training a ternary model from scratch. Based on this, BitCPM4 uses a QAT phase of approximately 350B tokens (the paper states "twice the number of tokens used in the learning rate decay phase").

Why this works: The paper's insight is that QAT from a pre-trained checkpoint can match from-scratch ternary training if given sufficient continued-training budget. This is important because from-scratch ternary training (as in BitNet) requires re-running the entire pre-training pipeline in low precision, which is computationally expensive. The two-stage approach reuses the existing high-precision pre-training, which is already done, and only invests additional compute in the adaptation phase.

Comparison with BitNet (Table 6):

(* marks results from the original BitNet paper.)

The 1B BitCPM4 model outperforms the 2B BitNet model on average (56.03 vs. 43.31) despite having half the parameters and using only 10% of the training tokens (350B vs. 4T for BitNet). However, the 0.5B version shows weaker performance on math (GSM8K 25.55, MATH500 10.20), which the paper attributes to "smaller model size restricts reasoning capabilities" (Section 3.3.2) and notes that quantization effectiveness follows a scaling law — larger models quantize better.

---

CPM.cu: Lightweight CUDA Inference Framework

The final component of the technical approach is the inference system that actually runs MiniCPM4 efficiently on end-side NVIDIA GPUs. CPM.cu is a ground-up implementation (not a modification of existing frameworks like vLLM or llama.cpp) that incorporates static memory management, kernel fusion, and specialized kernels for the paper's architectural innovations.

Static memory management: Unlike dynamic allocation used in server-side frameworks where batch sizes vary, CPM.cu pre-allocates all GPU memory buffers at initialization based on the maximum context length and batch size. This eliminates allocation overhead during inference and prevents fragmentation — critical for devices with limited memory where every megabyte counts.

Kernel fusion: Multiple small operations (e.g., residual addition, layer normalization, activation functions) are fused into single GPU kernels to reduce kernel launch overhead and memory bandwidth consumption. This is standard practice in optimized inference frameworks.

Efficient sparse attention kernel for InfLLM v2: A dedicated CUDA kernel implements the two-stage sparse attention computation described in Section 2.1. The kernel must handle the irregular memory access pattern of block selection (each query token attends to a different subset of blocks) while maintaining coalesced memory reads — a non-trivial GPU programming challenge.

---

FR-Spec: Frequency-ranked speculative sampling. Speculative sampling accelerates autoregressive decoding by using a lightweight "draft" model to propose candidate tokens, which are then verified in parallel by the target model. The paper identifies that the bottleneck in speculative sampling for end-side models is the language modeling head — the final linear projection from hidden states to vocabulary logits. Modern LLMs use large vocabularies (tens of thousands of tokens), making this operation expensive even for a single-layer draft model.

FR-Spec exploits the long-tail distribution of token frequencies: a small subset of tokens accounts for the vast majority of occurrences in natural language. The method:

1. Analyze token frequencies on large-scale pre-training data to establish a frequency ranking.

2. Select the top-$k$ tokens to form a reduced vocabulary subset $\mathcal{V}_{\text{high}}$. The paper selects approximately 25% of the vocabulary ($k = 0.25 \times |\mathcal{V}|$), which captures 95% of token occurrences.

3. Modify the draft model's language modeling head to only compute logits over $\mathcal{V}_{\text{high}}$: instead of multiplying the hidden state by a $|\mathcal{V}| \times d$ matrix, multiply by a $|\mathcal{V}_{\text{high}}| \times d$ matrix (Equation 9):

   $\tilde{\mathbf{W}}_{\text{LM}}[i, :] = \mathbf{W}_{\text{LM}}[\mathcal{V}_{\text{high}}[i], :], \quad i = 1, \ldots, |\mathcal{V}_{\text{high}}|$

   The draft distribution becomes (Equation 10):

   $\mathcal{D}_{\text{FR}}(\mathbf{x}) = \text{Softmax}(\mathbf{H}_{\text{draft}}(\mathbf{x}) \tilde{\mathbf{W}}_{\text{LM}}^T)$

4. The target model still operates over the full vocabulary. During verification, if the draft model proposes a high-frequency token, the target model checks it against the full distribution. If the target model would have selected a low-frequency token that the draft couldn't propose, the draft token is rejected and the target model's token is used instead. This preserves the mathematical equivalence of the output distribution — FR-Spec produces exactly the same tokens as standard speculative sampling, just faster.

Computational complexity: The head computation reduces from $O(n d |\mathcal{V}|)$ to $O(n d |\mathcal{V}_{\text{high}}|)$, where $n$ is the draft sequence length and $d$ is the hidden dimension. For $|\mathcal{V}_{\text{high}}| = 0.25 \times |\mathcal{V}|$, this is a 4× reduction. The softmax computation scales down proportionally.

---

P-GPTQ: Prefix-aware post-training quantization. GPTQ (Frantar et al., 2023) is the standard post-training quantization method that optimizes weight rounding by minimizing the error in layer outputs, using a Hessian matrix computed from calibration data:

$\mathbf{H} = \mathbf{X}^\top \mathbf{X}$

where $\mathbf{X} \in \mathbb{R}^{n \times d}$ is the calibration data (activations from running sample inputs through the model).

The paper identifies a failure mode: "when computing the covariance matrix for down-projection layers, particularly in those deeper Transformer blocks, the beginning of sentence token and some initial tokens consistently introduce significant statistical bias. These initial positions exhibit activation magnitudes 10× larger than subsequent tokens, disproportionately dominating the covariance structure" (Section 4.1.2). This is related to the "massive activations" phenomenon (Sun et al., 2024b) where initial tokens have outlier values that skew layer statistics.

P-GPTQ mitigates this by excluding the first $s$ token positions from the Hessian computation (Equation 12):

$\hat{\mathbf{H}} = \mathbf{X}_{\text{valid}}^\top \mathbf{X}_{\text{valid}}, \quad \mathbf{X}_{\text{valid}} = \mathbf{X}_{[s:]}$

where the paper finds that "token positions starting from $s = 4$ exhibit stable statistical features." This simple modification is compatible with other quantization techniques (rotation methods like Quarot, smoothing methods like AWQ), enabling integration into existing pipelines.

Evaluation (Table 7): On MiniCPM4-8B with all linear layers quantized to per-group INT4, using 1,024 randomly selected calibration sequences:

S-P-GPTQ (P-GPTQ with AWQ smoothing) achieves the smallest degradation from the FP16 baseline (0.67 points vs. 1.27 for standard GPTQ), confirming that prefix-aware Hessian computation improves quantization quality.

---

ArkInfer: Cross-Platform Deployment System

While CPM.cu is optimized for NVIDIA GPUs, ArkInfer addresses the fragmentation of end-side hardware (MediaTek, Qualcomm, Rockchip, and CPU-only devices) by providing a unified deployment abstraction.

Architecture (Section 4.2.1):

• Backend adapters: Normalize the varied APIs of different inference frameworks (NeuroPilot for MediaTek, Genie, RK-LLM for Rockchip, TensorRT-LLM for NVIDIA, llama.cpp for CPU) into a consistent interface.

• Unified Tensor structure: Wraps diverse data types and dimensions for consistent manipulation across backends.

• KV cache manager: Orchestrates historical state storage and retrieval, critical for efficient autoregressive generation.

• Abstract executor interface: Governs runtime execution of all model-related processes (encoding, decoding, sampling, preprocessing), enabling heterogeneous scheduling at the executor granularity.

Speculative and constrained decoding (Section 4.2.2):

• BiTA speculative decoding (Lin et al., 2024a): An alternative to EAGLE-2 that doesn't require a separate draft model. Bi-directional Tuning for Acceleration uses the target model itself for drafting through clever manipulation of attention masks. This is simpler to deploy across platforms because it avoids the need for a draft model architecture.

• Constrained decoding with Guidance: Ensures outputs adhere to specific formats (JSON, SQL) by constraining the token sampling process to only valid continuations. This is crucial for tool use applications where outputs must be parseable by downstream systems.

Model zoo frontend (Section 4.2.3): ArkInfer maintains a centralized collection of pre-adapted models for different platforms. An automated conversion pipeline transforms models into platform-specific formats, reducing the manual engineering effort of porting each new model release to every supported chip. Users can directly access and execute models from the zoo without managing platform-specific conversion themselves.

4. Key Insights and Innovations

Innovation 1: End-Side LLM Efficiency Requires Coordinated, Full-Stack Co-Design — Not Isolated Optimizations

The most distinctive intellectual contribution of MiniCPM4 is not any single technique but the meta-argument that efficiency for end-side deployment is a systems integration problem, not a point solution problem. The paper argues — implicitly through its architecture, and explicitly through the breadth of its interventions — that optimizing any one dimension (architecture, data, training, inference) in isolation yields diminishing returns, because the bottlenecks are coupled. A faster attention mechanism is useless if the model was trained on noisy data and needs more parameters to compensate. An efficient training recipe is wasted if the inference system can't exploit the resulting architecture on diverse hardware.

This framing departs fundamentally from how most efficiency-focused LLM papers operate. The dominant paradigm in the literature is to propose a single technique — a sparse attention variant (StreamingLLM, MInference, NSA, MoBA), a quantization method (GPTQ, AWQ, BitNet), a data filtering strategy (FineWeb-edu, DataComp-LM), or a training efficiency improvement (FP8 training, multi-token prediction) — and evaluate it in isolation against baselines that keep everything else fixed. The implicit assumption is that efficiency gains compose additively: solve the attention bottleneck, then solve the data bottleneck, then solve the quantization bottleneck, and the cumulative speedup is the product of individual speedups.

MiniCPM4 challenges this assumption by demonstrating that the interactions between components are first-order effects, not second-order corrections. Consider three examples:

• The InfLLM v2 architecture and the CPM.cu inference framework are co-designed. The sparse attention kernel (described in Section 2.1) uses a specific block size, semantic kernel stride, and query group sharing pattern that are chosen not just for algorithmic accuracy but for hardware efficiency on NVIDIA tensor cores (minimum 16 heads per group, block sizes matching memory transaction widths). If InfLLM v2 were evaluated with a generic inference framework that didn't implement the specialized Top-K kernel with LSE approximation, the speedup would be substantially lower. Conversely, the CPM.cu kernel's efficient LSE approximation only works because InfLLM v2's relevance scoring operates on mean-pooled semantic kernels rather than token-level representatives — a generic sparse attention kernel couldn't achieve the same speedup without the algorithmic design choice.

• The UltraClean data pipeline and the pre-training budget are co-optimized. The paper's claim that MiniCPM4-8B matches Qwen3-8B using 22% of the training data (8T vs. 36T tokens, Table 8) only holds because the data filtering is unusually aggressive — and the data filtering can only be that aggressive because the efficient verification strategy makes it practical to iteratively refine the classifier. A standard data filtering approach that required 1,200 GPU-hours per verification cycle would be too expensive to tune to the same level of precision, and the resulting model trained on 8T tokens would underperform. The data quality and the training budget are not independent variables; the quality determines how far the budget goes.

• The chunk-wise RL rollout and the stabilization techniques are co-required. The load balancing gains from chunking (Table 5: 42% reduction in sampling time) are only realizable because the paper simultaneously introduces chunk-level importance sampling, dual-clip, KL regularization, and garble filtering. Without these, the training would destabilize from distributional shift, and the efficiency gain would be offset by degraded model quality. The techniques form a coupled system where removing any one component breaks the others.

This systems perspective is not just a methodological preference — it is a diagnosis of why prior work has underdelivered on end-side LLM performance. The paper implies (without stating it confrontationally) that the field's tendency to publish point solutions creates a misleading picture of progress. A technique that shows a 2× speedup in a controlled ablation may deliver only a 1.2× speedup when integrated into a real deployment pipeline, because the bottleneck shifts to a different component that wasn't optimized. By presenting a fully integrated system rather than a single-method paper, MiniCPM4 makes the case that end-side efficiency is a property of the entire stack, not of any component in isolation.

The empirical evidence for this claim is distributed across the paper's evaluation sections, but the most compelling single data point is Figure 1: MiniCPM4 achieves approximately 7× decoding speedup over Qwen3-8B on 128K sequences on Jetson AGX Orin. Qwen3-8B uses dense attention; MiniCPM4 uses InfLLM v2 sparse attention. But the 7× is not purely an attention speedup — it also reflects the effects of FR-Spec speculative decoding (which is only effective because the MTP training objective produced a good draft head), P-GPTQ quantization (which preserves model quality due to prefix-aware calibration), and the CPM.cu kernel implementation. Disentangling these contributions is impossible because they are synergistic.

The paper's title and abstract position it as a model release, but the intellectual contribution is the demonstration that full-stack co-design can shift the Pareto frontier of the accuracy-efficiency tradeoff more than any single-method paper could suggest. This is a conceptual reframing, not a theorem, but it has practical consequences for how the field should organize research on efficient LLMs.

---

Innovation 2: Trainable Sparse Attention Must Separate Query and Key-Value Granularities to Enable Both Prefilling and Decoding Acceleration

The specific architectural insight of InfLLM v2 — that sparse attention mechanisms should operate at different granularities for queries (token-level) and key-values (block-level) — is a clean conceptual advance over prior work on sparse attention, even though it seems simple in retrospect. The paper effectively diagnoses a structural flaw shared by many previous approaches and provides a principled solution.

What prior work got wrong. The paper identifies two failure modes in existing sparse attention designs:

1. Training-free methods (StreamingLLM, MInference, XAttention, SpargeAttn) use heuristic relevance scoring based on static patterns or simple dot products. The paper's critique is that these can achieve high sparsity only by sacrificing accuracy: "These models can only be applied in prefilling acceleration due to their unsatisfactory sparsity." The implication is that without end-to-end training, the attention mechanism cannot learn to structure its key representations to make sparse selection reliable at the high sparsity levels needed for meaningful decoding speedup.

2. Trainable methods that use query blocking (MoBA) group consecutive query tokens into blocks that share the same selected context. The paper identifies a subtle but fatal problem: "query blocking operations can only accelerate long-sequence prefilling but cannot speed up the decoding process, as decoding requires token-by-token generation, and in most cases, query tokens cannot form a complete block." During autoregressive generation, tokens are produced one at a time — you can't form a block from a single token — so the training-time block structure creates a training-inference mismatch. The model is trained to attend based on group-level relevance scores, but at inference time, each token has to make its own selection using a degraded approximation of the group-level signal.

3. Trainable methods that add parameters for context selection (NSA) introduce separate attention components (compressed, selected, sliding window) with additional parameters. The paper's critique is practical: "these three attention components introduce additional parameters, which will lead to increased computational overhead for short sequences and threefold key-value storage costs for pre-training." For an efficiency-focused model, the overhead from these extra components can outweigh the sparsity gains — especially on short sequences, which are the common case.

The conceptual advance. InfLLM v2's key insight is to decouple the granularity of queries and key-values. Queries select blocks at the token level — each token independently chooses which key-value blocks to attend to, ensuring that decoding (where tokens are generated one at a time) works identically to training. Key-values are organized at the block level — they are selected and accessed as contiguous groups, enabling coalesced memory access and amortized relevance scoring. This decoupling resolves both problems simultaneously:

• Decoding acceleration works because each generated token selects its own blocks using the same mechanism as during training. There is no query blocking, no group-level approximation, no training-inference mismatch.

• Sparsity can be high because the block-level key-value organization makes the selection overhead O(l/s) rather than O(l) — the relevance scoring cost is proportional to the number of semantic kernels (which grows with sequence length divided by stride), not the number of tokens. This means the mechanism can achieve 95% sparsity on 128K sequences (attending to ~6K tokens) while keeping the selection cost manageable.

This decoupling principle is not obvious ex ante. The natural instinct — reflected in MoBA and other query-blocking approaches — is to apply sparsity symmetrically: if key-values are grouped into blocks, queries should be too, to maximize throughput. The paper's contribution is to recognize that this symmetry is actively harmful for decoding, and that the correct design is asymmetric: queries operate at the finest granularity the hardware allows (token-level), while key-values operate at the coarsest granularity that preserves relevance scoring accuracy (block-level).

Why this matters beyond incremental improvement. This insight changes how researchers should think about sparse attention. Prior work implicitly treated "sparse attention" as a single design problem — find a selection mechanism that is fast and accurate. InfLLM v2 shows that it is actually two coupled design problems: (1) how to select relevant context efficiently (the key-value side, where block-level granularity is appropriate), and (2) how to apply the selection during generation (the query side, where token-level granularity is essential). Treating these as separable with different optimal solutions is a conceptual advance that will likely influence future sparse attention designs.

The empirical evidence is in the efficiency evaluation (Figure 1): MiniCPM4 achieves consistent speedup in both prefilling and decoding across sequence lengths from 32K to 128K, while comparably-sized dense models (Qwen3-8B, GLM4-9B, Llama3-8B) show rapidly degrading throughput as sequence length increases. The 7× decoding speedup over Qwen3-8B at 128K is the headline number, but the more diagnostic finding is that MiniCPM4's throughput curve is much flatter — the sparse attention effectively decouples inference cost from sequence length, which is exactly what the decoupled granularity design enables.

---

Innovation 3: Data Quality Verification Can Be Decoupled from Full Model Training Through Targeted Annealing — Making Iterative Data Curation Practically Feasible

The UltraClean pipeline's efficient verification strategy is, at first glance, a simple engineering trick: instead of training a model from scratch to evaluate data quality, fine-tune a nearly-trained model during its annealing phase. But this "trick" has an important conceptual implication: data quality assessment does not require the same computational scale as model training, because the sensitivity of a model to data changes is highest near convergence, not at initialization.

This challenges an implicit assumption in much of the data curation literature. The standard approach to verifying that a filtering strategy improves data quality — adopted by FineWeb, DataComp-LM, CCI3-HQ, and many others — is to train a model from scratch on the filtered data and compare it to a baseline trained on unfiltered data. This is intuitively appealing because it directly measures what we care about (does better data produce a better model?), but it is computationally prohibitive for iterative refinement. A single verification run costs 1,200 GPU-hours for a 1B model on 100B tokens (Table 1), meaning you can afford maybe 5-10 experiments in a typical research budget. This forces data curation to rely on proxy metrics (perplexity on reference corpora, LLM-based quality scores, heuristic features) that may not correlate well with downstream performance.

The conceptual move. UltraClean's two-stage annealing strategy reframes data quality verification as a sensitivity measurement problem rather than a training-from-scratch problem. The key observation is that a model near convergence is highly sensitive to the data distribution — small amounts of high-quality data during the final decay phase produce measurable improvements in loss and benchmark scores, while low-quality data produces stagnation or degradation. This sensitivity means you can evaluate data quality using ~10B tokens of fine-tuning rather than ~100B+ tokens of from-scratch training — a ~10× cost reduction (110 vs. 1,200 GPU-hours).

The intellectual contribution is not the annealing technique itself (which is adapted from Llama 3.1's training recipe), but the recognition that this sensitivity can substitute for full-scale training in a data verification loop. Prior work used annealing for model improvement; UltraClean uses it for data evaluation. This is a repurposing of an existing technique for a qualitatively different purpose — quality assessment rather than performance optimization.

Why this matters beyond the 10× cost reduction. The practical consequence is that data curation becomes an iterative empirical science rather than a one-shot heuristic process. With verification costs at 110 GPU-hours, a research team can afford to run 20-30 experiments to tune their filtering strategy — testing different seed data sources, classifier architectures, positive-negative sample ratios, and classification thresholds. With verification costs at 1,200 GPU-hours, they can afford maybe 3-5. The qualitative difference is whether data curation is guided by systematic evidence or by intuition.

The paper provides evidence that this matters in practice: the classifier training recipe involves multiple iterative refinements (updating seed pools based on classifier output, adjusting positive-negative ratios, fine-tuning hyperparameters), and the paper states that "only classifiers demonstrating stable and reliable performance under efficient verification are used for large-scale data filtering." This implies that many candidate classifiers failed the verification step — failures that would have been invisible (or prohibitively expensive to detect) under a from-scratch verification paradigm.

The downstream result is in Table 2: UltraFineWeb-en improves average benchmark scores by 3.61 points over FineWeb and 1.33 points over FineWeb-edu (which itself was a state-of-the-art filtered dataset). These gains are modest in absolute terms but significant when multiplied across the 8.3T tokens of pre-training data — and they were achieved because the efficient verification strategy made it possible to tune the filtering pipeline to this level of precision.

A subtle but important point: the paper's verification strategy evaluates data quality conditional on the specific base model being used. This matters because different model architectures and tokenizers respond differently to the same data. The verification is done on a 1B model with the MiniCPM tokenizer, which shares vocabulary and architecture with the target 8B model — the assumption (consistent with $\mu$P scaling) is that data quality rankings transfer. This is a more principled approach than using generic quality metrics that ignore model-data interaction effects.

---

Innovation 4: The Primary Bottleneck in RL for Reasoning Is Load Imbalance, Not Algorithm Design — And Chunking Solves It If You Stabilize Correctly

The chunk-wise rollout strategy for reinforcement learning is, on its surface, an engineering optimization: break long trajectories into pieces so GPUs don't sit idle. But the paper's treatment reveals a deeper conceptual point: the dominant efficiency bottleneck in RL for reasoning is not algorithmic convergence speed but computational load imbalance during rollouts, and addressing this requires rethinking the rollout mechanism, not just tuning hyperparameters.

This diagnosis runs counter to the focus of most recent work on RL for LLMs. The research community's attention has been on algorithmic innovations — GRPO vs. PPO, whether to remove the KL penalty, how to design reward functions, when to use verifiable vs. learned rewards (DAPO, DeepSeek-R1, MIMO, OpenAI o1). The implicit assumption is that if we can make the algorithm converge faster (in terms of sample efficiency), training will become more practical. MiniCPM4 argues that for end-side models, sample efficiency is not the bottleneck — wall-clock efficiency is, and wall-clock efficiency is dominated by the fact that rollout lengths have high variance and synchronous training forces all GPUs to wait for the slowest example.

The diagnostic evidence is in Table 5: comparing vanilla rollout to Chunk-4K, the sampling time per step drops from 392.61 to 148.14 (normalized units) — a 62% reduction — while AIME accuracy is essentially unchanged (32.91 vs. 32.71 for AIME 2024, 25.21 vs. 26.04 for AIME 2025). This means the naive RL pipeline is spending over half its time waiting for straggler sequences, and eliminating this idle time comes at essentially zero accuracy cost. This is not a subtle algorithmic improvement; it's a massive operational inefficiency that prior work simply accepted as unavoidable.

Why this is conceptually novel rather than just good engineering. The chunk-wise strategy is not just "generate shorter sequences" — if you truncate sequences at a fixed length, you lose training signal from long reasoning chains and bias the model toward short (and potentially incorrect) solutions. The chunk-wise approach instead decouples generation from training granularity: generations happen in chunks for load balancing, but training operates on complete trajectories for correctness. This decoupling only works because the paper introduces a suite of stabilization techniques (chunk-level importance sampling, dual-clip, KL regularization with dynamic reference updates, garble filtering) that address the distributional shift from resuming trajectories under updated policies.

The intellectual contribution is the recognition that these stabilization techniques are the enabler — the chunking is trivial to implement, but making it work without destabilizing training requires solving a non-trivial off-policy credit assignment problem. The paper identifies that incomplete trajectories span multiple policy versions, so the standard importance sampling ratio used in GRPO must be computed with per-chunk behavior policies rather than a single $\pi_{\theta_{\text{old}}}$. The dual-clip and KL regularization prevent the policy from diverging too far from the (multiple) behavior policies that generated the partial trajectories. The garble filter prevents corrupted partial trajectories from poisoning the gradient.

This is not an algorithmic innovation in the sense of a new policy gradient variant — GRPO with chunk-level importance sampling is a straightforward extension. But it is a diagnostic innovation: it identifies that the path to efficient RL for reasoning is through solving the load imbalance problem, and that doing so requires treating the rollout mechanism as a first-class design concern (with its own stability requirements) rather than an implementation detail.

Significance beyond the immediate technique. This finding has implications for how the field should allocate research effort in RL for LLMs. Most current work optimizes for sample efficiency — how many training steps are needed to reach a given accuracy. MiniCPM4 suggests that for practical training, wall-clock efficiency matters more, and wall-clock efficiency is dominated by factors (load imbalance, communication overhead, memory management) that are invisible in the standard sample-efficiency metrics. The chunk-wise rollout strategy is one solution to one such factor, but the broader implication is that the field needs to pay more attention to systems-level efficiency in RL training, not just algorithmic advances.

The empirical results support this: at Chunk-8K, training achieves a 42% reduction in total step time while slightly improving AIME accuracy (34.79 vs. 32.91 on AIME 2024). This means the algorithm is not just faster — it's better, possibly because the chunking acts as a regularizer that prevents the model from overfitting to specific trajectory lengths. Whether this regularization effect is real and general is an open question, but the paper's data suggests that efficiency and performance need not trade off against each other when the bottleneck is correctly diagnosed.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses a suite of widely-adopted open-source benchmarks covering knowledge, reasoning, and code. For knowledge: MMLU (Hendrycks et al., 2020), CMMLU (Li et al., 2024a), CEval (Huang et al., 2023). For general reasoning: BigBench Hard (BBH) (Suzgun et al., 2023). For mathematical reasoning: GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), and AIME (MAA). For code reasoning: MBPP (Austin et al., 2021), HumanEval (Chen et al., 2021), LiveCodeBench (LCB) (Jain et al., 2024), and MultiPL-E. The evaluation framework is OpenCompass (Contributors, 2023). For the reasoning model MiniCPM4.1, the paper also reports MMLU-Redux and IFEval. Long-context evaluation uses the RULER benchmark (Hsieh et al., 2024), specifically the Needle-in-a-Haystack (NIAH) tasks at 32K and 128K context lengths.

• Base model(s). The paper develops two model sizes: MiniCPM4-0.5B and MiniCPM4-8B, both pre-trained primarily on Chinese and English corpora using the InfLLM v2 sparse attention architecture. MiniCPM4.1 is a hybrid reasoning variant of the 8B model, trained with additional long chain-of-thought SFT and reinforcement learning, capable of operating in both reasoning and non-reasoning modes. The paper also reports results for MiniCPM4.1 with sparse attention enabled, to measure the accuracy impact of the efficiency mechanism. For the RL efficiency experiments (Table 5), the paper uses DeepSeek-R1-Distill-Qwen-1.5B as the base model for evaluating the chunk-wise rollout strategy on the DAPO dataset — this is a different model from MiniCPM4, used specifically to validate the training algorithm rather than the final model.

• Metrics. All standard evaluation metrics are accuracy-based: for knowledge benchmarks (MMLU, CMMLU, CEval), accuracy is measured as the percentage of questions answered correctly. For mathematical reasoning (GSM8K, MATH500, AIME), the metric is exact-match accuracy of the final answer. For code benchmarks (HumanEval, MBPP), the metric is pass@1 — the fraction of problems solved correctly on the first attempt. LiveCodeBench and MultiPL-E use their standard evaluation protocols. Long-context understanding on RULER uses the weighted average accuracy across subtasks (NIAH-S, NIAH-MK, NIAH-MV, NIAH-MQ, QA1, QA2, VT, CWE, FWE). Efficiency evaluation uses throughput (tokens per second) for both prefilling and decoding on specified hardware (Jetson AGX Orin and RTX 4090). For the survey generation application (Section 6.1), the paper uses GPT-4o as a judge to evaluate Relevance, Coverage, Depth, Novelty, and FactScore on the SurveyEval dataset.

• Baselines. For MiniCPM4-0.5B: Qwen3-0.6B (Yang et al., 2025), Llama3.2-1B (Dubey et al., 2024), Gemma3-1B (Team et al., 2025). These models have 0.6B to 1B parameters and are "well-trained with trillions of tokens and trained with knowledge distillation." For MiniCPM4-8B: Qwen3-8B (Yang et al., 2025), GLM4-9B (GLM et al., 2024), Gemma3-12B (Team et al., 2025), LLaMA3.1-8B (Dubey et al., 2024), Phi4-14B (Abdin et al., 2024). For MiniCPM4.1-8B (reasoning model): Qwen3-8B, R1-Qwen3-8B, GLM-Z1-9B, MiMo-0530-7B, Nemotron-Nano-v2-9B. For the efficiency evaluation: Llama3-8B, GLM4-9B, Qwen3-8B, all evaluated on the same hardware. For the RL training efficiency ablation: a vanilla rollout baseline where the model generates complete responses for each query in a single pass.

• Generation budget / compute accounting. Standard evaluation uses greedy decoding or single-pass generation — the metrics are accuracy scores, not compute-normalized comparisons. The key compute-accounting dimension is training data volume (reported in trillions of tokens): MiniCPM4-8B uses 8.3T tokens (7T stable phase + 1T annealing phase + 0.3T long-context extension) compared to Qwen3-8B's 36T tokens and Qwen3-0.6B's 36T tokens. The paper explicitly frames this as the efficiency comparison: matching or exceeding accuracy with a fraction of the training data. For RL experiments (Table 5), compute is measured in normalized training step time and sampling time per step, with the vanilla rollout baseline normalized to provide relative comparisons. For inference efficiency (Figure 1), the metric is throughput (tokens/second) on specific hardware across sequence lengths from 32K to 128K.

• Cross-validation / statistical protocol. The paper does not report confidence intervals, standard deviations, or statistical significance tests for benchmark evaluations. For the AIME evaluations of the RL strategy (Table 5), the paper reports "the average performance over 16 independent runs" to account for variance. For the survey generation evaluation (Table 12), the test set consists of 20 examples from SurveyEval. For classifier training verification, the paper uses the Lighteval library in a zero-shot setting with no mention of statistical protocols. This is a notable methodological gap — without confidence intervals, it is difficult to assess whether the reported accuracy differences (often 1-3 percentage points on individual benchmarks) are statistically reliable, particularly given the small per-benchmark sample sizes (e.g., HumanEval has 164 problems, GSM8K has 1,319 test examples, but many knowledge benchmarks have 5-15K questions).

Main Quantitative Results

Standard Evaluation: MiniCPM4 vs. Similar-Sized Models

The headline result is in Table 8: MiniCPM4-0.5B achieves an average score of 52.99 across the standard evaluation suite, compared to 44.93 for Qwen3-0.6B, 34.76 for Llama3.2-1B, and 42.28 for Gemma3-1B. This represents an 8.06 point advantage over Qwen3-0.6B and a 10.71 point advantage over Gemma3-1B, despite having fewer parameters (0.5B vs. 0.6B and 1B) and using substantially less training data (1T tokens vs. 36T for Qwen3-0.6B). The largest individual gains are on knowledge-intensive benchmarks: MMLU (55.55 vs. 42.95 for Qwen3-0.6B, a 12.6 point advantage), CMMLU (65.22 vs. 42.05, a 23.17 point advantage), and CEval (66.11 vs. 45.53, a 20.58 point advantage). On reasoning benchmarks, the pattern is mixed: MiniCPM4-0.5B outperforms on BBH (49.87 vs. 28.32 for Qwen3-0.6B) and HumanEval (46.34 vs. 40.85), but underperforms on GSM8K (52.08 vs. 61.71 for Qwen3-0.6B) and MATH500 (29.60 vs. 50.20).

For MiniCPM4-8B, the average score is 81.13, compared to 80.55 for Qwen3-8B (+0.58), 74.78 for GLM4-9B (+6.35), 76.14 for Gemma3-12B (+4.99), 61.49 for LLaMA3.1-8B (+19.64), and 78.47 for Phi4-14B (+2.66). The key comparison is with Qwen3-8B: MiniCPM4-8B achieves comparable overall performance (81.13 vs. 80.55) while using only 22% of the training data (8T vs. 36T tokens). On individual benchmarks, MiniCPM4-8B shows clear advantages on C-Eval (81.36 vs. 80.35), CMMLU (80.62 vs. 77.58), and BBH (76.73 vs. 69.43), while Qwen3-8B leads on MMLU (77.55 vs. 75.83), GSM8K (93.25 vs. 91.51), and MATH500 (83.20 vs. 78.60). The code benchmarks are nearly identical: HumanEval 85.98 vs. 85.37, MBPP 77.04 vs. 78.99.

The paper emphasizes that MiniCPM4 achieves these results without knowledge distillation, unlike Qwen3, Llama3.2, and Gemma3, which all "employ knowledge distillation training strategies, using larger teacher models to guide the training of end-side models" (Section 5.2). This is presented as evidence that "high-quality data and efficient learning algorithms" can substitute for expensive teacher supervision.

Deep Reasoning Evaluation: MiniCPM4.1 vs. Reasoning Models

Table 9 reports results for MiniCPM4.1-8B (the hybrid reasoning model) on an extended benchmark suite including AIME (2024 and 2025), LiveCodeBench, and instruction-following (IFEval). The overall average for MiniCPM4.1 with full attention is 80.17, compared to 79.72 for R1-Qwen3-8B (+0.45), 78.02 for Qwen3-8B (+2.15), 75.27 for GLM-Z1-9B (+4.90), 75.02 for MiMo-0530-7B (+5.15), and 76.54 for Nemotron-Nano-v2-9B (+3.63). MiniCPM4.1 leads R1-Qwen3-8B on knowledge benchmarks (MMLU 86.38 vs. 85.36, MMLU-Redux 86.41 vs. 86.25, CMMLU 84.94 vs. 80.53, CEval 84.38 vs. 84.44) and on BBH (82.40 vs. 76.99), but trails on mathematical reasoning (AIME 2024: 83.33 vs. 83.33 tied; AIME 2025: 73.33 vs. 75.21; MATH500: 95.60 vs. 97.20) and code (LCB-v5: 58.68 vs. 62.87; LCB-v6: 52.00 vs. 53.14).

Critically, the paper reports the sparse attention variant of MiniCPM4.1 in a separate column: the overall average drops from 80.17 to 79.69, a 0.48 point decrease. The knowledge benchmarks are essentially unchanged (MMLU 86.66 vs. 86.38 — slightly higher with sparse attention; MMLU-Redux 86.05 vs. 86.41; CMMLU 84.72 vs. 84.94; CEval 85.75 vs. 84.38). Mathematical reasoning shows minor differences (AIME 2024: 80.83 vs. 83.33; AIME 2025: 72.08 vs. 73.33; MATH500: 97.40 vs. 95.60 — slightly better with sparse attention). The paper's interpretation is that "sparse attention has negligible impact on model performance while providing significant computational efficiency gains" (Section 5.2).

Long-Context Evaluation

The long-context evaluation uses the RULER benchmark with two configurations: the full attention model and the sparse attention variant, both at 32K context length (Table 10). The full attention model achieves a weighted average of 88.93 across all RULER subtasks. The sparse attention variant achieves 85.84, a 3.09 percentage point decrease. The largest gaps are on NIAH-MK (100.00 vs. 87.33, a 12.67 point drop) and NIAH-MQ (99.50 vs. 98.50, a 1.0 point drop). On NIAH-S, both variants achieve 100.00. On the more complex subtasks, the differences are modest: QA2 (54.00 vs. 56.00 — sparse attention actually performs better), CWE (62.60 vs. 60.40), FWE (87.33 vs. 87.33 — tied). The paper characterizes this as "maintained competitive performance" with "a modest 3.09 percentage point decrease."

Figure 7 extends this to 128K context using the Needle-in-a-Haystack (NIAH) task specifically. The paper states that MiniCPM4 "achieve[s] 100% accuracy on the needle in a haystack task" at 128K, with each token attending to only approximately 6K context tokens — a sparsity of 5%. This is presented as evidence that the sparse attention mechanism generalizes to context lengths 4× longer than the training context (32K training vs. 128K evaluation), enabled by YaRN position encoding extrapolation.

Efficiency Evaluation: Throughput on End-Side Hardware

Figure 1 presents the headline efficiency result: MiniCPM4-8B achieves approximately 7× decoding acceleration over Qwen3-8B on Jetson AGX Orin at 128K sequence length. The figure shows throughput (tokens/second) for four models (Llama3-8B, GLM4-9B, Qwen3-8B, MiniCPM4-8B) across sequence lengths from 32K to 128K, with separate curves for prefilling and decoding on two hardware platforms (Jetson AGX Orin and RTX 4090).

The key findings from Figure 1: (1) MiniCPM4-8B consistently achieves higher throughput than all baseline models across all sequence lengths and both hardware platforms, for both prefilling and decoding. (2) The throughput advantage grows with sequence length — at shorter sequences, the gap is smaller, but as sequence length increases to 128K, MiniCPM4's throughput degrades much more slowly than the dense baselines. This is attributed to the sparse attention mechanism maintaining constant computational cost per token for the selected blocks, while dense attention costs grow linearly with sequence length. (3) On Jetson AGX Orin (the more constrained end-side chip), the decoding speedup over Qwen3-8B at 128K is approximately 7× — the paper's headline claim. On RTX 4090, the speedup is smaller but still substantial (exact numbers not quoted in the text, visible in Figure 1).

The paper does not report separate ablation results isolating the contributions of sparse attention, FR-Spec speculative decoding, and P-GPTQ quantization to the total speedup, making it impossible to determine how much of the 7× gain comes from the architecture vs. the inference system optimizations.

RL Training Efficiency: Chunk-wise Rollout

Table 5 reports the efficiency of the chunk-wise rollout strategy compared to vanilla rollout, using DeepSeek-R1-Distill-Qwen-1.5B trained on the DAPO dataset for 150 steps on 64 A800 GPUs. All timing values are normalized to the vanilla baseline (vanilla = 488.57 for step time, 392.61 for sampling time). Chunk-4K reduces total step time by 42% (281.27 vs. 488.57) and sampling time by 62% (148.14 vs. 392.61), while AIME 2024 accuracy is essentially unchanged (32.71 vs. 32.91) and AIME 2025 improves slightly (26.04 vs. 25.21). Chunk-8K achieves the best accuracy (34.79 on AIME 2024, 26.67 on AIME 2025) while reducing step time by 41% (286.31) and sampling time by 56% (173.97). Chunk-16K reduces step time by only 26% (360.79) and sampling time by 36% (250.88), with comparable accuracy (32.50 on AIME 2024, 25.63 on AIME 2025).

The paper notes that while Chunk-4K further reduces sampling time compared to Chunk-8K (148.14 vs. 173.97), the total step time is nearly identical (281.27 vs. 286.31), because "the smaller chunk size, while alleviating sampling bottlenecks, introduces more frequent log probability computations for chunk-level importance sampling" (Section 3.2.4). This is identified as a tradeoff for future optimization.

Application Evaluations

Survey generation (Section 6.1.3, Table 12). MiniCPM4-Survey (8B, fine-tuned on the survey generation task) achieves an average content quality score of 3.50 (across Relevance, Coverage, Depth, Novelty), matching OpenAI Deep Research (driven by GPT-4o) at 3.50 and outperforming Webthinker (driven by QwQ-32B) at 3.13 and AutoSurvey (driven by Gemini-2.0-Flash-Thinking) at 3.16. On faithfulness (FactScore), MiniCPM4-Survey achieves 68.73, substantially higher than Naive RAG (43.68) and AutoSurvey (46.56). The ablation without RL ("w/o RL") scores 3.11 content quality and 50.24 FactScore, demonstrating that RL training contributes approximately +0.39 to content quality and +18.49 to FactScore. However, MiniCPM4-Survey still trails OpenAI Deep Research on Coverage (3.70 vs. 3.95) and matches on Novelty (3.00 vs. 3.00).

MCP tool use (Section 6.2.3, Table 13). MiniCPM4-MCP (8B) achieves a sample-weighted average accuracy of 88.3% for function name selection, 76.1% for parameter name selection, and 51.2% for parameter value selection across 16 MCP servers. This compares to GPT-4o (80.2%, 70.2%, 49.1%) and Qwen3-8B (83.5%, 67.7%, 43.8%). MiniCPM4-MCP outperforms both baselines on all three metrics, with the largest advantage on parameter name accuracy (76.1% vs. 70.2% for GPT-4o). The paper notes that Qwen3-8B "possesses the basic MCP Tool calling capability" but struggles with specialized tools (arXiv, Airbnb) where it "tends to apply prior knowledge from other tools when generating parameter names and passing parameter values, without adequately adjusting or adapting to the specific requirements of the given MCP tool." MiniCPM4-MCP, having been trained on MCP-specific demonstrations, avoids this transfer gap.

Variation across servers is substantial: on Airbnb, MiniCPM4-MCP achieves 96.4% function accuracy but only 50.0% parameter value accuracy; on Calculator, it achieves 100% function and parameter accuracy but 6.67% parameter value accuracy; on GitHub, it achieves only 62.8% function accuracy and 17.1% parameter value accuracy. The parameter value accuracy is consistently the weakest metric across all models and servers, suggesting that generating correct argument values from tool descriptions remains a fundamental challenge.

Ablation Studies and Robustness Checks

RL component ablation in survey generation (Table 12, "w/o RL" row): Removing the reinforcement learning phase and using only SFT reduces content quality from 3.50 to 3.11 (a 0.39 point decrease) and FactScore from 68.73 to 50.24 (an 18.49 point decrease). The largest degradation is in Novelty (3.00 → 2.25) and Depth (3.85 → 3.30). This confirms that the multi-stage RL training (chapter-level then survey-level) is essential for the model's ability to produce comprehensive, in-depth, and faithful surveys. However, this is a single ablation on a single application task — it does not isolate the specific contributions of chapter-level vs. survey-level RL, nor does it test whether the RL gains transfer to other long-form generation tasks.

Sparse vs. full attention ablation for reasoning (Table 9, "Sparse" column): Enabling InfLLM v2 sparse attention on MiniCPM4.1-8B reduces overall average from 80.17 to 79.69, a 0.48 point decrease. The largest individual drops are on AIME 2024 (83.33 → 80.83, -2.50), AIME 2025 (73.33 → 72.08, -1.25), and HumanEval (95.73 → 91.46, -4.27). Some benchmarks show minor improvements: MATH500 (95.60 → 97.40, +1.80), MMLU (86.38 → 86.66, +0.28), CEval (84.38 → 85.75, +1.37). This mixed pattern — sparse attention sometimes slightly outperforming full attention — is unexpected and not fully explained. The paper presents this as evidence that "sparse attention has negligible impact on model performance," but the -4.27 drop on HumanEval is material and suggests that code generation may be more sensitive to sparse attention than other tasks.

Sparse vs. full attention ablation for long context (Table 10): On the RULER benchmark at 32K, enabling sparse attention reduces weighted average accuracy from 88.93 to 85.84, a 3.09 point decrease. The largest individual drops are on NIAH-MK (100.00 → 87.33, -12.67) and NIAH-MQ (99.50 → 98.50, -1.00). The NIAH-MK (multi-key) task requires the model to locate multiple needles in the haystack simultaneously, and the 12.67 point drop suggests that sparse attention's block selection mechanism may occasionally miss one of the multiple relevant locations. However, on NIAH-MV (multi-value), sparse attention actually improves (91.50 → 94.50, +3.00). The paper does not analyze why some multi-needle variants degrade while others improve.

Chunk size ablation for RL efficiency (Table 5): Varying the chunk size (4K, 8K, 16K tokens) reveals a non-monotonic relationship with accuracy: Chunk-8K achieves the best AIME scores (34.79 on AIME 2024, 26.67 on AIME 2025), outperforming both Chunk-4K (32.71, 26.04) and Chunk-16K (32.50, 25.63). This suggests that there is an optimal granularity for chunking — too small (4K) may fragment reasoning trajectories excessively, while too large (16K) provides less load-balancing benefit. The paper does not explore what determines this optimum or whether it depends on the problem distribution.

Quantization method ablation (Table 7): Comparing five quantization variants of MiniCPM4-8B (all INT4 per-group, all linear layers quantized): FP16 baseline achieves 75.58 average accuracy across 7 benchmarks. Standard GPTQ drops to 74.31 (-1.27). P-GPTQ (prefix-aware Hessian) achieves 74.76 (-0.82). S-GPTQ (GPTQ with AWQ smoothing) achieves 74.63 (-0.95). S-P-GPTQ (combining prefix-aware Hessian with AWQ smoothing) achieves 74.91 (-0.67). The incremental benefit of prefix awareness over standard GPTQ is 0.45 points (74.76 vs. 74.31); the incremental benefit of smoothing over prefix awareness is a further 0.15 points (74.91 vs. 74.76). The total degradation from FP16 to the best quantized variant is 0.67 points, which the paper presents as evidence that P-GPTQ "achieves superior performance among quantized methods, exhibiting the smallest performance degradation." However, the ablation does not test varying the number of calibration sequences (fixed at 1,024) or the prefix exclusion length (fixed at $s = 4$), so it is unclear whether these values are optimal or merely ad-hoc.

Ternary model scaling (Table 6): BitCPM4-1B (ternary, trained via QAT from a high-precision checkpoint) achieves 59.24 on MMLU, 60.80 on GSM8K, and 56.03 average across 8 benchmarks. This outperforms BitNet-2B (trained from scratch in ternary) which achieves 53.17 on MMLU, 58.63 on GSM8K, and 43.31 average — BitCPM4 is 12.72 points higher on average despite having half the parameters and using only ~10% of the training tokens (350B vs. 4T). BitCPM4-0.5B (39.84 average) underperforms Qwen3-0.6B (44.93 average) in BF16, suggesting that extreme quantization of very small models incurs a larger relative penalty. The paper attributes this to a scaling law for quantization effectiveness. However, the comparison between BitCPM4-1B and BitNet-2B is confounded by model size (1B vs. 2B) and training data differences, making it unclear whether the advantage comes from the QAT method, the base model quality, or both.

ModelTunnel v2: $\mu$P vs. StepLaw (Table 3): Comparing hyperparameter optimization methods on small-scale models (150M to 700M parameters, 4B to 100B training tokens), $\mu$P and StepLaw (vanilla architecture) produce comparable results. Training loss and ScalingBench scores are very close across all model sizes and token budgets. The paper notes that StepLaw shows "slightly more instances of advantage" but the differences are "minimal" with "neither approach exhibiting consistently stable superiority." The GPU-hour cost comparison heavily favors $\mu$P (32 GPU-hours for $\mu$P search vs. 1M GPU-hours for StepLaw reproduction costs — the latter figure likely includes the cost of training models at multiple scales to fit the scaling law, not the per-experiment cost). This ablation is primarily about cost, not quality, and the paper's conclusion is pragmatic: $\mu$P provides comparable hyperparameter quality at a fraction of the search cost.

Data filtering quality comparison (Table 2): UltraFineWeb-en (45.89 average) outperforms FineWeb (42.28, +3.61) and FineWeb-edu (44.56, +1.33) on English benchmarks, while UltraFineWeb-zh (35.16 average) outperforms Chinese-FineWeb (33.18, +1.98) and Chinese-FineWeb-edu-v2 (34.55, +0.61) on Chinese benchmarks. These differences are based on training 1.2B-parameter models on approximately 100B tokens each and evaluating on standard benchmarks in a zero-shot setting. The paper does not report variance across training seeds, so it is unclear whether the observed differences (which are often 1-3 points on individual benchmarks) are statistically significant or within training noise.

Survey generation baseline comparison (Table 12): MiniCPM4-Survey (8B, after RL) achieves 3.50 content quality, matching OpenAI Deep Research (GPT-4o) and substantially outperforming Webthinker + QwQ-32B (3.13) and AutoSurvey + Gemini-2.0-Flash-Thinking (3.16). The factuality advantage is even larger (FactScore 68.73 vs. 46.56 for AutoSurvey). However, the baseline models use substantially larger or different backbone LLMs (GPT-4o, QwQ-32B, Gemini-2.0-Flash-Thinking), making this not a controlled model-size comparison. The paper's claim is that MiniCPM4-Survey is competitive despite being a smaller on-device model, but the comparison is not FLOPs-matched or parameter-matched — it is a qualitative demonstration of capability rather than a controlled efficiency experiment.

MCP tool use: per-server performance (Table 13): MiniCPM4-MCP outperforms GPT-4o and Qwen3-8B on aggregate metrics, but per-server variation is extreme. On Airbnb, function accuracy is 96.4% but parameter value accuracy is 50.0%. On Calculator, function and parameter name accuracy are 100% but parameter value accuracy is 6.67%. On GitHub, all three metrics are below 63%. This suggests that MiniCPM4-MCP has learned the surface-level patterns of tool calling (selecting the right function, naming parameters correctly) but struggles with generating semantically correct argument values — a deeper semantic understanding challenge that persists even after training on MCP demonstrations.

Critical Assessment

Claim 1: "MiniCPM4-8B achieves comparable performance to Qwen3-8B while using only 22% of the training data."

This is the paper's central efficiency claim, and the evidence in Table 8 supports it but with important nuance. The overall average scores are indeed comparable (81.13 vs. 80.55), and MiniCPM4 leads on Chinese knowledge benchmarks (CMMLU: 80.62 vs. 77.58; CEval: 81.36 vs. 80.35) and BBH (76.73 vs. 69.43), while trailing on MMLU (75.83 vs. 77.55) and some reasoning benchmarks (GSM8K: 91.51 vs. 93.25; MATH500: 78.60 vs. 83.20). The "comparable" claim holds in aggregate, but the performance profile is different — MiniCPM4 appears stronger on knowledge tasks and weaker on mathematical reasoning. Someone interested specifically in math reasoning would find Qwen3 noticeably better.

The 22% figure (8T / 36T tokens) assumes that training data volume is the only resource difference. However, Qwen3-8B also uses knowledge distillation (deploying a larger teacher model during training), which MiniCPM4 does not. The paper mentions this but does not quantify how much of Qwen3's performance comes from distillation vs. raw data volume. If a significant fraction of Qwen3's advantage on math comes from distillation rather than data volume, then the "22% of data" comparison overstates the efficiency gain — MiniCPM4 may be recovering gains that Qwen3 achieved through a different mechanism.

A more fundamental limitation: the comparison is on a single suite of benchmarks (8-9 metrics) with no measurement of variance. The differences between MiniCPM4 and Qwen3 on individual benchmarks are often 1-2 percentage points. Without confidence intervals or multiple training seeds, it is impossible to determine whether MiniCPM4 genuinely matches Qwen3 or merely falls within its error bars. The paper's framing ("comparable performance") is reasonable given the data shown, but a rigorous head-to-head would require multiple evaluation runs and statistical testing.

Claim 2: "MiniCPM4 demonstrates a 7× decoding speedup over Qwen3-8B on 128K sequences on end-side devices."

Figure 1 shows this result for the specific hardware configuration (Jetson AGX Orin) and sequence length (128K), and the visual evidence supports a large gap. However, several important caveats apply. First, the speedup is not decomposed — it combines the effects of InfLLM v2 sparse attention, FR-Spec speculative decoding, P-GPTQ quantization, and the CPM.cu inference framework. A reader cannot determine how much of the 7× comes from the architecture (which is the paper's primary innovation claim) vs. the inference system (which any model could use). Qwen3-8B with the same speculative decoding, quantization, and optimized kernels might close a substantial portion of the gap.

Second, the speedup is measured on 128K sequences specifically. At shorter sequences, the gap is smaller (visible in Figure 1). The paper's claim is technically correct for the stated configuration, but the headline 7× figure represents the most favorable case (longest sequence length, most constrained hardware), not an average across typical use cases. For users deploying on shorter documents or more powerful hardware (RTX 4090), the practical speedup may be substantially smaller.

Third, the efficiency evaluation compares MiniCPM4 against baseline models that use dense attention with standard inference frameworks. This is a reasonable baseline for demonstrating the value of the integrated system, but it conflates multiple innovations. An ablation that gave Qwen3 the same speculative decoding, quantization, and optimized kernel framework would isolate the architectural contribution of InfLLM v2 but is not provided.

Claim 3: "InfLLM v2 enables 95% sparsity on 128K sequences with negligible accuracy loss."

The evidence for this claim is distributed across multiple results. The sparsity figure is stated in Section 5.3: "each token only requires the model to attend 6K context tokens, which means on 128K context, the sparsity of MiniCPM4 is only 5%." The accuracy impact is reported in Table 9 (0.48 point decrease on the reasoning model) and Table 10 (3.09 point decrease on RULER). Whether these decreases are "negligible" depends on the use case. For standard QA benchmarks, a 0.48 point drop is indeed negligible. For RULER, a 3.09 point drop is more substantial, and the 12.67 point drop on the multi-key needle task (NIAH-MK) is clearly not negligible — it represents a meaningful degradation in the model's ability to locate multiple pieces of information in a long document.

The paper's framing ("sparse attention has negligible impact on model performance") glosses over this variability. The impact is negligible on average across a diverse benchmark suite, but is non-negligible on specific tasks (multi-key retrieval, code generation) that may matter for certain applications. A more precise claim would be: "Sparse attention incurs minimal degradation on knowledge benchmarks but can reduce accuracy on complex multi-hop retrieval and code generation tasks by 1-5 points."

Claim 4: "Chunk-wise rollout reduces RL sampling time by 42-62% with no accuracy degradation."

Table 5 supports this claim for the specific configuration tested (DeepSeek-R1-Distill-Qwen-1.5B, DAPO dataset, 150 steps, 64 A800 GPUs). Chunk-8K reduces step time by 41% and sampling time by 56% while achieving slightly better accuracy than the vanilla baseline (34.79 vs. 32.91 on AIME 2024). The evidence is strong for this configuration. However, the paper does not test: (1) whether the result holds for the actual MiniCPM4.1 model rather than a Qwen-derived model, (2) whether it scales to larger training runs (150 steps is relatively short), (3) whether it holds for different problem distributions (the RL data is heavily math and code), or (4) whether the accuracy improvement is a real regularization effect or a statistical fluctuation.

The claim about "no accuracy degradation" is also conditional on using the stabilization techniques (chunk-level importance sampling, dual-clip, KL regularization, garble filter). Without these, the paper implies that training would destabilize, but no ablation removing individual stabilization components is provided. The reader cannot assess which techniques are essential and which are optional.

Claim 5: "UltraClean filtering improves data quality, enabling better models with fewer tokens."

Table 2 provides evidence: UltraFineWeb-en improves average benchmark scores by 3.61 points over FineWeb using 1.2B models trained on 100B tokens. The verification is on proxy models, not the final 8B model. The assumption that data quality rankings transfer from 1.2B to 8B models (and from 100B to 8T token budgets) is plausible under the $\mu$P framework but is not empirically validated in this paper. A complete validation would require training the 8B model on unfiltered FineWeb and comparing — an experiment the paper understandably did not run given its cost, but which leaves the extrapolation claim untested.

Additionally, the data filtering benefit is measured on models trained with 100B tokens, but MiniCPM4-8B uses 8T tokens — 80× more. The relative benefit of data filtering may diminish at larger token budgets, as models become more data-saturated and the marginal value of additional quality decreases. The paper does not discuss this saturation effect or attempt to bound how much of the 22% data efficiency is attributable to filtering vs. other factors (architecture, training recipe, MTP objective).

What the experiments do not demonstrate:

• The relative contribution of each efficiency innovation to the total speedup. The paper presents MiniCPM4 as an integrated system, which is its stated philosophy, but this makes it impossible to determine whether the gains come primarily from sparse attention, data filtering, or inference system optimization. An ablation of Qwen3-8B with MiniCPM4's inference framework would isolate the architecture contribution; an ablation of MiniCPM4 with Qwen3's training data would isolate the data contribution. Neither is provided.

• Generalization to non-English, non-Chinese languages. The model is "primarily pre-trained on Chinese and English corpora" (Section 5.1), and the evaluation uses Chinese and English benchmarks. Performance on other languages is unknown.

• Generalization to tasks beyond the evaluation suite. The benchmark suite is diverse but still limited to standard academic evaluations. The application evaluations (survey generation, MCP tool use) partially address this but use small test sets (20 examples for surveys) and reference-based evaluation (GPT-4o as judge) that may not capture practical deployment quality.

• Statistical reliability of individual benchmark comparisons. No confidence intervals, standard deviations, or significance tests are reported for any benchmark evaluation. The paper reports precision to two decimal places, implying a level of measurement certainty that is not justified given the small sample sizes of many benchmarks (e.g., HumanEval: 164 problems, MBPP: ~500 problems).

• The cost of difficulty estimation in the data filtering pipeline. The efficient verification strategy reduces cost from 1,200 to 110 GPU-hours per experiment, but the total cost of the iterative classifier refinement (multiple rounds of seed selection, classifier training, and verification) is not reported. The claim that UltraClean enables efficient data filtering is qualitative; the total resource investment is not quantified.

Missing experiments that would strengthen the paper:

• An ablation giving Qwen3-8B the CPM.cu inference framework with InfLLM v2 kernels, speculative decoding, and quantization, to isolate the architecture contribution from the inference system contribution.
• Training the 8B model on unfiltered FineWeb (or a random subset of the same size) to directly measure the contribution of UltraClean filtering at the target model scale.
• Evaluating MiniCPM4 on a broader set of languages and tasks (translation, summarization, dialogue) to establish the generality of the efficiency-quality tradeoff.
• Multiple training runs with different seeds to establish the variance of the reported benchmark scores and enable statistical comparison with baselines.
• Latency (time-to-first-token) evaluation in addition to throughput, since latency is often the binding constraint for interactive end-side applications.
• Memory consumption measurements (GPU RAM, CPU RAM) for the quantized and sparse models on target devices, since memory is often the binding constraint for deployment, not raw throughput.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted For and Makes the Approach Impractical at Present

The entire compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so — generating 2,048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. At 2,048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The paper suggests future work on training models to predict difficulty directly from question text, but no such model is developed or evaluated. Until this gap is closed, the 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain.

6.2 Hard Problems Remain Essentially Unsolved — There Is No Path Forward When the Base Model Lacks the Capability

Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%.

This is a fundamental limitation: test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. The paper is candid about this (Section 7 takeaway box), but it means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path. This is not merely a performance gap — it is a hard capability ceiling that no test-time strategy can breach, which constrains the approach to domains where the base model already possesses latent competence.

6.3 The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate

As noted in Section 6.1, approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction: the model only sees incorrect-to-correct trajectories during training, so it has no signal for what to do when the current answer is already correct. The paper mitigates this with majority voting or verifier-based selection across the chain, but these are imperfect patches — the selection mechanism must correctly identify the correct answer among a chain that includes both correct and (subsequently) incorrect revisions, and there is no guarantee that the correct answer will be the highest-scoring one.

A more principled solution — such as training the model to recognize when no revision is needed — is not explored. The ReST^EM experiment (Appendix K, Figure 16) further highlights the fragility of revision training: attempting to optimize with RL-style training caused performance to degrade substantially with sequential revisions, suggesting the revision approach is sensitive to training methodology in ways that are not fully understood. The positive results depend on specific choices (offline data construction, edit-distance-based pairing) that may not transfer to other settings, and the 38% reversion rate represents a fundamental tension the paper does not resolve: the model is incentivized to always revise, but sometimes the best revision is no revision.

6.4 Revisions and Search Are Studied Independently, Never Combined — The Results Are a Lower Bound

The paper studies two complementary axes — PRM search and iterative revisions — but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

This is a significant gap because the two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better candidates), while PRM search improves candidate selection (finding the best among generated candidates). Applying beam search to revision model outputs — or using the PRM to guide which revisions to pursue — could yield gains beyond either method alone. The current results therefore represent a lower bound on what a fully integrated system could achieve. For a practitioner, the open question is whether combining them would compound the gains (if the mechanisms are complementary and their benefits add) or saturate (if both address the same underlying bottleneck of base model capability). The paper provides no evidence either way, leaving a clear path to further improvement unexplored.

6.5 The FLOPs-Matched Comparison Uses a Weak Baseline That Understates Pretraining's Potential

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This matters because a Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1) may shrink or reverse against a properly compute-optimal larger model. Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search. Giving the larger model even a modest test-time compute budget (say, best-of-8) would create a much stronger baseline that is never tested.

This does not invalidate the paper's central claim that test-time compute can be more efficient than pretraining in certain regimes, but it does mean the quantitative advantage is overstated relative to what a well-optimized pretraining pipeline would achieve. A practitioner deciding between investing in a larger model vs. inference-time strategies needs to know that the comparison is tilted in favor of the latter.

6.6 No Accounting for Latency or Wall-Clock Time Makes the Results Incomparable to Interactive Deployment Scenarios

The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores latency. Sequential revisions are inherently serial — each revision depends on the previous one — while parallel best-of-N can be executed simultaneously with sufficient hardware. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously.

For latency-sensitive applications (interactive assistants, real-time decision-making), the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. The paper does not discuss this tradeoff at all. A deployment engineer reading this work would need to know: for a given latency budget (e.g., 2 seconds), which strategies are actually viable, and how much of the theoretical 4× efficiency gain survives when latency constraints are imposed? The paper provides no guidance on this question, making the results difficult to translate into practical deployment decisions where both throughput and latency matter.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around small language models from a focus on cheap-and-weak toward a capable-and-deployable framing. Prior to MiniCPM4, the dominant assumption — reinforced by scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) and the visible performance of models like Phi, Gemma, and Qwen — was that small models inevitably sacrifice substantial capability relative to their larger counterparts, and that the primary engineering challenge is minimizing how much capability you lose per parameter you cut. MiniCPM4 challenges this assumption at the architectural level: it demonstrates that a 0.5B-parameter model can outperform a 1B-parameter model (Gemma3-1B by 10.71 points, Table 8), and an 8B model can match a 12B or 14B model (Gemma3-12B, Phi4-14B), while using dramatically less training data than comparably-performing dense models of the same size (22% of Qwen3-8B's tokens).

The conceptual shift is not merely "data quality matters" — that has been widely acknowledged since at least Chinchilla. It is that architecture, data quality, training algorithms, and inference systems form a coupled efficiency system where optimizing any component in isolation leaves most of the potential gain unrealized. The paper does not present InfLLM v2 as a drop-in replacement for dense attention that any model could adopt; it presents it as part of an integrated stack where the sparse attention kernel design (CPM.cu), the training objective (multi-token prediction producing draft heads for speculative decoding), the data filtering (UltraClean ensuring that the 5% of tokens attended to are the most informative ones), and the deployment system (ArkInfer abstracting across hardware) all cohere around the same goal.

This reframing has a concrete pragmatic consequence: it changes what "state-of-the-art for small models" means. A paper proposing a new small model cannot now just report accuracy on standard benchmarks — it must also report throughput on representative end-side hardware, memory consumption, and training data volume, because MiniCPM4 has demonstrated that these are all first-order dimensions of model quality, not afterthoughts. The paper's Figure 1, which puts throughput on the same visual plane as accuracy, makes this point visually: efficiency is not a secondary metric but a co-equal dimension of evaluation.

The work also reconciles a tension that has been building in the efficient LLM literature. On one side, there are architecture-first approaches (NSA, MoBA, BitNet) that redesign the model for efficiency but often sacrifice compatibility with standard training pipelines or inference frameworks. On the other side, there are data-first approaches (FineWeb-edu, DataComp-LM) that improve training efficiency but assume a standard dense architecture. MiniCPM4 demonstrates that these are not competing philosophies — they are complementary design dimensions, and the real gains come from optimizing both simultaneously. InfLLM v2 without UltraClean would waste its sparse attention budget on noisy tokens; UltraClean without InfLLM v2 would still be bottlenecked by quadratic attention on long sequences. The paper's contribution is to show that the whole is greater than the sum of its parts, and that this gestalt is measurable in throughput numbers on actual hardware, not just abstract FLOPs counts.

A more subtle landscape shift concerns the role of knowledge distillation. Several of MiniCPM4's baselines (Qwen3, Llama3.2, Gemma3) employ distillation from larger teacher models. MiniCPM4 explicitly does not, and the paper frames this as evidence that "high-quality data and efficient learning algorithms" can substitute for teacher supervision (Section 5.2). If this claim generalizes, it would change the economics of small model training: distillation requires deploying and running the teacher model, which consumes compute that could otherwise be spent on better data curation or more training tokens. The paper does not provide a FLOPs-matched comparison between distillation and data quality improvement, but the implication is that investing in data curation infrastructure may yield better returns than investing in teacher model compute — a hypothesis that the field should test explicitly.

Follow-Up Research This Work Enables

1. Scaling the data quality verification strategy to truly web-scale corpora with formal cost-benefit analysis. The paper's efficient verification strategy demonstrates that data quality can be evaluated at ~10% of the cost of from-scratch training, but the total cost of the iterative classifier refinement pipeline (multiple rounds of seed selection, classifier training, and verification) is not reported. A direct follow-up would benchmark the end-to-end cost: given a target training budget of T tokens and a raw corpus of size R ≫ T, what is the optimal allocation of compute between (a) running the UltraClean pipeline (including verification iterations) and (b) training the final model? The paper's claim that data quality outweighs data quantity is qualitative; this follow-up would quantify the tradeoff curve, enabling practitioners to decide how much filtering is worth the cost for their specific budget. The experiment would sweep the number of classifier refinement iterations, measure final model quality at a fixed training budget, and report the total GPU-hours including both filtering and training. The hypothesis is that the UltraClean pipeline's verification cost is amortized across the filtering of trillions of tokens and is negligible compared to the training cost savings — but this needs empirical validation.

2. Isolating the contribution of InfLLM v2 sparse attention from inference system optimizations. The paper's headline 7× decoding speedup over Qwen3-8B on 128K sequences (Figure 1) conflates sparse attention, speculative decoding (FR-Spec), quantization (P-GPTQ), and the custom CPM.cu kernel implementation. A critical follow-up would port the CPM.cu inference stack (FR-Spec, P-GPTQ, kernel optimizations) to a standard dense model like Qwen3-8B and measure the speedup attributable purely to the inference system. The residual speedup — any remaining advantage of MiniCPM4 over the inference-optimized Qwen3 — would represent the genuine contribution of InfLLM v2's sparse attention. This experiment is straightforward to implement (since Qwen3 uses a standard dense architecture compatible with speculative decoding and GPTQ) and would definitively answer whether trainable sparse attention is worth the architectural complexity or whether the same gains could be achieved by applying MiniCPM4's inference optimizations to any dense model.

3. Understanding the scaling behavior of sparse attention sparsity with model size and task complexity. InfLLM v2 achieves 95% sparsity on 128K sequences (attending to ~6K out of 128K tokens), but this is reported for the 8B model on a specific task (NIAH needle-in-a-haystack retrieval). The sparsity-accuracy tradeoff likely depends on both model scale and task type. A systematic study would train MiniCPM4 variants at multiple scales (0.5B, 1B, 4B, 8B) with the same InfLLM v2 architecture, vary the sparsity level (by adjusting the number of selected blocks k), and measure accuracy on tasks stratified by difficulty and required context length. The hypothesis from the paper's results is that larger models can tolerate higher sparsity because they learn more efficient key representations, and that retrieval tasks tolerate higher sparsity than multi-hop reasoning tasks (consistent with the 12.67-point drop on RULER NIAH-MK vs. 1.0-point drop on NIAH-MQ in Table 10). Quantifying this relationship would produce a "sparsity scaling law" analogous to precision scaling laws (Kumar et al., 2024), enabling practitioners to choose the optimal sparsity level for their model size and target task distribution.

4. Testing the generalization of data quality optimization as an alternative to knowledge distillation. The paper claims that MiniCPM4 matches Qwen3-8B without using distillation, but this comparison confounds data quality, data volume, architecture, and training recipe. A clean ablation would train two versions of the same base architecture (say, MiniCPM4-1B): one with distillation from a larger teacher (following the Qwen3 recipe), and one with UltraClean-filtered data but no teacher. Both would use identical training compute budgets. The comparison would directly measure whether high-quality data curation can substitute for distillation supervision on a per-FLOP basis. A null result (distillation wins) would suggest that the paper's advantage over Qwen3 may come primarily from architecture or other training factors, not data quality per se. A positive result would validate the paper's implicit claim and shift the field's investment toward data infrastructure rather than teacher model deployment.

5. Combining InfLLM v2 sparse attention with mixture-of-experts for compounding efficiency gains on end-side devices. InfLLM v2 reduces attention complexity; mixture-of-experts (MoE) reduces feedforward complexity. The two are orthogonal and potentially synergistic. However, MoE introduces additional memory overhead for loading multiple expert weights, which is particularly painful on memory-constrained end-side devices. A concrete follow-up would design an "InfLLM-MoE" variant where the sparse attention mechanism's block selection is coordinated with the MoE router — for example, using the same relevance scores that select attention blocks to also select which experts to activate, amortizing the routing computation. The experiment would benchmark throughput and memory consumption on Jetson AGX Orin and RTX 4090, comparing InfLLM-MoE against dense InfLLM and against standard MoE (e.g., DeepSeek-style) at the same total parameter count. The success criterion is whether the combined sparsity (attention + experts) yields a throughput improvement greater than the product of individual improvements, indicating synergy rather than just additive benefit.

6. Adapting the chunk-wise rollout strategy to heterogeneous compute environments and dynamic difficulty. The paper's RL efficiency experiments (Table 5) use 64 homogeneous A800 GPUs. End-side model training is increasingly likely to happen on heterogeneous clusters or even distributed across edge devices (federated learning scenarios). A natural extension would adapt the chunk-wise strategy to heterogeneous environments where different workers have different throughput — the chunk size could be dynamically adjusted per worker based on its generation speed, with faster workers taking larger chunks and slower workers taking smaller ones, maintaining load balance without a fixed global chunk size. The experiment would simulate a heterogeneous cluster with a mix of GPU types, measure the total idle time under vanilla rollout vs. adaptive chunk-wise rollout, and report both training throughput and final model quality. A positive result would make RL training for reasoning practical in academic or small-company settings where GPU resources are heterogeneous and limited.

7. Verifying the robustness of the "sparse attention degrades gracefully" claim under systematic adversarial testing. The paper reports that sparse attention has "negligible impact" on average accuracy (Table 9: -0.48 points) but shows material drops on specific tasks (HumanEval: -4.27, NIAH-MK: -12.67). An adversarial evaluation would construct inputs specifically designed to break InfLLM v2's block selection: documents where critical information is deliberately placed at positions that should score low under the kernel-based relevance mechanism (e.g., buried in semantically unrelated surrounding text, or distributed across block boundaries with the overlapping kernel stride failing to capture the cross-boundary dependency). The experiment would measure whether sparse attention creates systematic failure modes — certain input structures that the model consistently mishandles — that are invisible in aggregate benchmark scores. If such failure modes exist, they would inform deployment guidance (e.g., "do not use sparse attention for tasks requiring fine-grained code understanding across multiple function definitions") and motivate improvements to the relevance scoring mechanism.

Practical Applications and Downstream Use Cases

On-device document processing and survey generation with privacy guarantees. The MiniCPM4-Survey application (Section 6.1) demonstrates that an 8B model can produce literature surveys with content quality matching OpenAI Deep Research (GPT-4o) while achieving substantially higher factual accuracy (FactScore 68.73 vs. 46.56 for AutoSurvey, Table 12). The key deployment scenario is organizations with sensitive documents — legal firms reviewing case law, pharmaceutical companies surveying clinical trial literature, financial institutions analyzing regulatory filings — where uploading proprietary documents to cloud APIs is unacceptable. Running MiniCPM4-Survey on a local workstation with an RTX 4090 provides a fully private, air-gapped survey generation system. The throughput advantage matters here because survey generation is token-intensive (the paper's pipeline involves multiple rounds of retrieval, planning, and chapter-level writing), and the 7× long-context speedup on 128K documents means a survey that might take 20 minutes on a cloud API could complete in ~3 minutes locally, making interactive refinement practical.

Edge-deployed coding assistants with MCP tool integration. MiniCPM4-MCP (Section 6.2) achieves function name accuracy of 88.3% and parameter name accuracy of 76.1% across 16 MCP servers (Table 13), outperforming GPT-4o on both metrics. The deployment scenario is a developer working on an air-gapped machine (defense contractors, embedded systems developers, researchers with proprietary codebases) who needs an AI assistant that can interact with local tools (file systems, version control, build systems, code executors) through the Model Context Protocol. MiniCPM4-MCP running on a Jetson AGX Orin (common in robotics and automotive development environments) could serve as an on-device coding assistant that queries local documentation, executes test suites, and interacts with Git repositories without any network connectivity. The 51.2% parameter value accuracy (Table 13) remains a limitation — the model correctly identifies which tool to call but sometimes provides incorrect arguments — suggesting that this application would benefit from a human-in-the-loop confirmation step for tool invocation, which is natural in a coding assistant context where the developer already reviews AI-generated code.

Hybrid cloud-edge reasoning for latency-sensitive question-answering systems. MiniCPM4.1's hybrid reasoning capability (Table 9: 80.17 average with full attention, 79.69 with sparse attention on the 8B model) enables a tiered deployment architecture where the on-device model handles routine queries with sparse attention for speed, and only escalates to cloud-based larger models for problems that require deep reasoning. The difficulty estimator is implicit in the model's own behavior — queries that the on-device model answers with low confidence (detectable via output token probabilities or generation length in reasoning mode) are routed to the cloud. The key metric from the paper is that the sparse attention variant achieves comparable knowledge benchmark scores to the full attention variant (MMLU: 86.66 vs. 86.38, Table 9) but with dramatically better throughput on long sequences (Figure 1), meaning the on-device model can handle a high volume of factual queries at interactive speeds while reserving the cloud model's capacity for the hard reasoning problems (AIME-level math, which shows a 2.50-point drop with sparse attention). This hybrid architecture could reduce cloud inference costs by 60-80% if the query distribution is skewed toward factual and medium-difficulty reasoning tasks, while maintaining response quality on the long tail of hard problems.