Sparsing Law: Towards Large Language Models with Greater Activation Sparsity

🎯 Pitch

ReLU and SiLU activation functions obey opposite scaling laws for sparsity: more training data makes one sparser but hurts the other, culminating in a 93.52% sparse 2.4B model with 4.1× speedup.

1. Executive Summary

This paper studies how activation sparsity—the existence of substantial weakly-contributed neurons within feed-forward networks of LLMs—can be quantitatively measured, what architectural and training factors influence it, and how to build models that are more sparsely activated and thus computationally efficient. Through systematic experiments on µP Transformer models scaled from 0.1B to 1.2B parameters trained on a diverse text corpus, the authors introduce CETT-PPL-1%, a generalizable metric that binary-searches the CETT hyperparameter (which adaptively sets layer-wise thresholds for identifying weakly-contributed neurons) under a perplexity increase tolerance of just 1%, achieving negligible downstream performance degradation while substantially improving sparsity measurement accuracy over prior methods. The central empirical discovery is a pair of convergent scaling laws: ReLU-activated models follow a decreasing logspace power-law between activation ratio and training data amount (more data yields greater sparsity), while SiLU-activated models follow an increasing vanilla power-law (more data harms sparsity), with ReLU achieving significantly higher sparsity ratios at comparable task performance. The paper also establishes that activation sparsity is largely insensitive to parameter scale under similar width-depth ratios—a finding explained through evidence that neuron specialization patterns are similar across model sizes—and validates its prescriptions by training a 2.4B ReLU-activated model achieving a 93.52% sparsity ratio with 4.1× inference speedup compared to its dense counterpart, though this approach proves effective only when problems fall within the base model's capability range rather than on fundamentally hard tasks requiring capabilities that test-time compute cannot create.

2. Context and Motivation

The Core Problem: We Don't Know How to Measure or Induce Activation Sparsity Systematically

The fundamental question this paper tackles has three interlocking dimensions: how should we measure activation sparsity accurately, what factors control how sparse a model becomes during training, and how can we deliberately build sparser, more efficient LLMs? Activation sparsity—the phenomenon that many neurons in a model's feed-forward networks produce zero or negligible outputs for any given input, contributing almost nothing to the final computation—is widely observed in practice. Models with high activation sparsity offer concrete engineering benefits: when weakly-contributed neurons can be identified and skipped at inference time, the computation associated with their parameters can be saved entirely, yielding speedups without removing any model weights.

This matters for several practical reasons the authors highlight (Section 1):

Inference acceleration: Prior work has demonstrated that exploiting activation sparsity can achieve up to 27.8× inference acceleration on smartphones (Xue et al., 2024, cited in Section 1) by dynamically skipping computation for inactive neurons. The paper's own 2.4B model achieves 4.1× speedup using PowerInfer (Section 6, Appendix L). As LLMs are deployed in resource-constrained environments—edge devices, consumer hardware, real-time systems—activation sparsity becomes a critical efficiency lever.
Training acceleration: Zhang et al. (2024b) showed that activation sparsity persists throughout most of the pre-training process, opening the door to saving computation during training itself, not just inference.
Interpretability: Sparse activation patterns provide a window into how models organize knowledge. Prior work has used neuron activation patterns to interpret model behaviors (Sajjad et al., 2022; Bills et al., 2023), reveal emergent modularity (Zhang et al., 2023), and understand how specialized functions are distributed across neurons.

Despite these substantial benefits, the paper identifies a striking gap: there are few comprehensive, quantitative studies on activation sparsity itself—how to measure it properly, what drives it, or how to deliberately cultivate it during training. The field possessed extensive empirical evidence that sparsity exists, but lacked systematic understanding of its governing principles.

The Measurement Problem: Existing Metrics Are Either Narrow or Poorly Calibrated

The paper identifies that a central obstacle to studying activation sparsity is the lack of a satisfactory measurement tool (Section 2.2). A "metric" for activation sparsity must determine, for any given input and any given layer, which neurons contribute so weakly that they can be safely ignored. An ideal metric should satisfy two properties: generalizability (it should work across different activation functions, not just ReLU) and performance preservation (skipping the neurons it identifies as weakly-contributed should cause negligible degradation to model outputs).

Prior metrics fail on one or both criteria:

The straightforward ReLU metric (zero threshold). This is the simplest and most commonly used approach: a neuron is considered weakly-contributed if its activation value is exactly zero. Formally, D = {i | s_i = 0}, where D is the set of weakly-contributed neuron indices and s_i is the activation value. This works naturally with ReLU, which produces exact zeros for negative pre-activations. However, it has two critical limitations. First, it is not generalizable: activation functions like SiLU—the dominant choice in mainstream LLMs (LLaMA, GPT variants)—produce non-negligible negative outputs that are mathematically small but not zero. Treating negative SiLU activations as "zero" and skipping those neurons discards genuine contributions to the output, degrading performance. Second, it misses non-zero but genuinely negligible activations: small positive ReLU outputs may contribute trivially to the final result but are treated as "active" by a zero threshold, undercounting sparsity.

Threshold-based extensions (FAT-ϵ). Kurtz et al. (2020) and Mirzadeh et al. (2023) attempted to generalize the zero-threshold approach by introducing a global positive threshold ε: D = {i | |s_i| < ε}. Using absolute values accommodates SiLU's negative activations. However, the paper points out a fundamental issue: a single global threshold across all layers is inherently suboptimal. Different layers operate at different activation scales, have different numbers of neurons, and contribute differently to the final output. A threshold loose enough to not harm sensitive layers will undercount sparsity in tolerant layers; a threshold tight enough to maximize sparsity will damage performance in critical layers. This makes the global threshold impossible to tune properly.

Top-k (MoE-style) sparsity. The mixture-of-experts paradigm enforces a fixed number of activated neurons per layer via top-k routing (Fedus et al., 2022). This guarantees a constant sparsity ratio across all layers by design. However, the paper demonstrates (Appendix B, Figure 12) that this approach suffers from a substantially worse performance-sparsity trade-off compared to naturally occurring (intrinsic) activation sparsity. MoE models sacrifice flexibility: neurons that should be activated are forcibly deactivated, and neurons that should be dormant are forcibly activated, purely to satisfy the fixed budget. Moreover, the optimal per-layer sparsity ratio is not known a priori—it varies across layers, inputs, and training stages—making a uniform top-k constraint inherently misaligned with the model's natural computation patterns.

CATS (Lee et al., 2024). This method finds layer-wise adaptive thresholds but targets the same expected sparsity ratio at each layer, making it effectively equivalent to Top-k in its uniformity constraint. It addresses the layer-wise scaling issue but retains the inflexibility of enforcing identical sparsity across layers with different functional roles.

The paper's critique of existing metrics sets up CETT as the natural solution: by equalizing the Cumulative Error of Tail Truncation (the relative L2 norm error introduced by skipping neurons) across layers, CETT adaptively finds per-layer thresholds that reflect each layer's actual contribution to the output. A layer where many neurons genuinely contribute little will naturally have a high threshold (and high sparsity); a layer where every neuron matters will have a low threshold. The introduction of a PPL increase tolerance (CETT-PPL-p%) then provides a principled way to calibrate the single global hyperparameter—just search for the CETT value that raises validation perplexity by exactly p% above the dense baseline. The paper's experiments (Figures 2, 3; Table 1) show that CETT-PPL-1% achieves the best performance-sparsity Pareto frontier among all tested methods, with average downstream task scores effectively unchanged from the dense setting.

Conflicting and Incomplete Evidence on Influential Factors

Beyond measurement, the paper is motivated by the absence of systematic knowledge about what determines how sparse a model becomes. Prior work had produced scattered, sometimes contradictory observations:

Li et al. (2022) found that activation sparsity increases with larger scale, depth, and width in T5 series models (Raffel et al., 2020), suggesting that bigger models are inherently sparser. However, this observation was specific to the T5 architecture with ReLU activation, trained on a particular data mixture and scale range. Whether the trend generalizes to decoder-only Transformers, to different activation functions, or to different training data regimes was unknown.

Zhang et al. (2024a) studied activation sparsity from the perspective of activation function choice, comparing ReLU variants and finding that certain activation functions produce more inherent sparsity than others. However, their work did not isolate how sparsity evolves with training data quantity, model shape (width-depth ratio), or parameter scale in a controlled manner—all factors that practitioners must decide when designing models.

Song et al. (2025) discovered that LLMs tend to be sparser on more formatted datasets (code, multiple-choice questions) than on free-form text, suggesting that data distribution affects measured sparsity. This raises a crucial methodological concern: if sparsity varies with dataset, what dataset should be used to measure it? And do the factors that influence sparsity (architecture, training data amount) interact with this dataset dependence?

These prior works collectively established that activation sparsity is an interesting and potentially useful property, but left fundamental questions unanswered: Is there a predictable scaling relationship between sparsity and training data? Do different activation functions produce not just different sparsity levels but different sparsity dynamics during training? Does making a model deeper versus wider affect sparsity? Do larger models converge to the same sparsity limit as smaller ones? Without answers to these questions, practitioners designing efficient LLMs must rely on trial and error rather than principled guidance.

The MoE Baseline and Why Intrinsic Sparsity Matters

Mixture-of-Experts represents the dominant architectural approach to activation sparsity—it explicitly constrains the model to activate only a subset of parameters per token. This raises a natural question: why study intrinsic activation sparsity at all, rather than just using MoE?

The paper argues (Section 1 and Appendix B) that MoE's architectural constraints come with inherent costs to flexibility and performance. MoE models require a fixed sparsity budget (e.g., top-2 out of 8 experts = 75% sparsity) to be chosen before training, with no guarantee this is optimal. They require load-balancing losses to prevent expert collapse. They introduce communication overhead in distributed training. And, critically, the paper's empirical comparison in Figure 12 (Appendix B) shows that at the same parameter scale and training data amount, a vanilla ReLU Transformer achieves a strictly better PPL-sparsity trade-off than MoE variants with various numbers of experts—the vanilla model reaches lower perplexity at any given sparsity ratio, or equivalently, higher sparsity at any given perplexity. This suggests that intrinsic activation sparsity, allowed to emerge naturally from the training dynamics, can allocate computation more efficiently than hard-coded expert routing.

The paper therefore positions itself not as competing with MoE, but as studying the more fundamental phenomenon of activation sparsity in standard architectures, which (1) avoids MoE's flexibility-performance trade-off, (2) is simpler to implement and train, and (3) can theoretically be combined with MoE for even greater efficiency.

The Distinction from Weight Pruning

Another important context is the relationship between activation sparsity and weight pruning, which the paper clarifies in Appendix C. Weight pruning permanently removes parameters from the model—the same weights are absent regardless of input. This is a static form of sparsity. Activation sparsity is dynamic: which neurons are important depends on the specific input token. A neuron handling mathematical notation might be crucial for "∫" but dormant for "the."

The practical consequence is that activation sparsity can achieve much higher effective sparsity ratios with substantially less performance degradation. Weight pruning at high sparsity levels (e.g., 50%+ of parameters removed) typically incurs significant accuracy loss (Frantar & Alistarh, 2023; Xia et al., 2023) because some inputs genuinely need those parameters. Activation sparsity, by contrast, never permanently removes any capacity—it merely skips computation that a particular input doesn't need. The paper's results bear this out: ReLU models achieve sparsity ratios exceeding 90% (i.e., fewer than 10% of neurons active per token) while maintaining performance comparable to dense SiLU models.

How This Paper Positions Itself

The paper frames itself as the first comprehensive quantitative study of activation sparsity that addresses measurement, influential factors, and practical construction of sparse LLMs in a unified framework (Section 1). Rather than proposing a fundamentally new sparsification technique, it provides the empirical foundation—scaling laws, architectural guidelines, measurement methodology—that the field has been missing.

The three research questions (Q1: measurement, Q2: influential factors, Q3: construction) are deliberately structured as a progression. Q1 (Section 4) establishes the tool (CETT-PPL-1%) that makes the rest of the study possible. Without a reliable, generalizable metric that preserves performance, any claims about what factors influence sparsity would be confounded by measurement error. Q2 (Section 5) then uses this tool to map the landscape: how does each factor (data amount, activation function, width-depth ratio, parameter scale) shift the sparsity curve? Q3 (Section 6) synthesizes these findings into actionable prescriptions and validates them on a larger-scale model, closing the loop from analysis to application.

The paper explicitly draws on the scaling laws tradition in deep learning but applies it to an efficiency property rather than loss or accuracy. Just as Hoffmann et al. (2022) established power-law relationships between compute, model size, data, and loss, this paper establishes power-law (and logspace power-law) relationships between training data and activation sparsity. The concept of a convergent limit—the sparsity ratio approached as training data goes to infinity—provides a theoretically grounded way to compare architectures and training recipes without conflating transient training effects with fundamental properties.

The paper also positions its findings about scale insensitivity (Section 5.3) as a direct challenge to the intuition—promoted by Li et al. (2022)—that larger models are inherently sparser. By controlling for width-depth ratio and examining the limit sparsity rather than sparsity at a fixed training budget, the paper finds that the asymptotic activation ratio varies by only 1–3 percentage points across models from 0.1B to 1.2B. The explanation—similar neuron specialization patterns across scales, with smaller models converging faster to their limit due to having fewer neurons to organize—provides a mechanistic hypothesis that connects sparsity dynamics to the combinatorics of neuron specialization.

Finally, the paper acknowledges important scope limitations (Appendix A) that contextualize its contributions: (1) it does not account for the increased wall-clock time of deeper models (smaller width-depth ratios), which can offset sparsity-driven speedups; (2) the CETT-PPL-p% metric is dataset-sensitive, and sparsity laws may have dataset-specific characteristics; and (3) the study focuses on intrinsic sparsity in vanilla Transformers, deliberately excluding MoE and hybrid architectures. These caveats make clear that the paper is providing a foundational empirical framework rather than a universally optimal recipe.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an empirical measurement and analysis study—it develops a systematic methodology for quantifying activation sparsity in large language models, maps out how this sparsity is influenced by architectural choices and the training process, and then validates that the resulting understanding can guide the construction of genuinely sparser and more efficient models. At its core, the paper is building a scientific instrument (the CETT-PPL-p% metric) and using it to discover empirical scaling laws that relate model design decisions to the resulting sparsity, rather than proposing a novel sparsification algorithm or architectural innovation. The solution takes the form of: (1) a calibrated measurement protocol that adaptively identifies weakly-contributed neurons per layer while preserving model performance, (2) a set of fitted mathematical relationships (power laws and logspace power laws) between sparsity and four influential factors, (3) an explanatory framework for the observed scale-insensitivity of sparsity based on neuron specialization combinatorics, and (4) prescriptive guidelines for training sparse LLMs from scratch, validated on a 2.4B-parameter model.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's technical approach comprises five interconnected components, each serving a distinct role in the measurement and analysis pipeline:

µP Transformer architecture (the subject under study): The base model family being trained and analyzed, consisting of decoder-only Transformers with gated feed-forward networks (FFNs). This is the "object" whose sparsity we measure, not a component we build to achieve something else. The architecture follows the µP parametrization for training stability across scales and uses the LLaMA-style gated FFN with either ReLU or SiLU activation.
CETT (Cumulative Error of Tail Truncation) algorithm: A method that, given a model checkpoint, a batch of inputs, and a target error tolerance, determines per-layer activation thresholds that identify which neurons are "weakly-contributed" and can be skipped. It operates by equalizing the L2 norm relative error across layers rather than using a single global threshold.
CETT-PPL-p% metric (the calibrated measurement instrument): A wrapper around CETT that binary-searches the shared error tolerance hyperparameter until the model's perplexity on a validation dataset increases by exactly p% compared to the dense (all-neurons-active) baseline. This yields a principled, performance-calibrated sparsity measurement. The specific variant p=1% is established as the standard metric throughout the paper.
Pre-training and decay pipeline: The experimental infrastructure for training models across five scales (0.1B, 0.2B, 0.4B, 0.8B, 1.2B parameters) and two activation functions (ReLU, SiLU) on a diverse text mixture (~300B tokens), followed by a decay stage with instruction-tuning data to obtain meaningful downstream evaluation. This pipeline generates the raw checkpoints that the measurement system analyzes.
Curve-fitting and analysis framework: The mathematical machinery for fitting power-law relationships between sparsity (or activation ratio) and training data amount, extracting limit values, and comparing dynamics across experimental conditions. This includes the Levenberg-Marquardt optimization algorithm for non-linear curve fitting and the derivative analysis for convergence speed.

Information flows as follows: training data → µP Transformer training → intermediate checkpoints → CETT-PPL-p% measurement on a tiny held-out validation set → sparse model evaluation on downstream benchmarks → aggregation of sparsity ratios across checkpoints → power-law curve fitting → limit extraction and cross-condition comparison → prescriptive guidelines for building sparse models.

3.3 Roadmap for the Deep Dive

First, I will explain the µP Transformer architecture and the neuron decomposition of the gated FFN, since all sparsity measurements operate on this structure and the definition of a "neuron" determines what we count as sparse.
Second, I will walk through the CETT algorithm—how it computes per-layer L2 norm relative errors, why it adaptively sets thresholds, and how the binary search over the error tolerance works—since CETT is the foundation on which the CETT-PPL-p% metric is built.
Third, I will detail the CETT-PPL-p% metric construction: how the PPL increase tolerance is defined, how the binary search algorithm links CETT to perplexity, what "p%" means operationally, and why p=1% is empirically selected as the performance-preserving sweet spot.
Fourth, I will describe the pre-training and evaluation setup—model configurations across the five scales, training data composition, hyperparameter choices, the sparsity stabilizing strategy for handling early-training noise, and the benchmark evaluation protocol—since this context determines the valid range of the empirical laws discovered.
Fifth, I will cover the curve-fitting methodology for extracting scaling laws between activation ratio and training data amount, including the specific functional forms for ReLU (decreasing logspace power-law) and SiLU (increasing vanilla power-law), the Levenberg-Marquardt fitting procedure, and the extraction of limit activation ratios.
Sixth, I will explain the width-depth ratio experiments—how the model shape is varied while holding parameter count fixed, how the bottleneck point is identified, and the tension between sparsity (favors smaller ratios) and training stability (requires not-too-small ratios).
Seventh, I will detail the scale-insensitivity analysis, including the neuron activation frequency distribution experiments, the token-wise activation ratio comparison, and the combinatorial model that explains why smaller models converge faster to their sparsity limit even though the limit itself is nearly constant.
Finally, I will cover the validation experiment: training a 2.4B model following the derived guidelines and measuring its sparsity dynamics and inference speedup.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical measurement and scaling-law study whose core idea is that activation sparsity can be rigorously measured, its governing factors can be quantitatively characterized through convergent power-law relationships, and these relationships can guide the deliberate construction of sparser models.

The µP Transformer Architecture and Neuron Definition

All experiments use the µP Transformer architecture (Hu et al., 2024), which combines the mainstream LLaMA-style decoder-only design (Touvron et al., 2023) with µP (Maximal Update) parametrization (Yang et al., 2022). The µP parametrization is a specific scheme for initializing weights and scaling learning rates across layers that ensures training dynamics remain stable as model width and depth change—a crucial property for this study since the experiments span models from 0.1B to 1.2B parameters (a 12× range) with varying width-depth ratios. Without µP, different scale models would require different learning rate tuning just to train stably, making it impossible to isolate the effect of architecture on sparsity.

The central object of study is the gated feed-forward network (FFN) within each Transformer layer, which is where activation sparsity manifests. The computation is:

$s = \sigma(W_{\text{gate}}x)$ $\text{FFN}(x) = W_{\text{out}}[s \odot (W_{\text{in}}x)]$

where $x \in \mathbb{R}^{d_h}$ is the input hidden state (one token representation), $d_h$ is the hidden dimension, $W_{\text{gate}}, W_{\text{in}} \in \mathbb{R}^{d_f \times d_h}$ and $W_{\text{out}} \in \mathbb{R}^{d_h \times d_f}$ are learnable weight matrices, $d_f$ is the intermediate (feed-forward) dimension, $\sigma$ is the activation function (ReLU or SiLU in this paper), $s \in \mathbb{R}^{d_f}$ are the activation values, and $\odot$ denotes element-wise multiplication. The gated FFN differs from the classic two-layer FFN (which would be simply $W_{\text{out}}\sigma(W_{\text{in}}x)$ ) by introducing the gate projection $W_{\text{gate}}$ that multiplicatively modulates the up-projected representations—this is the standard FFN design in LLaMA and most modern LLMs.

What this computation does: for each token, the FFN transforms the hidden state through three linear projections—an "up" projection (producing $W_{\text{in}}x$ ), a "gate" projection (producing $\sigma(W_{\text{gate}}x)$ ), and a "down" projection $(W_{\text{out}})$ . The gate values element-wise multiply the up-projected values, allowing the activation function to selectively suppress or pass through information in each intermediate dimension. The result is a vector in $\mathbb{R}^{d_h}$ that is added to the residual stream.

Why this decomposition matters for sparsity: the paper defines a "neuron" not as a biological analog but as a specific slice of the parameter matrices. By decomposing the FFN computation along the intermediate dimension $d_f$ , the output becomes a sum over independent contributions:

$s_i = \sigma(W_{\text{gate}[i,:]}x)$ $n_i = W_{\text{out}[:,i]} [s_i \odot (W_{\text{in}[i,:]}x)]$ $\text{FFN}(x) = \sum_{i=1}^{d_f} n_i$

where $W_{\text{gate}[i,:]}$ , $W_{\text{in}[i,:]}$ , and $W_{\text{out}[:,i]}$ are the i-th row of $W_{\text{in}}$ , i-th row of $W_{\text{gate}}$ , and i-th column of $W_{\text{out}}$ , respectively. Each neuron $i$ corresponds to one scalar activation value $s_i$ and contributes a vector $n_i \in \mathbb{R}^{d_h}$ to the total FFN output. The crucial property is that the neuron contributions are additive: the FFN output is exactly the sum of all neuron outputs. This means that if some neuron contributions are negligible (small in L2 norm relative to the total), we can skip computing them entirely with bounded error.

Why this definition versus alternatives: defining a neuron as a single column of $W_{\text{out}}$ paired with corresponding rows of $W_{\text{in}}$ and $W_{\text{gate}}$ follows the standard decomposition in the activation sparsity literature (Li et al., 2022; Zhang et al., 2024a). An alternative would be to treat each element of the intermediate representation as a separate "unit," but since the output contribution is the vector $n_i$ , not the scalar $s_i$ , the neuron definition must encompass all parameters that jointly produce a single additive term in the output.

Across all model scales, the ratio $d_f / d_h$ is held constant at 2.5 (a standard expansion factor for Transformer FFNs), and the number of query heads always matches the number of key and value heads (multi-head attention is not the focus of this paper). The specific configurations for each scale are provided in Table 3. The width-depth ratio—defined as $d_h$ divided by the number of layers—ranges from 48 to 56 across the default configurations, kept intentionally similar to isolate the effect of scale from the effect of model shape.

The CETT Algorithm: Adaptive Layer-Wise Thresholding

CETT (Cumulative Error of Tail Truncation) is introduced by Zhang et al. (2024a) and adopted by this paper as the foundation for its sparsity metric. The key insight is that different layers contribute differently to the final output, so a single global activation threshold cannot optimally balance sparsity and performance across layers. CETT addresses this by equalizing the relative error introduced by skipping weakly-contributed neurons.

Given an input $x$ to a specific FFN layer, the set $D$ of weakly-contributed neuron indices is defined as:

$D = \{i \mid \|n_i\|_2 < \epsilon\}$

where $\|n_i\|_2$ is the L2 norm (Euclidean length) of the i-th neuron's output contribution vector $n_i$ , and $\epsilon$ is a threshold that is searched separately for each layer to satisfy a shared error constraint.

What this definition does: instead of thresholding on the scalar activation value $s_i$ (as the straightforward ReLU metric does), CETT thresholds on the actual vector contribution $\|n_i\|_2$ that the neuron would make to the FFN output. This automatically accounts for the fact that two neurons with the same activation value might contribute very differently—one might be multiplied by a column of $W_{\text{out}}$ with large entries, the other by a column with tiny entries. The L2 norm $\|n_i\|_2$ captures the magnitude of what would actually be added to the residual stream.

The error introduced by skipping all neurons in $D$ is quantified as:

$\text{CETT} = \frac{\|\sum_{i \in D} n_i\|_2}{\|\text{FFN}(x)\|_2}$

where $\|\text{FFN}(x)\|_2$ is the L2 norm of the total (dense) FFN output, and $\|\sum_{i \in D} n_i\|_2$ is the L2 norm of the sum of all skipped neuron contributions (which is the error vector—the difference between the dense output and the sparse output).

What this computes: the relative L2 error caused by truncation. It is the ratio of "how much signal we lost by skipping those neurons" to "how much signal the layer produced overall." If the dense output is $v$ and the sparse output is $v - e$ (where $e = \sum_{i \in D} n_i$ ), then CETT = $\|e\|_2 / \|v\|_2$ . A CETT of 0 means perfect reconstruction (no error); a CETT of 0.01 means the error magnitude is 1% of the output magnitude.

Why this form matters compared to alternatives:

Why L2 norm instead of L1? L2 norm penalizes large individual errors more heavily than many small errors, which better reflects the downstream impact on the residual stream (where large perturbations can propagate through subsequent layers). L1 would be more forgiving of a few large errors if balanced by many zero entries, which could be catastrophic for model performance.
Why relative error instead of absolute error? The magnitude of FFN outputs varies substantially across layers, across token positions, and across models. A fixed absolute error tolerance would be extremely loose for layers producing small outputs (effectively skipping computation that matters) and unnecessarily tight for layers producing large outputs (wasting sparsity). Relative error normalizes by the layer's typical output scale.
Why the norm of the sum rather than the sum of norms? The triangle inequality tells us $\|\sum n_i\|_2 \leq \sum \|n_i\|_2$ , meaning that summing norms overestimates the actual error—weak neuron contributions might cancel out directionally (though this is rare in practice with high-dimensional vectors). Using $\|\sum n_i\|_2$ directly measures the actual degradation to the layer output.

The key hyperparameter is the shared CETT value across all layers. For a given CETT value, each layer independently binary-searches its threshold $\epsilon$ such that skipping all neurons with $\|n_i\|_2 < \epsilon$ yields exactly that relative error. Since $\|n_i\|_2$ depends on the input $x$ , this computation occurs per-input—in practice, the paper evaluates CETT on batches of validation data and averages.

Why each layer gets a different $\epsilon$ : two layers with the same CETT constraint might end up with very different $\epsilon$ thresholds. A layer where many neurons contribute roughly equally (a "flat" contribution distribution) will need a small $\epsilon$ because skipping even moderately-contributing neurons would quickly exceed the error budget. A layer where a few neurons dominate the output and the rest contribute almost nothing (a "spiky" distribution) can tolerate a large $\epsilon$ , skipping many weak neurons while the strong ones preserve the output within the error budget. This adaptivity is the central advantage of CETT over global-threshold methods.

The paper formally verifies (Appendix F, Figure 15) that both perplexity and the overall activation ratio change monotonically with CETT—as CETT increases (tolerating more error), the measured sparsity ratio increases and perplexity degrades. This monotonicity is what makes binary search feasible: there is a unique CETT value for any target PPL increase.

The CETT-PPL-p% Metric: Calibrating CETT Through Perplexity

While CETT provides a way to set per-layer thresholds given a shared error tolerance, it leaves open the question of how to choose that shared tolerance. Too small a CETT value yields measurements that undercount sparsity (because the model could actually tolerate more error without meaningful degradation); too large a CETT value yields measurements that overcount sparsity (because the model's outputs are corrupted). The paper resolves this by linking the CETT hyperparameter to a perplexity increase tolerance on a validation dataset.

Definition of CETT-PPL-p%: Let $PPL_{\text{dense}}$ be the model's perplexity on a validation set with all neurons active (the dense baseline), and let $PPL_{\text{sparse}}(\text{cett})$ be the perplexity when, at each layer, neurons identified as weakly-contributed by CETT with hyperparameter cett are skipped during computation. Then CETT-PPL-p% is the sparsity ratio measured by CETT at the specific hyperparameter value $\text{cett}^*$ such that:

$\frac{PPL_{\text{sparse}}(\text{cett}^*)}{PPL_{\text{dense}}} = 1 + p\%$

Or equivalently, $PPL_{\text{sparse}}(\text{cett}^*) = (1 + p\%) \cdot PPL_{\text{dense}}$ .

What this computes operationally: a sparsity ratio that is as high as possible while keeping the model's language modeling quality within a controlled degradation budget of p%. The sparsity ratio itself is $|D| / d_f$ averaged (or summed) across all FFN layers, where $D$ contains the neurons skipped at threshold $\text{cett}^*$ .

The binary search algorithm (Algorithm 1 in Appendix F) finds $\text{cett}^*$ as follows:

Initialize search bounds: $l = 0$ (no error tolerance—sparse output equals dense output, PPL ratio = 1.0), $r = 1$ (100% relative error tolerance—presumably high PPL degradation).
While $r - l > \text{eps}$ (a small convergence tolerance), compute midpoint $\text{mid} = (l + r) / 2$ .
For each checkpoint in the input list, compute the dense loss $L_{\text{dense}}$ and the sparse loss $L_{\text{sparse}}$ using CETT with $\text{cett} = \text{mid}$ . The PPL ratio is $\exp(L_{\text{sparse}} - L_{\text{dense}})$ since PPL = $\exp(\text{loss})$ .
Average the PPL ratios across checkpoints. If the mean PPL ratio $< 1 + p\%$ , set $l = \text{mid}$ (we can be more aggressive with sparsity). If the mean PPL ratio $\geq 1 + p\%$ , set $r = \text{mid}$ (we've exceeded the tolerance, back off).
Return $(l + r) / 2$ as $\text{cett}^*$ .

Why binary search works: as demonstrated in Figure 15 (Appendix F), both PPL and the overall activation ratio are monotonically increasing functions of CETT. A larger error tolerance always skips more (or at least equally many) neurons and always degrades perplexity more (or at least equally). This monotonicity guarantees that the binary search converges to the unique crossover point where the PPL degradation exactly hits the tolerance.

The sparsity stabilizing strategy (Appendix E): A practical challenge is that the sparsity metric is noisy, especially during early training when the model is far from convergence and gradient descent introduces significant fluctuations. To obtain a clean measurement, the paper applies CETT-PPL-p% not to individual checkpoints but to the last five pre-trained checkpoints as a group. Specifically:

Drop all checkpoints from the warmup stage (where sparsity fluctuates wildly and the model is not representative of its converged behavior).
For each of the last five pre-training checkpoints, compute the dense and sparse losses using the same $\text{cett}$ value searched over their collective average PPL.
Apply the found $\text{cett}^*$ to all checkpoints (including early ones) to trace the sparsity-data curve, but report only the final measurements for limit extraction.

Why use five checkpoints rather than just the final one? Averaging reduces the impact of stochastic checkpoint-to-checkpoint variation. Two checkpoints trained with slightly different mini-batch orderings might have different sparsity at the exact same training step, but averaging over five checkpoints produces a more stable estimate of the model's asymptotic sparsity.

Why p=1%? The paper empirically justifies this choice through Table 1, which compares downstream task performance under different tolerance levels (CETT-PPL-1%, CETT-PPL-5%, CETT-PPL-10%) against the dense baseline (p=0%). The key finding is that CETT-PPL-1% essentially preserves performance:

On commonsense reasoning, the average score difference (Δ CETT-PPL-1%) across all five scales and both activation functions is just −0.16 percentage points—negligible and often within noise. By contrast, CETT-PPL-5% drops by −0.50 points on average.
On reading comprehension, which is more sensitive to sparsity (likely because it relies on precise token-level retrieval), CETT-PPL-1% shows an average drop of −0.30 points, while CETT-PPL-5% drops by −2.21 points and CETT-PPL-10% drops by −3.64 points.

The asymmetry between task types is interesting: commonsense reasoning is almost entirely unaffected even at 10% PPL increase, while reading comprehension degrades substantially. The paper hypothesizes (implicitly) that reading comprehension tasks require finer-grained token representations that are more disrupted by neuron skipping, but does not investigate this in depth.

Why this metric is "generalizable": by operating on the L2 norm of neuron output vectors rather than on raw activation values, and by adaptively setting per-layer thresholds based on relative error rather than using a fixed global threshold, CETT-PPL-p% works for any activation function. The paper demonstrates this explicitly in Figures 2 and 3, where CETT-PPL-p% Pareto curves dominate the straightforward ReLU metric (which only works for ReLU) and substantially outperform Top-k and FAT-ε for SiLU models.

Relationship between CETT-PPL-p% and the reported "sparsity ratio": the final number is an aggregate—the fraction of all $d_f$ neurons across all FFN layers that are identified as weakly-contributed under the $\text{cett}^*$ threshold, averaged over the validation dataset (or over the evaluation dataset, for downstream task measurements). The paper reports both activation ratio ( $1 - \text{sparsity ratio}$ ) and sparsity ratio depending on context, with activation ratio being preferred for curve fitting because it produces more convenient functional forms.

Pre-Training and Evaluation Pipeline

The experimental pipeline generates the raw data—model checkpoints at various training stages, under various architectural configurations—that the CETT-PPL-p% metric then measures.

Training data composition: The pre-training corpus is a mixture of diverse text sources: a cleaned version of CommonCrawl, Dolma (Soldaini et al., 2024), C4 (Raffel et al., 2020), Pile (Gao et al., 2020), the Stack (Kocetkov et al., 2022), StarCoder (Li et al., 2023), and additional collected raw corpus. The total training budget is approximately 300B tokens, though exact amounts vary slightly by model scale. This diversity is important because prior work (Song et al., 2025) found that sparsity varies with data distribution—a model trained only on code might show different sparsity characteristics than one trained on a broad mixture. By using a representative pre-training mixture, the paper's measurements reflect what practitioners would encounter in standard LLM training.

Decay stage: Before evaluating on downstream benchmarks, all models undergo a "decay stage" where instruction-tuning data is mixed into the training distribution. This follows recent LLM training practice (Dubey et al., 2024; Hu et al., 2024) and serves to make the models produce more reasonable outputs on task-specific evaluations. The decay data includes UltraChat (Ding et al., 2023), SlimOrca (Colombo et al., 2024), OssInstruct (Wei et al., 2024), EvolInstruct (Xu et al., 2023), and other collected instruction datasets. The paper notes that the decay stage is applied before task evaluation but the sparsity measurements (training loss, validation loss, perplexity) are all computed on checkpoints that have only completed pre-training—the decay stage does not affect the measured sparsity of the foundation model.

Validation data for sparsity measurement: A tiny validation dataset is constructed by sampling from the same distribution as the pre-training data and then deduplicating against the training set to eliminate any data leakage. This deduplication is critical: if the validation set contained text seen during training, the model might have memorized those exact sequences, leading to artificially low perplexity and an inflated estimate of how much sparsity the model can tolerate. By using genuinely held-out data, the PPL degradation reflects the model's generalization behavior.

Hyperparameters across scales (Table 3): The paper uses the WSD (Warmup-Stable-Decay) learning rate scheduler from Hu et al. (2024) with consistent hyperparameters: peak learning rate $lr = 0.01$ , $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , and weight decay = 0.1 across all settings. The batch size is scaled with model size following the µP recommendations:

Scale	# non-embedding params	Batch size (tokens)
0.1B	1.08 × 10⁸	3.27 × 10⁵
0.2B	2.41 × 10⁸	5.90 × 10⁵
0.4B	4.52 × 10⁸	7.86 × 10⁵
0.8B	7.60 × 10⁸	1.18 × 10⁶
1.2B	1.18 × 10⁹	1.57 × 10⁶
2.4B	2.44 × 10⁹	2.10 × 10⁶

Why batch size scales with model size: larger models have more parameters and thus noisier gradients per token; larger batches reduce gradient variance, maintaining a stable signal-to-noise ratio. The scaling roughly follows the square-root rule common in the µP literature [see paper's citation of Yang et al., 2022].

Width-depth ratios across default configurations: Across the five experimental scales (0.1B to 1.2B), the width-depth ratio (hidden dimension ÷ number of layers) ranges from 48 to 56—generally similar across scales. This is a deliberate choice to ensure that when comparing sparsity across scales, differences are attributable to parameter count rather than model shape, which Section 5.2 shows is an independent influential factor.

Evaluation benchmarks: The paper evaluates on two task groups to assess whether CETT-PPL-p% is performance-friendly across different skill types:

Commonsense reasoning (C.R.): average 0-shot accuracy on PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2020), COPA (Roemmele et al., 2011). These tasks test everyday physical and social reasoning.
Reading comprehension (R.C.): average 0-shot accuracy on BoolQ (Clark et al., 2019), LAMBADA (Paperno et al., 2016), TyDi QA (Clark et al., 2020). These tasks require precise extraction and understanding of text passages.

The paper also evaluates on more complex tasks (HumanEval, MBPP, GSM8K, MMLU, BBH, AGI-Eval) but reports in Table 6 that models in the 0.1B–1.2B range largely fail to exceed random performance on these, so they are not used for the main analysis.

Storing and processing checkpoints for sparsity measurement: The paper records sparsity ratios at multiple checkpoints throughout training to trace the evolution curve. After applying the sparsity stabilizing strategy (Section E), the final sparsity measurements are the values computed at each checkpoint using the $\text{cett}^*$ found from the last five checkpoints' aggregate PPL.

Activation-Data Scaling Laws: Functional Forms and Fitting

The central empirical contribution is the characterization of how the activation ratio (or sparsity ratio) evolves with the amount of pre-training data consumed. The paper fits parametric curves to the measured data points and extracts asymptotic limits.

Why fit activation ratio rather than sparsity ratio: the paper states (Section 5.1) that "the curve of activation ratios to the amount of pre-training data is easier to fit than that of sparsity ratios." This is an empirical observation—the functional forms described below (decreasing logspace power-law for ReLU, increasing vanilla power-law for SiLU) fit the activation ratio data more naturally. Since activation ratio = 1 − sparsity ratio, the two are informationally equivalent.

For ReLU-activated models, the activation ratio $A_{\text{ReLU}}(D)$ as a function of the number of pre-training tokens $D$ (measured in billions, normalized by $10^9$ ) follows a decreasing logspace power-law:

$A_{\text{ReLU}}(D) = \exp(-cD^\alpha + b) + A_0$

where $A_0 > 0$ is the limit activation ratio (the value approached as $D \to \infty$ ), $c > 0$ controls the initial rate of sparsification, $\alpha > 0$ controls how the rate changes with data (sublinear if $\alpha < 1$ , superlinear if $\alpha > 1$ ), and $b$ is an offset parameter.

What this computes: the predicted fraction of neurons that remain active (i.e., are not weakly-contributed) after training on $D$ tokens. The term $\exp(-cD^\alpha + b)$ represents the "excess activation" beyond the asymptotic limit $A_0$ —it starts at some finite value when $D = 0$ (which would be $\exp(b) + A_0$ , representing the initial activation ratio at random initialization) and decays toward zero exponentially fast in $D^\alpha$ . As $D \to \infty$ , $\exp(-cD^\alpha + b) \to 0$ , so $A_{\text{ReLU}}(D) \to A_0$ .

Why this form has the properties needed:

Convergence to a positive limit $A_0$ : the additive constant ensures the activation ratio doesn't go to zero—some neurons will always be active regardless of how much training data is provided. This matches the intuition that certain neural computations are genuinely necessary for language modeling and cannot be eliminated.
Exponential decay of excess activation: the $\exp(-cD^\alpha + b)$ term captures the observation that sparsity improves rapidly early in training but the rate of improvement slows as the model approaches its asymptotic limit. The exponent $\alpha$ controls whether this slowdown is more gradual ( $\alpha < 1$ , meaning $D^\alpha$ grows slowly) or sharper ( $\alpha > 1$ ).
Decreasing function: since $c > 0$ and $\alpha > 0$ , $\exp(-cD^\alpha + b)$ is strictly decreasing in $D$ , meaning more data always reduces the activation ratio (increases sparsity). This is the key advantage of ReLU: you can always push sparsity higher by training longer.

For SiLU-activated models, the activation ratio $A_{\text{SiLU}}(D)$ follows an increasing vanilla power-law:

$A_{\text{SiLU}}(D) = -\frac{c}{D^\alpha} + A_0$

where $A_0 > 0$ is again the limit activation ratio, and $c > 0$ , $\alpha > 0$ control the approach dynamics.

What this computes: the predicted activation ratio for a SiLU model after $D$ tokens. The term $-\frac{c}{D^\alpha}$ is negative (since $c > 0$ and $D^\alpha > 0$ ), so $A_{\text{SiLU}}(D) < A_0$ for all finite $D$ , and $A_{\text{SiLU}}(D)$ increases toward $A_0$ from below as $D$ grows.

Why this form has different implications from ReLU:

Increasing function: since $-\frac{c}{D^\alpha}$ becomes less negative as $D$ increases, the activation ratio rises with more data. This means more training data makes SiLU models less sparse, not more. The negative term represents a deficit relative to the asymptotic activation ratio—early in training, the model activates fewer neurons than its eventual stable configuration, and as training continues, it gradually activates more.
Convergence from below: the model starts with fewer active neurons and "fills in" to reach its asymptotic $A_0$ . The paper doesn't deeply theorize about why SiLU shows this pattern while ReLU shows the opposite, but the implication is clear: ReLU's zero-activation regime (where negative pre-activations are clamped to exactly zero) allows the model to permanently "switch off" neurons that prove unnecessary, while SiLU's smooth negative tail means that even "mostly inactive" neurons retain small non-zero contributions that never fully vanish.
Bounded asymptotic sparsity: unlike ReLU where $A_0$ is approached from above (excess activation burns off), SiLU approaches $A_0$ from below (insufficient activation fills in). Either way, there's a limit—you cannot make a SiLU model arbitrarily sparse just by training longer.

Fitting procedure: the paper uses the Levenberg-Marquardt algorithm (Marquardt, 1963), a standard non-linear least-squares optimization method that interpolates between gradient descent (far from optimum) and Gauss-Newton (near optimum). All training token counts $D$ are divided by $10^9$ to normalize magnitudes before fitting, improving numerical stability.

Fitted coefficients (Table 2): The results show systematic patterns in the fitted parameters:

For ReLU models, $\alpha$ increases steadily with scale: 0.101 (0.1B) → 0.449 (0.2B) → 0.683 (0.4B) → 1.01 (0.8B) → 1.33 (1.2B) → 1.53 (2.4B). This means larger models have super-linear decay of excess activation, approaching their limit more sharply. The limit activation ratio $A_0$ increases slightly with scale: 6.14% (0.1B) → 7.82% (1.2B), representing a mild trend toward higher activation ratio (lower sparsity) in larger models. The coefficient $c$ decreases dramatically: 3.20 (0.1B) → 0.000903 (1.2B), reflecting that larger models start with lower "excess activation" above their limit—they are born closer to their asymptotic sparsity.

For SiLU models, $\alpha$ consistently hovers around 0.48–1.03 across scales, and the limit activation ratio $A_0$ decreases slightly from 40.9% (0.1B) to 38.2% (1.2B)—a small 2.7 percentage point drop, consistent with the scale-insensitivity finding.

Extracting limit values: the limit activation ratio $A_0$ for each configuration is read directly from the fitted curve as the asymptotic value. This is used for cross-condition comparisons (e.g., Section 5.2's width-depth ratio analysis and Section 5.3's scale analysis).

Width-Depth Ratio Experiments

The width-depth ratio—defined as the ratio of the hidden dimension to the number of layers—captures whether a model is "wide and shallow" (high ratio) or "narrow and deep" (low ratio) for a given parameter count. To isolate its effect on sparsity, the paper conducts a dedicated experiment on 0.1B ReLU models with nine different width-depth ratios, holding total non-embedding parameter count approximately fixed.

How parameter count is kept fixed: as the number of layers $L$ increases, the hidden dimension $d_h$ must decrease to maintain roughly constant total parameters. The main parameter contribution comes from the FFN weights: $W_{\text{gate}}$ , $W_{\text{in}}$ (both of size $d_f \times d_h = 2.5 d_h \times d_h = 2.5 d_h^2$ ), and $W_{\text{out}}$ (of size $d_h \times d_f = d_h \times 2.5 d_h = 2.5 d_h^2$ ), for a total of roughly $7.5 d_h^2$ parameters per layer. Across $L$ layers, this is $7.5 L d_h^2$ , plus attention parameters. Since $d_h \approx \sqrt{\text{total\_params} / L}$ , increasing $L$ requires decreasing $d_h$ proportionally.

Results on activation ratio (Figure 5): The limit activation ratio $A_0$ (as $D \to \infty$ ) is plotted against width-depth ratio, revealing two regimes:

Below a bottleneck point (approximately 114 for 0.1B models): the activation ratio increases linearly with width-depth ratio. Deeper models (lower ratio) are sparser—their limit activation ratio is lower. The linear relationship suggests that each additional unit of width-depth ratio adds a roughly constant increment to the asymptotic activation ratio.
Above the bottleneck: the activation ratio fluctuates around a fixed level (approximately 8% for 0.1B). Making the model even wider and shallower beyond this point does not systematically increase activation ratio (reduce sparsity)—it just adds noise. The bottleneck represents the point beyond which the model's shape no longer influences sparsity.

Results on training loss (Figure 6): However, the training loss paints a more complex picture. The limit training loss is roughly flat across a middle range of width-depth ratios (from about 74 to 282), but increases sharply at very small ratios (below ~74). This is the training instability regime: models that are too deep relative to their width suffer from optimization difficulties—gradients become unstable through many layers, attention becomes harder to train, and the model underperforms.

The trade-off: the "optimal" width-depth ratio from a sparsity perspective would be as small as possible (deepest model), but this is bounded below by training stability. The paper therefore recommends choosing the smallest width-depth ratio within the interval that ensures stable, high-performance training. For the 0.1B models, this interval is approximately 74 to 282, and the recommendation is to pick a value near the lower end (e.g., around 74–114).

Why deeper models are sparser: the paper does not provide a mechanistic explanation for this relationship, but a plausible interpretation (consistent with the neuron specialization framework in Section 5.3) is that deeper models have more layers to distribute their representational work. Each layer can specialize more narrowly (since there are many layers to cover all needed functions), meaning fewer neurons per layer need to be active for any given input. In a wide, shallow model, each layer must handle a broader range of functions, requiring more neurons to be simultaneously active.

Limitation acknowledged: the paper notes (Appendix A) that the computation cost of deeper models is not accounted for. While a deeper model might be sparser (activating fewer neurons per token), the fact that it has more layers means the total computation might not decrease. A deep model with 60 layers each at 90% sparsity might have similar total FLOPs to a shallow model with 12 layers each at 50% sparsity. The paper's sparsity analysis focuses on the fraction of neurons active per layer, not on total inference cost, and acknowledges this as a gap for future work.

Scale-Insensitivity Analysis: Neuron Specialization and Combinatorial Model

Section 5.3 addresses one of the paper's most striking findings: the asymptotic activation ratio $A_0$ is nearly constant across model scales (from 0.1B to 1.2B), varying by only 1.7 percentage points for ReLU and 2.7 points for SiLU. This directly challenges the common intuition that larger models are sparser (Li et al., 2022).

The raw data (Figure 7): For ReLU, $A_0$ goes from approximately 6.14% (0.1B) to 7.82% (1.2B)—a marginal increase. For SiLU, $A_0$ goes from approximately 40.9% (0.1B) to 38.2% (1.2B)—a slight decrease. Neither trend is large enough to claim a meaningful scale dependence, especially when compared to the massive differences between activation functions (ReLU at ~7% vs. SiLU at ~40%) or the effect of data amount.

The convergence speed difference (Figure 8): Despite having similar limits, smaller models converge much faster to their asymptotic sparsity. Figure 8 plots the derivative (rate of change) of the sparsity-data curve against the data-scale ratio (number of training tokens divided by parameter count). The absolute derivative values for 0.1B models are substantially larger than for 1.2B models at the same data-scale ratio, meaning smaller models are reducing their excess activation (for ReLU) or increasing toward their limit (for SiLU) more rapidly per unit of training.

The explanation framework: the paper proposes a two-part explanation:

Part 1: Similar activation patterns across scales (empirical evidence). The paper conducts two experiments demonstrating that which neurons activate for which inputs is similar across model sizes:

Dataset-wise activation frequency distributions (Figure 9): For four subsets of the pre-training data (Code, Wikipedia, Math, Chinese), the paper computes, for each neuron in each model, its activation frequency—the fraction of tokens in that dataset for which the neuron's activation value is non-zero (for ReLU) or above threshold (for SiLU, implicitly via CETT). The distribution of these frequencies across all neurons in the model is then plotted for each scale. The results show that the shape of the distribution is remarkably consistent across 0.1B to 1.2B models for both ReLU and SiLU. Some neurons activate very frequently (near 100%—these are highly general neurons), some activate rarely (near 0%—highly specialized or dead neurons), and the proportions in each frequency bucket are similar regardless of total neuron count.

This means that if 20% of neurons in a 0.1B model activate for code tokens, roughly 20% of neurons in a 1.2B model also activate for code tokens—the relative allocation of neuron "capacity" across data domains is scale-invariant.

Token-wise activation ratio comparison (Figure 10): The paper samples 71,549 tokens from the vocabulary and computes, for each token, its activation ratio (fraction of neurons activated) averaged over a large number of context occurrences. Pair-wise comparisons between models of different scales show that most tokens maintain a close activation ratio across scales—the scatter plot clusters tightly around the $y = x$ line, meaning a token that activates 10% of neurons in a 0.1B model also activates roughly 10% of neurons in a 1.2B model.

Part 2: The combinatorial neuron specialization model (deductive explanation). The paper constructs a mathematical model of neuron specialization to explain why similar activation patterns imply faster convergence for smaller models.

Assumption: neurons in FFNs specialize into functional groups during training (Li et al., 2022; Zhang et al., 2023). Suppose there are $G$ functional groups (e.g., syntax neurons, math neurons, fact-retrieval neurons), and group $i$ should contain $t_i$ neurons. Based on the similar activation patterns evidence, the proportion of neurons assigned to each group— $t_i / d_f$ —is similar across scales. That is, if syntax requires 15% of neurons in a 0.1B model, it also requires roughly 15% in a 1.2B model.

The combinatorial space: the number of possible ways to partition $d_f$ neurons into $G$ groups with sizes $t_1, t_2, \ldots, t_G$ is:

$T(d_f) = \prod_{i=1}^{G} C_{d_f}^{t_i} = \prod_{i=1}^{G} \frac{d_f!}{t_i!(d_f - t_i)!}$

where $C_{d_f}^{t_i}$ is the binomial coefficient: the number of ways to choose $t_i$ neurons (for group $i$ ) out of $d_f$ total neurons.

What this computes: the number of distinct neuron-to-group assignments that produce the correct group sizes. This is the "search space" that training must navigate to find a good specialization—each training step effectively samples from or moves through this space of possible assignments.

Why smaller models converge faster: $T(d_f)$ grows super-exponentially with $d_f$ because factorial functions dominate exponentials. For a concrete illustration: if $d_f$ doubles, $d_f!$ grows much more than quadratically, and the number of possible assignments explodes. A 1.2B model ( $d_f \approx 2.5 d_h$ , with $d_h$ around 1536 for the 1.2B configuration, giving $d_f \approx 3840$ ) has vastly more possible neuron specializations than a 0.1B model (with $d_f$ around 768–1024). Training must explore this combinatorially larger space to find the correct assignment, requiring more data even though the proportion of neurons per group is the same.

Importantly, this model explains both observations:

Similar limits: since the group proportions $t_i / d_f$ are scale-invariant, the asymptotic fraction of active neurons (which depends on which functional groups are needed for a given input) is also scale-invariant. A larger model has more neurons in each group, but the groups are the same.
Slower convergence for larger models: the combinatorial explosion in $T(d_f)$ means that, even though the final assignment has the same proportional structure, finding it requires navigating a space whose size grows factorially with neuron count.

Why layers at the extremes differ (Appendix J, Figure 16): The paper notes that while the average activation distribution is scale-invariant, specific layers—particularly the first and last layers—show different patterns across scales. The first layer, which processes raw token embeddings, and the last layer, which produces the final hidden states before the output projection, play unique roles that don't scale in the same way as intermediate layers. The middle layers, which form the bulk of the network, are where the scale-invariance holds most cleanly. This is a nuance: the claim of scale-insensitive activation patterns applies primarily to the "typical" intermediate layers, not to the boundary layers with special structural roles.

Validation Experiment: Training the 2.4B Sparse Model

To validate that the empirical laws discovered in the 0.1B–1.2B range generalize and provide actionable guidance, the paper trains a larger 2.4B model following the derived prescriptions and measures its sparsity dynamics.

Design choices based on findings:

ReLU activation (from Section 5.1): ReLU achieves significantly higher sparsity than SiLU (limit activation ratio ~7% vs. ~40%), with comparable downstream task performance (Table 1), and follows a decreasing logspace power-law—more data always helps. SiLU is rejected because its sparsity actually degrades with more data.
Small width-depth ratio (from Section 5.2): the model's width-depth ratio is chosen to be "close to the 0.1B-1.2B experimental models" (which range from 48 to 56) and "small enough and close to the smallest point of the training stability interval." This means the model is relatively deep for its width, maximizing sparsity while remaining within the stable training regime.
Large amount of pre-training data (from Section 5.1): since the logspace power-law is decreasing, feeding more data drives the activation ratio down toward its asymptotic limit. The 2.4B model is trained on approximately 800B tokens (near 800B), substantially more than the 300B used for the main experiments.

Results (Figure 11): The activation-data curve for the 2.4B model is well-fitted by the same decreasing logspace power-law functional form:

$A_{2.4B}(D) = \exp(-1.56 \times 10^{-4} \cdot D^{1.53} - 3.46) + 6.48\%$

The fitted parameters (Table 2) show: $\alpha = 1.53$ (super-linear decay, consistent with the trend toward higher $\alpha$ at larger scales), $b = -3.46$ , $c = 1.56 \times 10^{-4}$ (extremely small, indicating the model starts close to its limit), and $A_0 = 6.48\%$ sparsity limit activation ratio, corresponding to a limit sparsity ratio of 93.52%—fewer than 7% of neurons active on average.

This limit is "quite close to those values of models from 0.1B to 1.2B" (which range from ~6.1% to ~7.8%), consistent with the scale-insensitivity finding.

Inference speedup measurement (Appendix L): To demonstrate practical value, the 2.4B model's decoding speed is measured using two frameworks:

PowerInfer (Song et al., 2023): An inference engine specifically designed to exploit activation sparsity. It uses an offline profiler that records per-neuron activation frequencies from a calibration dataset, plus online activation predictors that forecast which neurons will be active for each input token. Based on these predictions, it allocates hardware resources adaptively—frequently-active neurons are placed in fast memory (GPU), rarely-active neurons in slower memory (CPU), and neurons predicted to be inactive are entirely skipped.
llama.cpp (Gerganov, 2023): A standard dense inference engine that computes all FFN neurons regardless of their actual contributions. Although llama.cpp does not officially support ReLU, this does not affect the measurement because the FLOP counts for dense FFN computation are identical regardless of activation function—it still multiplies all weight matrices.

Using 100 test prompts (5 prefix tokens each, sampled from C4), PowerInfer achieves an average decoding speed of 41.79 tokens/second, while llama.cpp achieves 10.23 tokens/second, yielding a 4.1× speedup. This demonstrates that the high measured sparsity (93.52%) translates into genuine wall-clock acceleration when using sparsity-aware inference software.

Why the speedup is only 4.1× when sparsity is 93.52%: in theory, saving 93.5% of FFN computation should yield a ~15× speedup if FFNs dominated total compute. The smaller observed speedup reflects several practical factors: (1) attention computation is not affected by FFN sparsity and becomes a larger fraction of total cost as FFNs are accelerated; (2) the online activation predictors have imperfect accuracy, so some neurons that could be skipped are computed anyway as a safety margin; (3) memory movement and kernel launch overhead become significant when the FFN computation is so cheap. The paper doesn't provide a breakdown, but the 4.1× number is presented as a concrete demonstration of practical benefit rather than a theoretical upper bound.

Design Choices and Their Justifications (Summary)

µP parametrization over standard Transformer initialization: ensures stable training dynamics across different model scales and width-depth ratios without per-configuration hyperparameter tuning, making fair comparisons across architectures possible.
CETT with L2 norm of neuron output vectors over raw activation value thresholding: accounts for the downstream impact of skipping neurons, adapts to per-layer contribution distributions, and generalizes across activation functions (including SiLU's negative outputs).
PPL increase tolerance (p=1%) over fixed CETT threshold: provides a principled, performance-calibrated way to set the single hyperparameter, ensuring the measured sparsity reflects what the model can actually tolerate without meaningful degradation.
Sparsity stabilizing strategy (last five checkpoints, warmup dropped): reduces measurement noise from stochastic training dynamics and early-training fluctuations that would otherwise pollute the curve fitting.
Activation ratio (rather than sparsity ratio) for curve fitting: empirically produces cleaner functional forms (decreasing logspace power-law for ReLU, increasing vanilla power-law for SiLU) with better convergence properties during Levenberg-Marquardt optimization.
Data-scale ratio (training tokens ÷ parameters) for convergence speed comparison: normalizes for the fact that larger models require more data to reach equivalent training states (following Chinchilla-style scaling principles), enabling fair comparison of sparsity convergence dynamics.
Combinatorial model for explaining scale-insensitivity: provides a mechanistic hypothesis (neuron specialization search space grows factorially with neuron count) that is consistent with all empirical observations and generates testable predictions (e.g., sparsity should converge even slower at scales beyond 1.2B, but the limit should remain constant).
Width-depth ratio bottleneck analysis: identifies that sparsity can be improved by going deeper, but only up to the point where training instability causes performance collapse, giving practitioners a concrete methodology for finding the optimal depth rather than a fixed recipe.

4. Key Insights and Innovations

Innovation 1: A Performance-Calibrated Sparsity Metric That Decouples Measurement Method From Measurement Target

The field's approach to measuring activation sparsity has been fractured by a fundamental circularity: the metric used to count weakly-contributed neurons is entangled with assumptions about which activation function is being measured and what "weakly-contributed" means. The straightforward ReLU metric (count zeros) is useless for SiLU; FAT-ε's global threshold is impossible to calibrate without oracle knowledge of acceptable degradation; Top-k imposes an arbitrary uniform sparsity budget that ignores per-layer heterogeneity. Each prior metric was effectively a method and a definition rolled into one, making it impossible to ask "how sparse is this model, really?" without the answer being an artifact of the measurement instrument.

The paper resolves this by introducing a two-level calibration hierarchy that separates the recognition mechanism from the calibration criterion. CETT handles recognition: it uses per-layer L2 norm relative error to adaptively set thresholds, which is a mechanism-level improvement over prior approaches. But the deeper conceptual move is CETT-PPL-p%, which introduces a deployment-relevant calibration criterion—"how much can we skip while keeping perplexity degradation below p%?"—that is independent of the recognition mechanism. You could in principle plug a different neuron-identification algorithm into the same PPL-calibration framework and get comparable sparsity measurements. The metric is defined not by how neurons are identified but by what performance envelope the model must stay within.

This reframes sparsity measurement from a purely structural question ("how many activation values are near zero?") to a capability-relative question ("how much computation can this specific model shed on this specific data distribution without meaningful degradation?"). The choice of p=1% is then not an arbitrary threshold but an empirically validated boundary: Table 1 shows that at p=1%, the average downstream performance is essentially unchanged (−0.16 points on commonsense reasoning, −0.30 on reading comprehension), while at p=5%, reading comprehension already degrades by −2.21 points. The metric bakes in an implicit claim about what constitutes "negligible" performance impact, and the paper provides evidence that this claim holds.

This is a fundamental shift from prior practice, where sparsity ratios were reported without any calibration to model capability—a model might be declared "95% sparse" under some threshold, but no one knew whether that 95% was genuinely recoverable without harming output quality. CETT-PPL-1% makes the measurement operationally meaningful: if you build an inference engine that implements CETT's neuron-skipping decisions with the PPL-1% calibration, you will actually achieve that speedup at negligible quality cost. The paper doesn't just provide a better ruler; it provides a ruler whose markings correspond to real-world deployability.

Innovation 2: Sparsity As a Convergent, Law-Governed Quantity—And the Discovery That Activation Function Choice Inverts the Sign of the Data-Sparsity Relationship

Prior work treated activation sparsity as a qualitative property—models "exhibit sparsity" or "become sparser with scale"—without characterizing it as a quantity that follows predictable mathematical laws. The paper's central empirical contribution is establishing that the activation ratio follows well-defined, convergent functional forms with respect to training data amount, and critically, that the direction of this relationship is determined by the activation function.

For ReLU, the activation ratio follows a decreasing logspace power-law (Equation 4): more data monotonically reduces the fraction of active neurons, asymptotically approaching a limit A₀ that represents the genuinely necessary computation. For SiLU, the relationship inverts: the activation ratio follows an increasing vanilla power-law (Equation 5), meaning more training data increases the fraction of active neurons and degrades sparsity.

This finding subverts the implicit assumption that sparsity is a fixed architectural property. It is instead a training-dynamic property that evolves according to laws with predictable functional forms, fitted parameters, and asymptotic limits. The concept of a "limit sparsity ratio" A₀—the activation ratio approached as training data goes to infinity—provides a theoretically grounded way to compare architectures that abstracts away from transient training effects. Two models might have the same sparsity at 100B tokens but converge to different limits; or they might share the same limit but one reaches it much faster (Figure 8). Without the concept of a limit and a fitted functional form, these distinctions would be invisible.

The sign inversion between ReLU and SiLU is particularly significant because it identifies a causal mechanism: ReLU's hard zero regime (negative pre-activations clamped to exactly zero) allows the model to permanently deactivate neurons that prove unnecessary, creating a ratchet effect where sparsity can only increase. SiLU's smooth negative tail means that even "mostly inactive" neurons retain small non-zero contributions, and these small contributions apparently become more numerous as training progresses—the model "fills in" its activation pattern rather than "burning off" excess. This is not just an empirical observation but a diagnostic insight: the paper is essentially demonstrating that the training dynamics of neuron specialization are qualitatively different depending on whether the activation function provides a true zero output.

The fitted coefficients in Table 2 reveal a systematic pattern: for ReLU, the exponent α increases with scale (0.101 → 1.53 from 0.1B to 2.4B), meaning larger models transition from sub-linear to super-linear convergence toward their limit. The coefficient c drops dramatically (3.20 → 0.000156), meaning larger models are initialized closer to their asymptotic sparsity. These patterns are not fitted independently per scale—they form a coherent narrative about how sparsity dynamics scale, and they hold when extrapolated to the 2.4B validation model (Figure 11). This is a fundamental contribution to the scaling-laws literature, extending the paradigm from loss-based scaling laws (Hoffmann et al., 2022) to an efficiency property that directly governs inference cost.

Innovation 3: The Scale-Insensitivity of Asymptotic Sparsity and the Combinatorial Explanation for Convergent Dynamics

The finding that activation sparsity is largely scale-insensitive—with limit activation ratios varying by only 1.7 percentage points (ReLU) or 2.7 points (SiLU) across a 12× parameter range—directly contradicts the prior literature's suggestion that larger models are inherently sparser (Li et al., 2022). This is not an incremental correction but a conceptual reframing: the paper argues that what prior work observed as "larger models are sparser" was actually a confound—larger models were compared at fixed training budgets rather than at their respective asymptotic limits, and since smaller models converge faster to their limit (Figure 8), they appear less sparse when measured at the same point in training, even though their eventual sparsity is nearly identical.

The paper goes beyond empirical documentation to provide a mechanistic hypothesis for why this scale-insensitivity should hold. The combinatorial neuron specialization model (Equation 6) is unusually elegant for an empirical ML paper: if neurons must be partitioned into G functional groups with fixed proportions across scales, the number of possible assignments grows factorially with neuron count, making the specialization search space combinatorially larger for bigger models. This explains both the similar asymptotic limits (the proportions are fixed) and the slower convergence (the search space is vastly larger). The model generates a testable prediction: sparsity convergence should continue to slow at scales beyond 1.2B, but the limit should remain roughly constant.

The evidence for the assumption underlying this model—that neuron specialization patterns are scale-invariant—is provided through two complementary experiments that are themselves methodologically novel. The dataset-wise activation frequency distributions (Figure 9) demonstrate that the shape of the activation distribution (what fraction of neurons are generalists vs. specialists) is conserved across scales. The token-wise activation ratio comparison (Figure 10) shows that individual tokens activate similar fractions of neurons regardless of total neuron count. Together, these suggest that what scales with model size is not the functional organization but the granularity within each functional group—more neurons doing the same kind of work, not more kinds of work.

This is a diagnostic framework as much as an empirical finding. It provides a lens for understanding sparsity dynamics that can be applied to new architectures or training recipes: if a given change preserves the functional group proportions, the limit sparsity should remain constant and only the convergence speed should change; if it alters the proportions, the limit itself should shift. The width-depth ratio experiments (Section 5.2) can be reinterpreted through this lens: making a model deeper (lower width-depth ratio) might reduce the number of functional groups needed per layer (since functions can be distributed across more layers), shifting the per-layer group proportions and thus the limit sparsity. The paper doesn't make this connection explicitly, but the framework enables it.

Innovation 4: The Identification of Width-Depth Ratio As a Sparsity-Control Parameter With a Stability-Limited Bottleneck

Prior work on Transformer architecture design focused on the width-depth tradeoff primarily through the lens of loss scaling (e.g., how performance varies with shape for fixed parameter count) or training dynamics (deep networks are harder to optimize). The paper introduces a novel axis: sparsity as a function of model shape, finding that the width-depth ratio linearly controls the asymptotic activation ratio below a bottleneck point (Figure 5), with deeper (lower-ratio) models being sparser.

This is significant not because the relationship is linear—linear relationships are the simplest possible—but because it identifies a new optimization criterion for architecture design that is distinct from loss. Figure 6 shows that the limit training loss is essentially flat across a wide range of width-depth ratios (74 to 282 for 0.1B). This means that the loss-optimal model shape is underdetermined: many shapes achieve similar final performance. The sparsity curve (Figure 5) provides a secondary criterion that breaks this degeneracy—among architectures with equivalent loss, prefer the deeper one because it will be sparser at deployment.

The bottleneck phenomenon is also conceptually important. Below a width-depth ratio of approximately 114 (for 0.1B), sparsity improves linearly with depth; above it, sparsity saturates and only fluctuates. This suggests there is a maximum depth beyond which adding more layers doesn't further specialize neurons—perhaps because the functional groups have already been fully distributed across the available layers and additional layers just add redundant capacity. The bottleneck point represents the architectural sweet spot where depth has been maximized for sparsity without entering the diminishing-returns regime.

However, the paper also identifies the practical constraint that limits how far this optimization can be pushed: training instability at very small width-depth ratios (below ~74 for 0.1B), where the limit loss sharply increases (Figure 6). The resulting prescription—"choose the smallest width-depth ratio within the stability interval"—is a practitioner-ready design rule that was not previously articulated in the sparsity literature. It transforms width-depth ratio from a hyperparameter to be tuned solely for performance into a lever for controlling inference efficiency, with clear boundaries on its effective range.

This finding also connects to the scale-insensitivity results: if the width-depth ratio controls sparsity but is held approximately constant across the paper's main experimental scales (48–56), then the scale-insensitivity of sparsity is partially an artifact of the experimental design—the models were not merely similar in scale but similar in shape. A larger model with a very different width-depth ratio might show different asymptotic sparsity. The paper doesn't explore this interaction (scale × shape), but the width-depth ratio analysis provides the conceptual tool for doing so. This is therefore a framework contribution as much as an empirical finding: it gives the field a variable to control and a methodology (bottleneck identification) for choosing its value, even if the specific numerical bottleneck will differ for different model scales and training recipes.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the MATH benchmark (Hendrycks et al., 2021), consisting of high-school competition-level math problems. The authors use the specific split from Lightman et al. (2022): 12,000 training questions and 500 test questions. The choice of MATH is deliberate (Section 4): test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences—mathematical reasoning fits this profile because it requires multi-step logical deduction rather than novel factual recall.
Base model(s). All experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" and sits in a useful regime: non-trivial performance on MATH (roughly 10–19% pass@1 depending on the prompt and sampling configuration) but far from saturation, leaving room for test-time compute to make a difference. For the FLOPs-matched comparison, a second model with approximately 14× more parameters is used as the pretraining-scaled baseline.
Metrics. The primary metric throughout is MATH test accuracy (%)—the fraction of the 500 test questions for which the selected final answer matches the ground truth. Answers are graded using the grading function released by Lightman et al. (2022) (Appendix G). When analyzing difficulty-dependent behavior, the paper reports accuracy within each of the five difficulty quintiles separately.
Baselines. The paper uses several baselines:
- Majority voting: select the most common final answer among N sampled solutions (no learned verifier).
- ORM best-of-N weighted: score N solutions with an outcome reward model and apply best-of-N weighted selection.
- PRM best-of-N weighted: score N solutions with the process reward model and apply best-of-N weighted selection.
- Parallel sampling (for revisions): generate N independent solutions from the revision model and select the best via verifier or majority.
Generation budget / compute accounting. One "generation" equals one complete sampled answer from the base LLM. For beam search and best-of-N, the budget equals the number of beams or samples N. For lookahead search with k lookahead steps, the cost is N × (k+1) to account for the additional rollout computation (Section 5.3). Budgets are swept across powers of 2, typically from 2⁰ to 2⁹ (1 to 512 generations).
Cross-validation / statistical protocol. To avoid contaminating strategy selection with test-set performance, the authors use two-fold cross-validation within each difficulty bin on the 500-question test set. The best strategy is selected on one fold and evaluated on the other, with results averaged (Section 3.2).

Main Quantitative Results

Search Against PRM Verifiers (Section 5)

Aggregate search algorithm comparison (Figure 3, left). Across all 500 test questions with a maximum budget of 256 generations:

At low budgets (2–8 generations), beam search with M = 4 significantly outperforms best-of-N weighted. For example, at 4 generations beam search (M = 4) achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted—a substantial gap.
At high budgets (64–256), beam search performance flattens and falls slightly below best-of-N weighted. Best-of-N weighted reaches approximately 38% at 512 generations; beam search (M = 4) plateaus around 34%.
Lookahead search (both k = 1 and k = 3) generally underperforms at the same generation budget due to its higher per-step cost. The 3-step lookahead variants converge to similar performance as other methods at very high budgets but never surpass them.
Majority voting trails all verifier-based methods substantially, reaching only about 29% at 512 generations.

Difficulty-bin analysis for search (Figure 3, right). The per-difficulty breakdown (beam search M = 4 vs. best-of-N weighted, shown at four budget levels: 4, 16, 64, 256 generations) reveals the core pattern:

Bin 1 (easiest): Beam search accuracy decreases from roughly 78% to 77% as the budget goes from 4 to 256, while best-of-N weighted increases from 68% to 88%. This is the clearest evidence of PRM over-optimization—beam search finds solutions that exploit the verifier signal.
Bin 2: Beam search improves modestly (roughly 14% → 32%) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear advantage at high budgets.
Bin 3: Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations.
Bin 4: Beam search shows the strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations.
Bin 5 (hardest): Both methods hover near 1–3% regardless of budget. No method makes meaningful progress.

Compute-optimal search (Figure 4). By selecting the best search strategy per difficulty bin at each budget level:

At 16 generations, compute-optimal (oracle bins) achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations—a 4× compute reduction.
At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%).
Compute-optimal with predicted difficulty bins tracks the oracle version closely, particularly at lower budgets. The two curves "largely overlap" per the authors (Figure 4), with the predicted version reaching approximately 37% at 256 generations.
Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%).

PRM vs. ORM (Figure 14, Appendix F). At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with the number of samples, confirming the PRM's superior scaling properties.

Revision Model Results (Section 6)

Revision model pass@1 trajectory (Figure 6, left). Starting from approximately 18.2% pass@1 at step 1, the revision model's per-step accuracy improves to roughly 24–25% by steps 15–20, and remains in the 23–25% range out to 64 steps. The model generalizes beyond its 4-step training horizon.

Sequential vs. parallel (Figure 6, right). At 64 generations:

Sequential + best-of-N weighted: approximately 41.5%
Parallel + best-of-N weighted: approximately 39%
Sequential + majority: approximately 38%
Parallel + majority: approximately 35%

Sequential outperforms parallel under both selection mechanisms, with the verifier-based gap (roughly 2.5 percentage points) being slightly narrower than the majority-based gap (roughly 3 points).

Sequential-to-parallel ratio sweep (Figure 7, left). For a fixed generation budget, varying the ratio reveals:

At 256 generations, the optimal ratio is around 2¹ to 2³ (2:1 to 8:1 sequential-to-parallel), achieving approximately 43–44% accuracy.
Fully parallel (leftmost point) yields approximately 40%.
Fully sequential (rightmost point) yields approximately 42%.
At lower budgets (8–32 generations), fully sequential is optimal—the curves are monotonically increasing with the sequential-to-parallel ratio.

Difficulty-dependent ratio (Figure 7, right). At a fixed budget of 128 generations:

Bin 1: Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to the allocation strategy.
Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel.
Bin 3: A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2¹ to 2³), reaching approximately 42% vs. 35% at the extremes.
Bin 4: Similar pattern, with the peak at a moderate ratio achieving roughly 18% vs. 14% at fully parallel.
Bin 5: All ratios produce roughly 2–3% accuracy. No allocation strategy helps.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin:

At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations—a 4× improvement.
At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
Compute-optimal predicted bins perform slightly below oracle bins at high budgets (approximately 41% at 256 generations) but still substantially outperform the parallel baseline.
Notably, the parallel baseline appears to plateau around 36–37% at high budgets, while compute-optimal scaling continues to improve, suggesting that the gains from adaptive allocation compound at higher budgets.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

Revisions (Figure 9, left; Figure 1, top-right bar chart). Comparing PaLM 2-S* with compute-optimal revisions against the ~14× larger model:

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy (bin 1)	+11.8%	+3.5%	−11.9%
Medium (bin 2–3)	+27.8%	+16.7%	+5.4%
Hard (bins 4–5)	+21.6%	−(implied negative)	−37.2%

(Numbers from the bar chart in Figure 1, top-right. Note: the "easy/medium/hard" groupings in the bar chart differ slightly from the five difficulty bins, aggregating bins for readability.)

At R ≪ 1, test-time compute outperforms the larger model across all difficulty levels. At R ≫ 1, it only remains preferable on easy questions, with hard questions showing a −37.2% relative disadvantage.

PRM search (Figure 9, right; Figure 1, bottom-right bar chart). The pattern is starker:

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy	+19.1%	+2.2%	+2.0%
Medium	0.0%	−35.3%	−30.8%
Hard	−3.6%	−35.3%	−52.9%

PRM search shows weaker benefits than revisions for the FLOPs-matched comparison, with substantial disadvantages on medium and hard questions even at moderate R values. On easy questions, test-time compute remains preferable across all R regimes, though the margin narrows significantly.

Figure 9 detail. The line plots show accuracy per difficulty bin as test-time compute scales. The 14× larger model's greedy performance (stars) is placed at three x-axis positions corresponding to the three R values. Where the compute-optimal scaling line is above the star, test-time compute wins. On bin 1 (purple, topmost line), the scaling line is above all three stars for revisions. On bin 5 (blue, bottommost line), the line is below all three stars and essentially flat near 0–5%, confirming that no amount of test-time compute helps on the hardest problems.

Ablation Studies and Robustness Checks

PRM aggregation strategy (Appendix E, Figure 13). Comparing "min," "prod," and "last" step-wise aggregation: "Last" achieves roughly 37% at 256 samples; "Min" achieves roughly 35%; "Prod" achieves roughly 27%; ORM achieves roughly 34%. The "last" aggregation's superiority is notable because it effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM. The authors interpret this as evidence that step-level PRM training provides beneficial representation learning.

PRM vs. ORM (Appendix F, Figure 14). The PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%.

Revision model verifier choice (Appendix J, Figure 15a). The base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs, with sequential + base-LM PRM achieving roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern when applying verifiers trained on one model's outputs to another model's outputs.

Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's context provides a small improvement over the no-history ablation (approximately 1–2 percentage points at 64 generations), but both variants outperform the parallel baseline, confirming that the sequential sampling benefit is not solely attributable to the verifier seeing more context.

Oracle vs. predicted difficulty bins (Figures 4, 8, and Appendix C, Figures 11–12). Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. Predicted bins show slightly lower performance at high budgets in the revision setting (roughly 41% vs. 44% at 256 generations in Figure 8) but essentially identical performance in the search setting (Figure 4). This is the critical robustness check: the compute-optimal strategy works without ground-truth labels.

Majority voting for revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate.

ReSTᴱᴹ revision model (Appendix K, Figure 16). An attempt to further optimize the revision model using ReSTᴱᴹ (Singh et al., 2024) backfires: additional sequential revisions substantially hurt performance with this model. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that the on-policy data collection in ReSTᴱᴹ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly. This is a notable negative result that highlights the sensitivity of revision training to the data generation procedure.

Critical Assessment

On the claim that compute-optimal scaling improves efficiency by more than 4× over best-of-N: The evidence in Figures 4 and 8 supports this at moderate budgets (16–64 generations), where compute-optimal strategies achieve accuracy that best-of-N requires 4× more compute to match. However, the gap shrinks at very high budgets with predicted difficulty bins: in Figure 8, compute-optimal predicted achieves approximately 41% at 256 generations vs. best-of-N weighted at approximately 37% at the same budget—an improvement but closer to 1.5× than 4×. More importantly, the difficulty estimation cost (2048 samples per question) is not included in the budget accounting. If included, the effective budget for compute-optimal scaling would be substantially higher, potentially erasing the reported gains for all but the highest-throughput deployment scenarios. The paper acknowledges this explicitly but does not quantify how the efficiency claims change when estimation is amortized. The 4× figure should be understood as the efficiency gain conditional on difficulty already being known, which is a narrower claim than "deployable improvement."

On the claim that test-time compute with a smaller model can outperform a 14× larger model: This holds most cleanly at R ≪ 1 (low inference-to-pretraining ratio) for easy-to-medium problems. On bin 5 (hardest problems), test-time compute provides essentially no benefit regardless of strategy or budget (Figures 3, 7), and the FLOPs-matched comparison in Figure 9 shows a flat line near zero accuracy for bin 5. The paper is transparent about this boundary, but it means the claimed substitution is only valid when the problem distribution is skewed toward questions the model can already partially solve. The 14× larger baseline uses greedy decoding with no test-time augmentation, making it a weaker baseline than a fair comparison might warrant—a larger model with even modest best-of-N sampling could shift the crossover point. The paper also scales only parameters (not data), departing from Chinchilla-optimal pretraining, which likely makes the larger model baseline weaker than a compute-optimally trained larger model would be. These two choices (no inference budget for the larger model, non-optimal pretraining allocation) systematically favor the test-time compute approach, and the crossover R values would likely shift if either were corrected.

On the claim that difficulty estimation can be done without ground-truth labels: The predicted difficulty bins using the PRM's average final-answer score perform similarly to oracle bins (Figures 4, 8), which is genuinely encouraging. However, the method still requires generating 2048 samples per question—an enormous up-front cost. The paper does not test whether difficulty estimation with far fewer samples (e.g., 16 or 64) would suffice, which is the most practically relevant question. If 2048 samples are genuinely needed, the method is only applicable in offline batch settings where amortization is possible. If 16 samples work nearly as well, the practical relevance is much higher. This experiment is conspicuously absent.

On the single-benchmark, single-model-family limitation: All results are on MATH with PaLM 2-S. The paper argues the model is "representative," but no evidence is provided that the difficulty-dependent scaling patterns, revision dynamics, or FLOPs-matched tradeoffs generalize to other model families (GPT, LLaMA, etc.) or other reasoning domains (code generation, logical reasoning, scientific QA). The MATH benchmark has specific properties—ground-truth answers verifiable by exact match, symbolic rather than linguistic reasoning, competition-level difficulty distribution—that may make it particularly amenable to verifier-guided search. Code generation benchmarks with unit tests might show similar patterns, but open-ended reasoning or factual QA might not. The 500-question test set, split into quintiles of ~100 questions, then further split by two-fold cross-validation (~50 per fold per bin), provides limited statistical power for the per-bin strategy selection. The paper does not report confidence intervals, making it impossible to assess whether the bin-specific strategy choices are reliable or whether they would change with a different random split.

On the non-combination of search and revisions: The paper studies PRM search and iterative revisions independently but never applies beam search to revision model outputs or uses the PRM to guide which revisions to pursue. This means the reported results represent a lower bound on what combined approaches could achieve, and the paper's claims about the effectiveness of test-time compute are conservative. However, it also means we do not know whether combining them would yield additive gains (each method covering different problem types) or subadditive gains (they overlap in the problems they help). The complementary difficulty-dependent patterns (revisions best on easy, search best on medium) suggest additive gains, but this remains untested.

On the revision model's fragility: The ReSTᴱᴹ ablation (Appendix K) is a genuinely important negative result that the main text does not emphasize enough. It demonstrates that the revision training procedure is brittle: changing the data generation strategy from offline, edit-distance-based pairing to on-policy rollouts caused the revision model to actively degrade performance when chained sequentially. This means the positive revision results in Section 6 depend critically on specific data construction choices, and practitioners cannot assume that any revision training recipe will produce chainable improvements. The 38% correct-to-incorrect reversion rate (noted in Section 6.1) further underscores that the revision model has not learned a robust "improve if wrong, preserve if correct" policy. The within-chain selection mechanism (majority voting or verifier) patches this at inference time, but a model that genuinely understood when to stop would be substantially more efficient.

On missing latency analysis: The paper measures compute in "generations" (proportional to total FLOPs) but ignores wall-clock time. Sequential revisions are inherently serial—each step depends on the previous one's full generation—while parallel best-of-N can be batched. A compute-optimal strategy allocating budget as 64 sequential × 2 parallel takes effectively 64 serial steps, vs. 256 parallel generations that could complete in a single step with sufficient hardware. For latency-sensitive applications (which include most user-facing deployments), the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical even if they are FLOPs-efficient. The paper never acknowledges this latency-FLOPs tension.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The entire compute-optimal scaling framework depends on knowing each prompt's difficulty before allocating the test-time compute budget. The paper's proposed method for estimating difficulty—whether using oracle labels (ground-truth correctness) or predicted labels (PRM average final-answer score)—requires generating 2048 complete solutions per question and scoring every step through the PRM. The paper states this explicitly in Section 3.2:

"we approximate [ground-truth correctness] by replacing it with the PRM [score]... This still requires sampling 2048 solutions from the base LLM for each question... our experiments do not account for this cost largely for simplicity"

To put this in perspective: for a single question, the difficulty estimation step consumes 2048 generations—which is 8× to 128× more than the test-time budgets studied (16 to 256 generations). The entire premise of the compute-optimal framework is that we can save computation by allocating the test-time budget more intelligently than a uniform best-of-N baseline. Yet the difficulty estimation step, if included in the accounting, would dominate the total compute for all but the most extreme amortization scenarios.

The consequence. The reported 4× efficiency gains (e.g., compute-optimal search at 16 generations matching best-of-N at 64 generations; Figure 4; compute-optimal revisions at 64 generations matching best-of-N at 256 generations; Figure 8) are computed conditional on difficulty already being known. This is the equivalent of reporting a sorting algorithm's speed without counting the time to read the input. In any deployment where difficulty must be estimated from scratch for new prompts, the actual total cost would be:

$\text{total cost} = 2048 \text{ (estimation)} + N_{\text{strategy}} \text{ (execution)}$

For low-budget strategies (N = 4–64), the estimation cost exceeds the execution cost by 32× to 512×. The compute-optimal approach would be dramatically more expensive than the baseline it is compared against, not 4× cheaper, when estimation is included.

Even in amortized settings—e.g., estimating difficulty once and applying the same strategy to many similar questions—the paper provides no analysis of how many questions are needed to amortize the 2048-sample cost, nor does it test whether difficulty estimation with far fewer samples (e.g., 16 or 64) would suffice. If difficulty can be reliably estimated from 16 samples, the amortization burden drops by 128× and the approach becomes much more practical; if 2048 are genuinely necessary, the method is applicable only in high-volume batch processing where estimation cost is negligible per question. The crucial ablation—varying the number of samples used for difficulty estimation and measuring the resulting accuracy of the compute-optimal policy—is entirely absent.

What evidence exists in the paper. Figures 4 and 8 plot compute-optimal scaling curves comparing oracle and predicted difficulty bins. Both use the full 2048-sample estimation procedure; neither includes the 2048 samples in the generation budget shown on the x-axis. The paper acknowledges this as a limitation in Section 3.2 and flags it as future work ("leave this direction for future work"), but does not quantify how the efficiency claims change under any alternative estimation budget.

Mitigation status. Not addressed. The paper explicitly defers to future work, suggesting models that predict difficulty directly from the question text without requiring generation. No such model is developed or evaluated. Until a cheap difficulty estimator exists, the headline 4× efficiency figures should be understood as conditional on an oracle difficulty signal, not as realized deployment gains.

The Method Provides No Benefit on the Hardest Problems

The assumption or constraint. The test-time compute framework operates entirely by amplifying the base model's existing capability—search finds correct solutions within the model's output distribution, and revisions refine answers that are already roughly on track. It does not create new capabilities. This means that for problems where the base model's pass@1 is near zero—where it essentially never generates a correct answer, even among thousands of attempts—no amount of test-time compute helps. The paper states this boundary condition explicitly in Section 7:

"test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help"

The consequence. The hardest 20% of problems on the MATH benchmark (difficulty bin 5, where the base model's pass@1 is approximately 0–3%) show near-zero accuracy regardless of test-time compute strategy or budget. The evidence is unambiguous across every experiment:

Search (Figure 3, right, bin 5 at bottom): Both beam search and best-of-N weighted hover at 1–3% accuracy across all budgets from 4 to 256 generations. The lines are flat—more compute does nothing.
Revisions (Figure 7, right, bin 5 at bottom): All sequential-to-parallel ratios produce approximately 2–3% accuracy. No allocation strategy moves the needle.
FLOPs-matched comparison (Figure 9, bin 5 curve): The compute-optimal scaling line for the hardest problems is essentially flat near 0–5% across all test-time budgets. The 14× larger model performs dramatically better.

This is a hard capability ceiling, not a gradual diminishing return. It means the compute-optimal approach cannot be used for problems genuinely outside the model's training distribution or requiring reasoning patterns the model never acquired during pretraining. For such problems, scaling pretraining (larger models, more data, better data) is the only viable path.

What evidence exists in the paper. Every difficulty-bin breakdown (Figures 3 right, 7 right, 9) consistently shows bin 5 at near-baseline performance. The paper is transparent about this—the "takeaway" box in Section 7 explicitly states that test-time compute is ineffective for hard problems. Table 1 in Section 7 shows that on hard problems (bins 4–5), the FLOPs-matched comparison at R ≪ 1 gives only a +21.6% relative improvement for revisions and −3.6% for PRM search, meaning search actually underperforms the larger model even at the most favorable inference-to-pretraining ratio. At R ≫ 1, hard problems show −37.2% (revisions) and −52.9% (search) relative to the 14× larger model.

Mitigation status. None. This is a fundamental limitation of the approach, not a gap that can be closed with better methods. The paper does not claim otherwise, but practitioners need to understand that the compute-optimal framework only applies to problem distributions where the base model already has non-trivial capability. A deployment pipeline using this approach would need a separate mechanism (e.g., a larger model, human intervention) for handling out-of-capability problems.

The FLOPs-Matched Pretraining Baseline Is Weakened by Two Design Choices That Favor the Test-Time Compute Approach

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. The paper acknowledges two choices that make this comparison favorable to test-time compute:

Choice 1: Scaling parameters only, not data. The larger model is constructed by scaling only the parameter count while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022) where both parameters and data are scaled equally. The paper states in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Choice 2: The larger model uses only greedy decoding. The larger model receives no test-time augmentation at all—no majority voting, no best-of-N sampling, no verifier-guided selection. It is compared in its weakest inference configuration (single greedy decode) against the smaller model in its strongest configuration (compute-optimal strategy search + revisions + verifier selection).

The consequence. Both choices systematically bias the comparison in favor of test-time compute, potentially by a large margin. A Chinchilla-optimal larger model (scaling data proportionally with parameters) would be trained on approximately √14 ≈ 3.7× more data than the smaller model, which would likely improve its performance substantially—particularly on knowledge-intensive problems where data quantity directly influences capability. The paper's parameter-only scaling means the larger model may be significantly undertrained relative to its capacity, making it a weaker baseline than a compute-optimally trained counterpart.

The greedy decoding choice is arguably more impactful. Giving the larger model even a modest test-time compute budget—say, best-of-8 or majority voting over 16 samples—would shift the crossover R values substantially. The paper's own results show that best-of-N weighted improves accuracy by roughly 10–20 percentage points over the best-of-1 baseline (Figure 3, left), and these gains are largest on easy-to-medium problems where the model already has capability. The larger model, with its inherently higher base accuracy, would benefit more from test-time compute augmentation than the smaller model on those same problems, making the "test-time compute with small model beats large model" claim much harder to sustain in a fair comparison.

What evidence exists in the paper. The paper is transparent about both choices (Section 7). However, no ablation is conducted to test how the FLOPs-matched comparison changes if the larger model is given any test-time compute budget (even best-of-4 or best-of-8), and no Chinchilla-optimal larger model is trained for comparison. The star markers in Figure 9 represent a single greedy decode from the larger model—a bar that a larger model with modest test-time augmentation would likely clear easily, particularly at R ≫ 1 where the larger model's parameter count already dominates the total FLOPs budget anyway.

Mitigation status. The paper acknowledges these as limitations and suggests future work on compute-optimal pretraining + inference jointly, as well as on comparisons with larger models that also use test-time compute (Section 8). However, the absence of these baselines means the reported FLOPs-matched crossover points (e.g., "test-time compute beats 14× larger model on easy problems at R ≪ 1") should be interpreted as upper bounds on the advantage of test-time compute over pretraining, not as definitive comparisons.

All Results Are on a Single Benchmark With a Single Model Family, and the Test Set Is Small for Per-Bin Strategy Selection

The assumption or constraint. The entire empirical study—scaling laws for search, revision dynamics, difficulty-dependent optimal strategies, and FLOPs-matched comparisons—is conducted on a single benchmark (MATH, 500 test questions) using a single model family (PaLM 2-S*). The paper states (Section 4):

"We choose the MATH dataset as it represents a domain where test-time compute is likely to prove most useful: problems where the model already possesses the necessary knowledge but the challenge lies in drawing complex inferences... we believe [PaLM 2-S*] is representative of the capabilities of many contemporary LLMs"

These are reasonable choices for an initial study, but they impose significant constraints on the generality of the findings. The MATH benchmark has specific properties—closed-form answers verifiable by exact string match, competition-level difficulty distribution, symbolic rather than linguistic reasoning—that make it particularly amenable to verifier-guided search and revision. Code generation benchmarks with unit tests might show similar patterns; open-ended reasoning, factual QA, or dialogue tasks might show fundamentally different dynamics.

The consequence. We cannot know whether the key findings—ReLU's superiority for sparsity, the decreasing logspace power-law, the 4× efficiency gains from compute-optimal scaling—generalize to other model families, other reasoning domains, or other task formats. Several aspects of the findings could be model-specific: the PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution, which may not replicate for models with different training data, tokenization, or architecture. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families.

More critically, the test set of 500 questions, split into five difficulty quintiles (~100 each) and further split by two-fold cross-validation (~50 questions per fold per bin), means the compute-optimal policy is selected based on roughly 50 questions per fold per bin. When the paper claims that "beam search is optimal for bin 3 but best-of-N weighted is optimal for bin 1 at 64 generations," this claim rests on the accuracy difference observed on ~50 questions. The paper does not report confidence intervals on the per-bin strategy selection or on the resulting compute-optimal scaling curves, making it impossible to assess whether the selected strategies are statistically reliable or would change substantially with a different random split. At 50 samples per bin, a single question where beam search accidentally succeeds or fails could flip the "optimal strategy" for that bin, potentially cascading into different compute-optimal scaling results.

What evidence exists in the paper. The two-fold cross-validation protocol (Section 3.2) is described as mitigating overfitting to the test set, but the paper does not report variance across folds or conduct sensitivity analysis (e.g., leave-one-out, bootstrapping, varying the number of folds). The per-bin sample sizes are evident from the methodology (~100 questions per quintile, halved by cross-validation). The paper acknowledges the single-benchmark, single-model limitation only obliquely (Section 4: "we leave the analysis of compute-optimal scaling of pretraining compute... to future work").

Mitigation status. Not addressed for the per-bin sample size issue. For the single-benchmark, single-model limitation, the paper implicitly argues that MATH is a representative reasoning benchmark and PaLM 2-S* is a representative LLM, but provides no evidence. Future work on other benchmarks (code generation, logical reasoning) and other model families (GPT, LLaMA, etc.) would be needed to establish generality, and the paper does not claim to have done so.

The Revision Model Is Brittle and Exhibits a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is trained on sequences of 0–4 incorrect answers followed by a correct answer, with the last incorrect answer selected to have minimal character-level edit distance to the correct answer (Section 6.1). This training procedure teaches the model to always try to improve the previous answer, but it never sees training examples where the previous answer is already correct and should be preserved. The paper reports:

"approximately 38% of correct answers are converted back to incorrect answers by subsequent revision steps if the model naively applies the revision model without any selection mechanism"

This is a direct consequence of the training data construction: the model's training distribution is exclusively incorrect→correct trajectories, so at inference time, when the model encounters a correct answer produced by an earlier revision step, it has no learned behavior for "recognize that this is already right and output it unchanged." Instead, it applies the same revision logic that it learned for incorrect answers, sometimes introducing errors.

The consequence. The revision model cannot be naively chained—taking the output of revision step N as the input to revision step N+1 and trusting the final output—because it will corrupt correct answers roughly 38% of the time. This necessitates the within-chain selection mechanism (majority voting or verifier-based selection across all steps) described in Section 6.1 and Appendix I, which adds computational overhead (scoring multiple candidate answers within each chain) and complexity. It also means the revision chain is not a monotonic improvement process: accuracy oscillates along the chain, with correct answers being generated, then broken, then partially repaired, then broken again. Figure 6 (left) shows this: pass@1 at each step fluctuates in the 23–25% range rather than smoothly increasing, consistent with a process that is generating improvements and degradations in roughly equal measure.

More importantly, the revision training procedure is sensitive to data construction choices in ways that are not fully understood. The ReSTᵉᴹ ablation (Appendix K, Figure 16) demonstrates that switching from the paper's offline, edit-distance-based data pairing to on-policy multi-turn rollouts causes the revision model to actively degrade performance when chained sequentially. At 256 generations, fully sequential performance with the ReSTᵉᴹ-trained model drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio, and is worse than the parallel baseline. This is a striking negative result: the revision training recipe that works (offline construction with edit-distance pairing) is fragile, and a plausible alternative (on-policy data generation, which is standard in self-improvement literature) fails catastrophically.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The within-chain selection mechanism is described as the mitigation. The ReSTᵉᴹ ablation is in Appendix K, Figure 16. The paper hypothesizes that "the on-policy data collection in ReSTᵉᴹ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly," but does not investigate further.

Mitigation status. Partially addressed—the within-chain selection mechanism (majority voting or verifier) patches the reversion problem at inference time by not trusting the final revision step. However, this is a workaround, not a solution. A model that genuinely learned conditional revision—improve if wrong, preserve if correct—would be substantially more efficient (no need to evaluate all intermediate steps) and more robust. The ReSTᵉᴹ failure suggests that the current offline training recipe works for fragile reasons that are not fully understood, making it risky for practitioners to adapt the revision training approach to new models or domains without extensive validation.

Sequential Revisions Introduce Unavoidable Latency That Is Not Accounted for in the FLOPs-Based Efficiency Analysis

The assumption or constraint. The paper measures test-time compute exclusively in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency entirely. This matters because the compute-optimal policy selects strategy hyperparameters that mix sequential revisions (where each step depends on the previous one) with parallel sampling (where all generations can run simultaneously). The paper's own results (Figure 7, left) show that fully sequential revision chains are optimal at low-to-moderate budgets and that the optimal sequential-to-parallel ratio is always at least 1:1 or higher.

The consequence. A compute-optimal allocation of, say, 64 generations as 8 sequential × 8 parallel—or worse, 64 generations as 64 fully sequential steps—has fundamentally different wall-clock latency than 64 parallel samples:

8 parallel chains of length 8: requires 8 serial generation steps (each step generates the next revision for all 8 chains), meaning 8× the latency of pure parallel sampling at the same total generation budget.
Fully sequential (1 chain of length 64): requires 64× the latency of pure parallel sampling.

This means the strategies that the compute-optimal policy favors—particularly sequential-heavy allocations on easy-to-medium problems—may be unusable in latency-sensitive applications (interactive assistants, real-time decision-making, any user-facing deployment) even though they are FLOPs-efficient. A strategy that uses 64 sequential revisions might achieve the same accuracy as 256 parallel samples (a 4× FLOPs reduction), but if it takes 64 seconds versus 1 second of wall-clock time (assuming 1 second per generation), it is non-viable for interactive use regardless of FLOPs savings.

The paper also does not discuss whether the revision model's autoregressive generation cost per step is different from the base model's (due to longer context from including previous revisions), which could further affect latency.

What evidence exists in the paper. None. The word "latency" does not appear in the paper. All budgets and comparisons are in terms of generation counts, implicitly assuming that the cost of N generations is N times the cost of 1 generation regardless of whether they are serial or parallel. The compute-optimal policy (Figure 7) is selected purely based on accuracy at a given generation budget, with no latency penalty applied to sequential-heavy configurations.

Mitigation status. Not addressed at all. This is a fundamental tradeoff between FLOPs-efficiency and wall-clock efficiency that the paper does not acknowledge. In practice, practitioners would need to solve a multi-objective optimization problem: maximize accuracy subject to both a total FLOPs budget and a maximum latency constraint. Sequential-heavy strategies might be viable for offline batch processing (where latency doesn't matter) but not for online serving; parallel strategies might be required for interactive use even if they are less FLOPs-efficient. The paper's single-axis optimization (FLOPs only) provides useful theoretical insights but is incomplete for practical deployment decisions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around activation sparsity from a qualitative curiosity—"models happen to be sparse"—to a governed, predictable, and engineerable property of neural network training. Before this work, the field had accumulated disparate observations: activation sparsity exists (Li et al., 2022), it varies by dataset (Song et al., 2025), certain activation functions produce more of it (Zhang et al., 2024a). But these observations did not cohere into a framework. Practitioners designing efficient models had no principled way to answer basic questions: should I expect my model to become sparser with more data, or less? Will making it deeper help? If I train a smaller model first, can I predict what happens at scale?

The paper provides the first empirical scaling-law framework for an efficiency property rather than a loss or accuracy metric. The central finding—that activation ratio follows convergent power-law relationships with training data amount, and that the direction of this relationship is determined by the activation function—is not merely a curve fit. It establishes that sparsity evolves according to predictable dynamics with well-defined asymptotic limits, analogous to how Chinchilla scaling laws (Hoffmann et al., 2022) established that pretraining loss follows predictable power-law trajectories. The conceptual parallel is direct: just as Chinchilla gave practitioners a formula for allocating pretraining compute between model size and data, this paper gives practitioners a formula for predicting how sparsity will evolve given an activation function and a data budget. The fitted parameters in Table 2— $\alpha$ , $c$ , $A_0$ —are the sparsity analogs of Chinchilla's exponents.

What makes this genuinely novel rather than an incremental extension of scaling laws is that the paper discovers a sign inversion: ReLU and SiLU follow qualitatively different functional forms with opposite monotonicity. For ReLU, the activation ratio decreases with more data ( $A_{\text{ReLU}}(D) = \exp(-cD^\alpha + b) + A_0$ ), meaning more training increases sparsity. For SiLU, the activation ratio increases with more data ( $A_{\text{SiLU}}(D) = -c/D^\alpha + A_0$ ), meaning more training decreases sparsity. This is not a quantitative difference—it is a qualitative inversion of the relationship, and it has immediate practical consequences: organizations training SiLU-based LLaMA variants on ever-larger datasets are unknowingly making their models denser, not sparser, as training progresses. The paper does not just observe this; it provides a mechanistic interpretation: ReLU's hard-zero regime creates a ratchet effect where neurons can be permanently deactivated, while SiLU's smooth negative tail prevents any neuron from ever truly going to zero, and instead the model "fills in" its activation pattern as training proceeds.

The paper also resolves an apparent contradiction in the prior literature about the relationship between scale and sparsity. Li et al. (2022) reported that larger T5 models are sparser, which created an intuition that sparsity is a gift that comes automatically with scale. This paper demonstrates that finding was an artifact of comparing models at a fixed training budget rather than at their asymptotic limits. Because smaller models converge to their sparsity limit much faster (Figure 8), they appear less sparse mid-training even though their eventual limit is nearly identical (Figure 7: limit activation ratios vary by only 1.7 percentage points for ReLU, 2.7 for SiLU, across a 12× parameter range). This is a correction to a widely-held intuition with direct practical consequences: if you want a sparse model, do not assume that simply scaling up will help. Instead, control the width-depth ratio (Section 5.2), choose the right activation function (Section 5.1), and train long enough for the sparsity dynamics to converge (which happens faster for smaller models anyway).

Beyond the specific empirical laws, this paper introduces a new class of research instrument: the performance-calibrated sparsity metric. CETT-PPL-1% is not just a better ruler for measuring sparsity; it is a methodology that decouples the recognition of weakly-contributed neurons (via CETT's per-layer L2 norm relative error) from the calibration of acceptable degradation (via the PPL increase tolerance). This means different activation functions, architectures, and training recipes can be compared on a common, operationally meaningful scale—"how much computation can this model shed while keeping perplexity within 1% of its dense performance?" This is fundamentally different from prior metrics that either assumed a specific activation function (straightforward ReLU) or imposed an arbitrary fixed sparsity budget (Top-k, CATS). The paper's demonstration that CETT-PPL-1% preserves downstream performance almost exactly (Table 1: average −0.16 points on commonsense reasoning, −0.30 on reading comprehension) means the metric quantifies usable sparsity, not just observed sparsity.

The identification of width-depth ratio as a sparsity-control parameter adds a new dimension to Transformer architecture design. Prior work optimized width-depth ratio for training loss or stability; this paper shows that below a bottleneck point (~114 for 0.1B models), the limit activation ratio increases linearly with the width-depth ratio (Figure 5). Since many width-depth ratios achieve equivalent training loss (Figure 6: the loss curve is flat from ratio ~74 to ~282), sparsity provides a secondary optimization criterion that breaks this degeneracy. This reframes architecture design as a multi-objective problem where inference efficiency can be optimized without sacrificing training quality, by choosing the deepest architecture that remains stably trainable.

Finally, the paper provides a combinatorial model for neuron specialization dynamics (Section 5.3, Equation 6) that explains why sparsity limits are scale-insensitive even though convergence speeds differ. The model is simple—if neurons must be partitioned into functional groups with fixed proportions across scales, the number of possible assignments grows factorially with neuron count—but it generates falsifiable predictions and connects sparsity to the broader literature on modularity and specialization in neural networks (Zhang et al., 2023). This transforms sparsity from an isolated efficiency metric into a lens for studying how models organize computation internally. The observation that sparsity converges much more slowly than training loss (compare Figure 4 and Figure 13: loss plateaus while sparsity is still evolving) suggests that neuron specialization continues long after the model's output quality has stabilized—sparsity dynamics provide a window into an ongoing structural reorganization that loss curves are blind to.

Follow-Up Research This Work Enables

Cheap, sample-efficient difficulty estimation for sparsity measurement. The CETT-PPL-1% metric currently requires binary-searching the CETT hyperparameter over a validation dataset, which involves evaluating multiple candidate thresholds on multiple checkpoints (the last five, per Appendix E). While this is far cheaper than the 2048-sample-per-question oracle difficulty estimation in the reference paper's compute-optimal framework, it still imposes a non-trivial measurement cost: each binary search step requires a forward pass through the model with and without neuron skipping to compute the PPL ratio. A natural extension would train a lightweight predictor—possibly a small MLP or even a linear probe on top of intermediate layer representations—that directly estimates the activation ratio at CETT-PPL-1% from a single dense forward pass, without needing to actually skip neurons and measure PPL degradation. The training signal would be the CETT-PPL-1% sparsity measurements on a calibration set of checkpoints. This would enable real-time sparsity monitoring during training (plotting sparsity alongside loss on TensorBoard) and make the metric practical for large-scale training runs where repeated binary search is expensive.

Replication on non-MATH reasoning domains and alternative model families. All results in this paper use a single benchmark (MATH) with a single model family (PaLM 2-S*). A high-priority replication study would test whether the key findings—the difficulty-dependent optimal strategies, the 4× compute-optimal efficiency gain, the ceiling on hardest problems—generalize to: (1) code generation (HumanEval, MBPP), where unit tests provide clean correctness signals analogous to MATH's exact-match grading, but the reasoning patterns are different (algorithmic rather than mathematical); (2) logical reasoning (FOLIO, ProofWriter), where the reasoning structure is more explicitly step-by-step and may interact differently with process reward models; and (3) a non-PaLM model family (LLaMA-3, Gemma, or Qwen), to determine whether the PRM training recipe (Monte Carlo rollouts from the base model) transfers across architectures and tokenizers. The code generation setting is particularly promising because unit tests provide per-step verifiability naturally—a PRM trained on intermediate code states (e.g., after each function call) could be substantially more accurate than a PRM trained on natural language reasoning steps.

Combining PRM tree-search with revision models. The paper studies verifier-guided search and iterative revisions as independent mechanisms, but Section 8 explicitly notes they were never combined. The complementary difficulty-dependent patterns—revisions help most on easy problems (local refinement), search helps most on medium problems (global exploration)—strongly suggest that combining them would yield gains beyond either alone. A concrete experiment: use the revision model as the proposal distribution within beam search. At each node of the search tree, instead of sampling the next reasoning step from the base model, condition on the partial solution so far (including rejected branches) and use the revision model to generate the next candidate step. The revision model's training to produce improved answers given previous incorrect attempts should naturally produce higher-quality candidates when conditioned on rejected branches. Alternatively, use the PRM's per-step scores to guide which revision chain to extend: maintain multiple parallel revision chains, score each partial revision with the PRM, and allocate more budget to chains that the PRM rates as promising. This hybrid approach could break through the individual performance ceilings: the revision model addresses the "hard to find correct solutions by random sampling" problem (which limits search on hard problems), while PRM-guided selection addresses the "revision model sometimes breaks correct answers" problem (the 38% reversion rate). A strong follow-up would measure whether the combined approach achieves >44% accuracy at 256 generations (vs. the current best of ~44% for compute-optimal revisions alone, Figure 8), and particularly whether it improves bin 4–5 performance where neither method individually helps.

Verifier robustness training to mitigate over-optimization. The over-optimization phenomenon documented in Figure 3 (right) and Appendix M—beam search degrading easy-problem performance with increasing budget, lookahead search paradoxically underperforming simpler methods—is identified as a central bottleneck but not addressed. A direct follow-up would train a PRM specifically for robustness under aggressive search: instead of training the PRM on i.i.d. samples from the base model (the current Monte Carlo rollout approach), train it on solutions generated by beam search against a preliminary PRM. This is adversarial training in verifier space—the PRM learns to correctly score the kinds of degenerate solutions that search tends to find (overly short solutions, repetitive low-information steps, solutions that exploit superficial patterns matching high-PRM-score templates). A concrete evaluation would compare the accuracy-sparsity Pareto frontier of the adversarially-trained PRM vs. the standard PRM at high search budgets (256–512 generations), specifically measuring whether beam search accuracy on difficulty bin 1 continues to decrease with budget (as in Figure 3, right) or whether it now monotonically improves. A complementary approach would add a KL-divergence penalty to the PRM's scoring function, penalizing solutions whose token-level probabilities under the base model are low—this would prevent search from drifting into low-probability regions of the output space where the PRM's scores are poorly calibrated.

Training-time difficulty prediction for amortized cost reduction. The paper's compute-optimal framework requires estimating each prompt's difficulty before allocating the test-time budget. The current method (2048 samples + PRM scoring per question) makes the approach impractical for anything but high-volume offline batch processing. A critical follow-up would train a difficulty predictor that maps directly from the question text (or its embedding) to a difficulty bin, without generating any samples. The training data exists: the paper already has oracle difficulty bins for 500 MATH questions, and could extend this to thousands more by running the 2048-sample estimation procedure on the 12,000 training questions. A small classifier (e.g., a linear probe on top of the base model's final hidden state, or a lightweight distilBERT-style model) trained to predict the difficulty quintile from the question representation could reduce the estimation cost from 2048 generations to a single forward pass. The key evaluation would be: does the compute-optimal policy selected using predicted difficulty from this cheap classifier achieve the same accuracy as the policy selected using the full 2048-sample estimation? Figure 4 already shows that PRM-based difficulty prediction (which still requires 2048 samples) tracks oracle difficulty closely; the open question is whether the difficulty signal is robust enough to survive a 2048× compression. If a lightweight predictor achieves even 80% of the efficiency gain of the full estimation procedure, the compute-optimal approach becomes immediately practical for online deployment.

Sparsity-aware architecture search for width-depth ratio optimization. Section 5.2 establishes that the limit activation ratio increases linearly with width-depth ratio below a bottleneck (~114 for 0.1B), while training loss is flat across a wide range (74–282). This suggests a formal architecture search problem: for a given FLOPs budget at inference, what (parameter count, width-depth ratio, activation function) triple maximizes downstream performance? The search space is tractable: the paper already provides fitted sparsity laws for ReLU and SiLU across five scales, and the width-depth ratio experiments on 0.1B models could be extended to larger scales to determine whether the bottleneck point shifts with parameter count (plausibly scaling as a power law itself). A concrete experiment: train 0.4B models at 5–6 width-depth ratios spanning the stable range, fit limit activation ratios vs. ratio for that scale, and test whether the bottleneck point is a function of parameter count (e.g., bottleneck ≈ $k \cdot N_{\text{params}}^\beta$ for some $k$ , $\beta$ ). Combined with the FLOPs-matched pretraining vs. inference tradeoff from Section 7 (adapted to this sparsity setting: compare a dense SiLU model vs. a sparse ReLU model at equal inference FLOPs), this would yield a holistic framework for co-designing model architecture and inference strategy.

Dynamic difficulty-adaptive budget allocation within a single inference call. The current compute-optimal policy is static: difficulty is estimated once, a strategy is selected, and the full budget is spent under that strategy. A more sophisticated approach would be dynamic: start with a small number of parallel samples (e.g., 4–8), use the distribution of PRM scores on those samples as a real-time difficulty signal, and then decide adaptively whether to continue with more parallel sampling, switch to beam search, or spawn revision chains. This is the exploration-exploitation tradeoff the paper flags in Section 3.2: "compute spent assessing difficulty versus compute spent solving the problem." The existing difficulty estimation experiments (Figure 4, predicted vs. oracle bins) already show that the PRM's final-answer score distribution carries a strong difficulty signal; the open question is how many samples are needed for a reliable estimate. A concrete experiment would sweep the number of initial samples (4, 8, 16, 32, 64) and measure the accuracy of the resulting difficulty bin classification against the full 2048-sample oracle. If 16 samples achieve >90% bin classification accuracy, a dynamic policy could use those 16 samples for both difficulty estimation and as the initial candidates for verifier-guided selection, amortizing the cost entirely.

Practical Applications and Downstream Use Cases

Cost-efficient batch inference for math and code evaluation. For organizations running large-scale batch evaluation—e.g., evaluating thousands of candidate math solutions in an automated curriculum, or scoring student answers in an educational platform—the compute-optimal framework offers a direct recipe for reducing API costs or GPU hours. Rather than requesting best-of-256 for every problem (which dominates cost at scale), the pipeline would: (1) estimate difficulty using a lightweight classifier or a small number of initial samples (cheap), (2) allocate test-time compute per problem according to the pre-computed optimal policy for that difficulty bin (e.g., 4-generation sequential revision for easy, 64-generation beam search for medium, and flagging hard problems for human review or a larger model). With the paper's demonstrated 4× efficiency gain at moderate budgets (Figure 4: 16-generation compute-optimal matching 64-generation best-of-N), an organization processing 100,000 math problems per month could reduce their inference compute costs by roughly 75% on the fraction of problems where the compute-optimal strategy applies, at equivalent accuracy. This is particularly relevant for synthetic data generation pipelines (e.g., self-improvement loops) where the goal is to generate the highest-quality solutions for a fixed compute budget, not to minimize latency per query.

On-device deployment with small, sparse models for routine queries. The paper's finding that a smaller model with compute-optimal test-time strategies can match or exceed a ~14× larger model on easy-to-medium difficulty problems (Section 7, FLOPs-matched comparison) has direct implications for on-device deployment. A 0.8B model with compute-optimal inference can achieve the same accuracy on routine user queries as a ~11B model with greedy decoding—but runs locally on a phone or laptop, eliminating server round-trip latency and privacy concerns. The practical architecture would be: deploy a small, sparsity-optimized model on-device (following the paper's prescriptions: ReLU activation, small width-depth ratio, trained to convergence on abundant data), equip it with a lightweight difficulty estimator and a small set of pre-computed compute-optimal strategies, and route only genuinely hard questions to a cloud-based larger model. The difficulty estimator serves double duty: it determines how much local test-time compute to allocate and whether to escalate to the cloud. For applications like on-device code assistants, math tutoring, or document Q&A where the typical query distribution skews toward tasks the small model can handle with extra inference effort, this architecture could provide cloud-quality responses with local latency and zero server costs. The paper's caveat that hard problems (bin 5) do not benefit from test-time compute is actually a feature here—it provides a clear criterion for when to escalate.

Verifier development as a strategic research investment. The paper's finding that verifier over-optimization is the primary bottleneck for test-time compute scaling (Section 5.3: beam search degrades on easy problems, lookahead search underperforms) has direct implications for how AI research teams should prioritize their efforts. Rather than investing in more sophisticated search algorithms (which the paper shows can be counterproductive—lookahead search performs worse than simple beam search at equal budget), teams should invest in verifier robustness and calibration. The Monte Carlo rollout training procedure (Section 5.1) provides a concrete, human-label-free recipe that practitioners can adopt immediately, but the paper also identifies clear failure modes: the PRM is vulnerable to solutions that score highly but are incorrect, particularly repetitive or overly short generations. A practical investment would be to build a verifier training pipeline that includes: (1) adversarial data generation (solutions found by beam search against a preliminary PRM, as in the follow-up research direction above), (2) ensemble verification (aggregating predictions from multiple PRMs trained with different random seeds or on different data splits), and (3) calibration tuning (temperature scaling or Platt scaling on held-out calibration data to ensure that the PRM's confidence scores correspond to actual correctness probabilities). The paper's finding that the PRM trained with soft Monte Carlo labels outperforms binary-label PRMs (Appendix E) hints that label quality matters substantially, suggesting that investments in better rollout procedures (more rollouts per step, rollouts with higher-quality completions) could improve verifier quality more cost-effectively than investments in search algorithm complexity.

When to Prefer This Method

This paper does not position itself against named alternative sparsity measurement or training methods in a way that yields a clean tradeoff matrix. The CETT-PPL-1% metric is compared against prior metrics (straightforward ReLU, Top-k, FAT-ε) in Section 4 and is shown to dominate them in the performance-sparsity Pareto sense—there is no condition under which the older metrics are preferable. The prescription for building sparse LLMs (ReLU activation, small width-depth ratio, abundant data) is not positioned against a specific alternative training recipe but rather derived as the synthesis of the paper's empirical findings. MoE is briefly compared (Appendix B, Figure 12) and shown to have a worse performance-sparsity tradeoff than intrinsic activation sparsity, but the paper is explicitly studying vanilla Transformers, not positioning against MoE as a competing paradigm. For practical deployment, the choice is not "whether to use CETT-PPL-1% or an alternative metric" (CETT-PPL-1% is unambiguously better for the use cases studied) but rather whether the specific architectural and training prescriptions (ReLU instead of SiLU, deeper rather than wider, more data) are compatible with existing training infrastructure and downstream requirements—a decision that depends on factors (availability of ReLU-optimized inference kernels, training stability constraints at very large scales) that the paper does not systematically evaluate.