Scaling Laws for Sparsely-Connected Foundation Models

🎯 Pitch

Sparse models can outperform dense ones with the same number of non-zero parameters—but only when trained far beyond the standard compute-optimal (Chinchilla) regime, where dense scaling saturates. The authors derive a joint scaling law showing that optimal sparsity rises with training duration, and a 75% sparse model acts like a 2.16× larger dense model in the infinite-data limit. This reveals sparsity's surprising benefit as a late-training accelerator, not a universal win.

1. Executive Summary

This paper studies how weight sparsity affects the scaling behavior of Transformers trained on massive datasets—Vision Transformers on JFT-4B and T5 models on C4—by developing the first joint scaling law for sparsely-connected foundation models that relates sparsity S, number of non-zero parameters N, and training data D to validation loss. The law treats sparsity as a multiplicative capacity modifier via a saturating power-law term $(a_S(1-S)^{b_S} + c_S)$ that multiplies the standard size-scaling component, while remaining independent of data scaling (i.e., the original $D$ term is preserved). From this law the authors derive an analytic expression for optimal sparsity $(S_{opt})$ , the sparsity level yielding the best loss for a given inference size and training budget—finding that $S_{opt}$ increases with longer training, with 50% sparsity becoming optimal at < 2× Chinchilla training duration for vision and < 3× for language under sparse-FLOPs accounting, and that in the infinite-data limit a 75% sparse model provides an effective capacity gain equivalent to a ~2.16× larger dense model. The key boundary condition emerges from the extrapolation: sparse models can outperform dense ones of equivalent non-zero parameter count, but only when training substantially beyond the dense compute-optimal (Chinchilla) point, establishing that sparsity's benefits manifest primarily in the overtraining regime where dense capacity scaling begins to saturate.

2. Context and Motivation

The Core Gap: We Don't Know How Sparsity Scales

The paper addresses a fundamental question at the intersection of two major trends in deep learning: scaling laws for foundation models and model compression via weight sparsity. While both areas have seen extensive research independently, their intersection—understanding how sparsity affects the scaling behavior of models trained on massive datasets—remains almost entirely unexplored. This gap matters because it leaves practitioners unable to answer basic practical questions: if I have a fixed inference budget (number of non-zero parameters), should I train a dense model or a larger sparse model? At what sparsity level does the capacity gain from additional zeroed parameters exceed the accuracy loss from the pruning process? And critically, does the answer depend on how long I train?

The paper frames this as a predictability problem. Across the scaling laws literature, one of the central achievements has been the ability to forecast performance from high-level model attributes (parameter count, data volume, compute budget) without running the actual experiment. This predictive capability enables rational allocation of training resources—the Chinchilla compute-optimal line tells you exactly how to trade off model size against training tokens for a given FLOPs budget. But before this paper, no analogous predictive tool existed for sparse models. You could not plug sparsity into a scaling formula and determine whether it would help or hurt, let alone estimate the optimal sparsity for a given training regime.

Why This Gap Matters: The Collision of Two Trends

The practical significance of closing this gap stems from several converging forces in modern ML deployment:

Training budgets keep growing, and overtraining is increasingly common. The Chinchilla optimal point—where a model is sized to match the available training data—represents a specific point on the compute-efficiency frontier. But in practice, many important models are trained far beyond Chinchilla. The authors note several examples: LLaMA 2 7B is trained with approximately 14× Chinchilla data, and the largest language models are routinely trained with 100×+ more data than would be compute-optimal for a dense model (Brown et al., 2020; Touvron et al., 2023b). This overtraining regime is precisely where dense models begin to saturate—the returns from additional data diminish, because the model's capacity (its ability to absorb more information) becomes the bottleneck rather than the amount of available data. If sparsity increases effective capacity without increasing the non-zero parameter count (and thus inference cost), then sparse models could be especially valuable in this regime. But until this paper, there was no systematic framework to test that hypothesis.

Inference efficiency directly impacts deployment costs. The non-zero parameter count of a model determines its inference FLOPs (and, in practice, its memory footprint and latency). If a 50% sparse model with N non-zero parameters can match the accuracy of a dense model with 1.5N parameters (which it does, approximately, based on Table 2), then sparse deployment yields a 33% reduction in inference cost for the same quality. At the scale of modern foundation model APIs serving billions of queries, this translates to enormous absolute savings. But this tradeoff only makes sense if the sparsity gain is predictable and reliable—you need to know whether you'll get 1.5×, 1.6×, or 2.17× gain for a given sparsity level, and whether that gain holds at the model sizes and training budgets you actually use.

Pruning benchmarks have used misleading evaluation protocols. This is perhaps the most subtle but important motivation. The paper argues in Section 2 ("Fair Evaluation in the Presence of Strong Scaling") that standard pruning evaluation practices—largely inherited from the ResNet50/ImageNet era—are fundamentally inappropriate for foundation models trained on massive datasets. The core issue is that Transformers on JFT-4B or C4 exhibit very different scaling behavior than ResNets on ImageNet. In the classic setting:

Overfitting is common. On ImageNet, extending training of a dense ResNet50 quickly leads to overfitting, so there's a natural ceiling. A sparse model that trains for longer (which most pruning procedures effectively do, since they start from a pretrained checkpoint and then continue training during pruning) doesn't face a meaningful dense baseline that also trained for that longer duration.
Small dense models aren't competitive. Naively scaling down a ResNet50 by a factor of 10 doesn't produce a good small model—you need carefully engineered architectures like MobileNet or EfficientNet. This means pruning papers can safely compare a 50% sparse ResNet50 against the original ResNet50 without worrying about whether a comparably-sized dense model would have performed better.

But in the foundation model setting, all three of these assumptions break down:

No overfitting. Zhai et al. (2022) showed that ViT performance on JFT only begins to saturate after extremely long training. The authors state that overfitting is "virtually non-existent." This means that if you prune a ViT that was pretrained on 100M images and then continue training during pruning for another 100M images, the fair comparison is against a dense model trained on 200M images—not the original 100M-image checkpoint. And that dense-200M model will be significantly better.
Simple scaling works. Unlike ResNets, basic width and depth scaling of Transformer architectures produces strong competitive models at any size. A dense model with the same number of non-zero parameters as a 50% sparse model is a legitimate—and strong—baseline that must be included.
Training compute matters. A smaller dense model trained for longer can outperform a larger one trained for fewer steps with the same total FLOPs. If sparsification consumes additional training compute (which it does, both from training a larger initial model and from the pruning process itself), then the fair baseline must equalize total compute, not just parameter count.

This creates a much harder evaluation setting than what most prior sparsity work has faced. The authors explicitly acknowledge: "it is currently unclear whether weight-sparse foundation models can win at all in this highly challenging setting, where all these factors are properly accounted for." The paper's framework—fitting a joint scaling law and then deriving optimal sparsity analytically—is designed expressly to answer this question without running every possible comparison experiment.

Where Prior Work Falls Short

The paper identifies several categories of prior work that are relevant but insufficient:

Pruning literature on small-scale benchmarks. The vast majority of sparsity research (Hoefler et al., 2021 survey) operates on relatively simple tasks—ResNet50/ImageNet or BERT/GLUE—where models are substantially overparameterized and training data is limited enough that overfitting provides a natural ceiling. State-of-the-art methods range from gradual magnitude pruning (Zhu & Gupta, 2017) to Hessian-based metrics (Singh & Alistarh, 2020; Frantar et al., 2021) to sparse training from scratch (Mocanu et al., 2018; Evci et al., 2020). These methods can often achieve very high sparsity levels with minimal accuracy loss on these benchmarks. However, the paper argues that the conclusions from these settings do not transfer to the foundation model regime because:

The scaling behavior is different (dense models keep improving with more data, so there's no convenient ceiling)
The baselines are different (simply scaled-down dense models are competitive)
The compute dynamics are different (overtraining is the norm, not the exception)

Rosenfeld et al. (2021) is the closest prior work in spirit, studying the relationship between width, depth, and weight density for pruned ResNets. However, their experiments use the comparatively tiny CIFAR-10 dataset, where the data/compute dimension—which the present paper identifies as the crucial variable determining optimal sparsity—is essentially irrelevant because models quickly saturate or overtrain. The present work explicitly extends the investigation to the massive-dataset regime where data scaling remains active throughout training.

Limited prior work on large-model sparsity. The paper cites two efforts that did prune large Transformers, but both use evaluation protocols that the authors argue are flawed:

The Gopher appendix (Rae et al., 2021) prunes decoder-only language models and finds that sparse models can outperform dense ones when trained for the same number of steps. But the authors point out this is an unfair comparison: the sparse model consumed significantly more total FLOPs (because it was trained as a larger dense model that was then pruned, and training continued during pruning). The paper does not investigate whether this advantage persists when dense models are given equivalent compute budgets.
Cerebras (2022) prunes GPT-like models and reports approximately 5× gains, but uses more than 5× training data relative to the dense baseline. The authors explicitly analyze this result through their scaling law (Section 6) and find it is actually consistent with their framework: if you query the T5 scaling law with the Cerebras setup (larger sparse model + much more data vs. smaller dense model + less data), the law predicts exactly the sparse advantage that was observed. The issue is not that the result is wrong, but that it conflates sparsity gains with the benefits of additional training, making it impossible to attribute the improvement to sparsity per se rather than simply training longer on more data.
SparseGPT (Frantar & Alistarh, 2023) achieved non-trivial sparsity on very large models without retraining, but the paper notes it's unclear whether this works on "more recent, smaller and much less undertrained networks"—the highly overparameterized nature of the GPT-scale models used may be a special case.

Scaling laws for dense models are well-established but don't cover sparsity. Kaplan et al. (2020) established the basic power-law relationship $L(N, D) = (a_N/N)^{b_N} + (a_D/D)^{b_D} + c$ for dense Transformers. Hoffmann et al. (2022) extended this to derive compute-optimal training recipes (the Chinchilla line). More recently, various specialized scaling laws have been developed—for model shape optimization (Alabdulmohsin et al., 2023), routing mechanisms in mixture-of-experts models (Clark et al., 2022), data repetition (Muennighoff et al., 2023), and downstream task transfer (Caballero et al., 2023). But none of these incorporate weight sparsity. The paper's contribution is to add sparsity as a new dimension to this scaling law framework, with the key finding that sparsity affects the capacity (size) scaling term multiplicatively while leaving the data scaling term unchanged.

Efficiency techniques exist but without scaling guidance. Quantization (Frantar et al., 2022; Dettmers & Zettlemoyer, 2022), mixture-of-experts (Fedus et al., 2022; Riquelme et al., 2021), and weight sparsity are all complementary techniques for reducing inference costs. But for any given deployment—with a specific model family, dataset, training budget, and target inference cost—there's no principled way to decide which technique (or combination of techniques, at what intensity) will yield the best performance. This paper doesn't solve that full optimization problem, but it provides the first analytical tool for evaluating the sparsity dimension, which is a necessary component.

How This Paper Positions Itself

The paper positions itself as a first-principles empirical investigation rather than a proposal for a new pruning method. This is an important distinction: the goal is not to achieve state-of-the-art pruning results on a particular benchmark, but to understand the fundamental relationship between sparsity, model size, and training data in a way that is predictive and generalizable. This mirrors the philosophy of the original Kaplan et al. (2020) scaling laws paper—the contribution is the analytical framework and the empirical regularities it reveals, not a specific model or training recipe.

Several deliberate choices reflect this positioning:

The pruning method is deliberately simple and robust. The authors use gradual magnitude pruning (GMP) with a cubic schedule, explicitly stating that this choice was made for "robustness and scalability across a wide range of setups, rather than to fully maximize performance in a particular one." They tested alternatives (AC/DC, STE, RigL) and found them to be similar or worse while being more sensitive to hyperparameters at scale. The implication is that the scaling law's qualitative form should be robust to improvements in pruning methodology—only the coefficient values might shift, not the functional relationship between S, N, and D.

The focus is on validation loss, not downstream accuracy. The paper deliberately uses pretraining validation loss as the target metric, because it's the cleanest available signal that directly reflects model capacity and training quality without the confounding factors of task-specific fine-tuning. As they note in the discussion, "sparsity rates can probably be improved significantly when pruning is performed directly for more specialized applications." By working with pretraining loss, they establish a lower bound on what's achievable—downstream performance should be at least as favorable for sparsity.

The evaluation protocol is designed to be maximally unfavorable to sparsity. By insisting on equal non-zero parameter comparisons, equal total data, and equal total compute, the paper constructs a setting where sparsity has to justify itself through pure capacity efficiency—there's no free lunch from training longer or using more parameters. This is intellectually honest and provides conservative estimates of sparsity's value, but it also means the gains reported in the paper (e.g., 2.17× effective capacity gain at 75% sparsity) should be interpreted as worst-case relative to what might be achievable in less constrained settings.

The scope is deliberately broad: vision and language, different architectures, different datasets. By fitting the same functional form (Equation 3) to both ViT/JFT-4B and T5/C4—two very different domains with different architectures, objectives (multi-label classification vs. masked language modeling), and dataset characteristics—the paper builds a case for the universality of the sparsity scaling relationship. The fitted coefficients differ (as they should, since the tasks differ), but the qualitative structure—a multiplicative sparsity term on the capacity component, independent of the data component—holds across domains. This cross-domain validation is methodologically important because it suggests the law is capturing something fundamental about how sparsity interacts with neural network capacity, not just an artifact of a particular experimental setup.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical analysis paper that builds a predictive model (a scaling law) for how weight sparsity affects Transformer performance when training on massive datasets, rather than proposing a new pruning method. The core idea is that sparsity acts as a multiplicative capacity modifier: removing weights via pruning changes the effective model capacity in a predictable way that can be captured by a saturating power-law term, and this effect is independent of how much data the model trains on, meaning the relationship between sparsity, non-zero parameters, and training data can be expressed as a single joint equation that enables optimal sparsity calculations.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that interact sequentially:

Training Pipeline — A standard Transformer training setup (ViT on JFT-4B for vision, T5 on C4 for language) augmented with gradual magnitude pruning (GMP) applied during training. This produces models at specified sparsity levels, non-zero parameter counts, and training durations.
Sweep Grid — A systematic grid of experimental configurations spanning model sizes (7 sizes for ViT, 4 for T5), sparsity levels (0%, 50%, 75%, 87.5%), and training steps (4 durations for ViT, 3 for T5). Executing every combination generates the raw data (validation loss measurements) that the scaling law is fitted to.
Scaling Law Fitting Procedure — A parametric equation $L(S, N, D)$ with 7 free parameters is fitted to all sweep data simultaneously using Huber loss minimization with BFGS optimization. The fitted law predicts validation loss from sparsity, non-zero parameters, and training data, enabling both interpolation (predicting performance of unseen configurations within the sweep range) and extrapolation (predicting performance of larger models).
Optimal Sparsity Derivation — Using the fitted scaling law, the authors derive an analytic expression for $S_{opt}(N, C)$ , the sparsity level that minimizes validation loss for a fixed inference size and training compute budget. This derivation requires defining a cost model for sparse training (Equation 4) and then differentiating the loss with respect to sparsity to find the optimal contour (Equation 5).

Information flows as: sweep configurations → training runs → validation loss measurements → joint fit of all coefficients → analytical derivation of optimal sparsity → iso-contour plots showing which sparsity is best for any (N, C) point.

3.3 Roadmap for the Deep Dive

First, the experimental setup and sweep grid design, since the scaling law is purely empirical—understanding what data it's fitted to and why the grid was designed this way is essential for interpreting the law's claims and limitations.
Second, the dense scaling law background (Equation 2), because the sparse law is built as an extension of this well-established form, and understanding the base case makes the sparse modifications interpretable.
Third, the core sparse scaling law (Equation 3), its derivation from preliminary observations, the meaning of each term, and the fitting procedure—this is the central contribution of the paper.
Fourth, the optimal sparsity derivation (Equations 4-6), including the sparse cost model $c_{mul}(S)$ and the differentiation that yields the optimal contour—this shows how the law transitions from descriptive to prescriptive.
Fifth, the limit performance analysis (Equation 7), which answers the practical question: "how much larger does a dense model need to be to match a sparse one?"

3.4 Detailed, Sentence-Based Technical Breakdown

Experimental Setup and Sweep Grid Design

The paper constructs a systematic experimental grid across two domains (vision and language), varying three factors independently: model size (via non-zero parameter count), sparsity level, and training duration. The goal is to collect validation loss measurements at enough points in this 3D space that a parametric scaling law can be reliably fitted.

Vision setup: ViT on JFT-4B. Vision Transformers following the standard ViT architecture (Dosovitskiy et al., 2021) are trained for multi-label image classification on JFT-4B (Dehghani et al., 2023), a dataset of 4 billion images. The training recipe follows Zhai et al. (2022). Model sizes are scaled by varying only the Transformer's hidden dimension while keeping all other architectural parameters (number of layers, attention heads relative to width, MLP expansion ratio) constant. This produces 7 target non-zero parameter counts in 2× increments: 0.66M, 1.33M, 2.65M, 5.31M, 10.62M, 21.23M, and 42.47M. Each size is trained for 4 different durations: 55K, 110K, 220K, and 440K steps, with the longest corresponding to approximately 1.8 billion images seen. At each (size, duration) combination, four sparsity levels are applied: 0% (dense baseline), 50%, 75%, and 87.5%—corresponding to 1×, 2×, 4×, and 8× compression rates respectively, where compression rate is defined as $1/(1-S)$ . The total ViT grid contains $7 \times 4 \times 4 = 112$ runs.

Language setup: T5 on C4. Encoder-decoder T5 models (version 1.1, Google, 2023b) are trained for masked language modeling on C4 (Raffel et al., 2020a), a dataset of 150+ billion tokens. Training follows the original T5 recipe (Raffel et al., 2020b) with the modification that the AdaFactor optimizer's second moment is not factorized—a choice made because "factorized moments for pruning and sparse optimization are not yet very well studied." Additionally, batch size is set to 128 (rather than the default) to ensure sufficient optimization steps even at lower total training data volumes. Model sizes again scale by varying only the hidden dimension, producing 4 target non-zero parameter counts in 4× increments: 1.3M, 5.3M, 21M, and 85M. Each is trained for 3 durations: 250K, 500K, and 1M steps, with the longest corresponding to approximately 65 billion tokens (given batch size 128, $10^6 \times 128 = 128 \times 10^6$ tokens per step, so 1M steps ≈ 128B tokens, though the exact mapping depends on sequence length). The same four sparsity levels are used. Total T5 grid: $4 \times 3 \times 4 = 48$ runs.

Why this grid design? Several deliberate choices shape the grid:

Powers of 2 for size increments ensures even spacing on a log scale, which is appropriate for power-law fitting where relationships are linear in log-log space.
Sparsity levels up to 87.5% are chosen because preliminary experiments indicated "quickly diminishing returns for very sparse models" that are "in addition quite expensive to train." Higher sparsities (e.g., 93.75% or 96.875%) would require training models 16× or 32× larger, which becomes computationally prohibitive for a grid sweep.
The ViT grid is denser than T5 (7 vs. 4 sizes, 4 vs. 3 durations) by design—the ViT setup serves as the primary, more extensive sweep while T5 validates cross-domain generalization with a more efficient design.

Pruning procedure: gradual magnitude pruning (GMP). The sparsification method is deliberately simple and robust to avoid hyperparameter sensitivity across the diverse grid points. GMP (Zhu & Gupta, 2017) works by periodically removing the weights with the smallest absolute magnitudes according to a predetermined schedule. The specific configuration:

Schedule: cubic, $S \cdot (1 - (1-t)^3)$ where $t \in [0,1]$ is the fraction of the pruning interval elapsed.
Start point: 25% of total training steps completed (so the model has a reasonable initialization before pruning begins).
End point: 75% of total training steps completed (leaving the final 25% of training for fine-tuning the sparse structure).
Update frequency: every 100 steps.

All linear layers in the Transformer backbone are sparsified uniformly. For T5, the embedding layers are also sparsified "to the amount necessary for parameter matching a smaller dense version." This uniform approach avoids "layer collapse" (Wang et al., 2020), where some layers get disproportionately pruned and become performance bottlenecks.

Why GMP over alternatives? The authors tested AC/DC (Peste et al., 2021), STE (Lin et al., 2020), and RigL (Evci et al., 2020)—methods that achieve strong results on classic benchmarks. They found these performed "similar or worse" while being "more sensitive to hyper-parameters as we scale." A second-order diagonal variant of GMP (Kurtic et al., 2022) performed "almost identically." The key requirement for a scaling law sweep is consistency and reliability across diverse configurations without per-run tuning—GMP satisfies this at the cost of potentially sub-maximal absolute performance, but since the goal is understanding relationships rather than state-of-the-art numbers, this tradeoff is acceptable.

A crucial implementation detail: sparsity-aware RMS. T5's AdaFactor optimizer scales learning rates by the root-mean-square (RMS) of each weight tensor. For sparse tensors, this creates a problem: "a tensor with many zeros tends to have a lower RMS, resulting in a smaller learning rate." This is counterproductive because higher sparsity requires more recovery during pruning, not less. The solution: always compute the RMS only over unpruned (non-zero) weights. The same correction is applied to AdaFactor's RMS clipping threshold, though the authors note this is "much less critical."

Data repetition policy. The paper assumes an "ideal training setup with no data repetition" and focuses on "the non-bottlenecked regime." For ViT on JFT-4B (4 billion images), the maximum training duration of 440K steps at typical batch sizes samples approximately 1.8 billion images, well within a single epoch. For T5 on C4 (150+ billion tokens), 1M steps at batch size 128 with typical sequence lengths (512 tokens) corresponds to roughly $10^6 \times 128 \times 512 \approx 65 \times 10^9$ tokens, again within a single pass. This matters because data repetition changes scaling behavior (Muennighoff et al., 2023), and the paper's law is fitted in the single-epoch regime where scaling is cleanest.

Validation loss recording. For every run in the sweep, the final validation loss is recorded. For ViT, this is multi-label classification loss on JFT-4B's validation split. For T5, this is masked language modeling loss (cross-entropy) on C4's validation split. These losses serve as the target variable $L$ in the scaling law.

Dense Scaling Law Background

Before introducing sparsity, the paper recapitulates the standard dense scaling law that serves as the foundation. This is well-established from Kaplan et al. (2020) and Hoffmann et al. (2022).

The validation loss of a dense Transformer is modeled as:

$L(N, D) = \left(\frac{a_N}{N}\right)^{b_N} + \left(\frac{a_D}{D}\right)^{b_D} + c$

where $N$ is the number of model parameters, $D$ is the amount of training data, $a_N$ and $a_D$ are constant coefficients, $b_N$ and $b_D$ are scaling exponents, and $c$ is the irreducible loss (a lower bound representing inherent stochasticity of the task).

What it computes: the predicted validation loss given model size and training data. The first term $(a_N/N)^{b_N}$ captures the capacity-limited regime—loss decreases as a power law in parameter count, asymptoting toward zero as $N \to \infty$ (ignoring $c$ ). The second term $(a_D/D)^{b_D}$ captures the data-limited regime—loss decreases as a power law in training data, asymptoting toward zero as $D \to \infty$ . The constant $c$ represents the Bayes error or inherent entropy floor that even infinite parameters and infinite data cannot overcome. The total loss is the sum of these three components, reflecting the additive nature of capacity and data bottlenecks: when both $N$ and $D$ are large, loss approaches $c$ ; when either is small, the corresponding term dominates.

Why this form: Power laws are the empirically observed relationship between scale and loss across many orders of magnitude for Transformer models (Kaplan et al., 2020). The additive decomposition into separate capacity and data terms is motivated by the observation that these represent distinct bottlenecks—you can be limited by not having enough parameters to represent the function, OR by not having enough data to learn it, OR both. The additivity (rather than, say, a product or minimum) reflects that these limits combine: even with infinite data, a too-small model can't achieve loss below its capacity limit, and vice versa. The exponents $b_N$ and $b_D$ are typically stable for a given task (e.g., language modeling vs. image classification), while the coefficients $a_N$ and $a_D$ shift with architectural or optimization changes.

Key property for this paper: This form separates the size and data dependencies. As we will see, the sparse extension preserves this separation—sparsity modifies only the size term (acting through capacity) while leaving the data term unchanged.

Preliminary Observations that Motivate the Sparse Law

Before proposing the full sparse scaling law, the authors execute the T5 sweep (Table 1) and visualize the results to identify qualitative patterns. Figure 2 shows validation loss (with the lower bound $c$ subtracted) versus non-zero parameter count on log-log axes, with separate curves for each sparsity level, grouped by training duration (250K, 500K, 1M steps).

Three key observations emerge from these plots:

Observation 1: Parallel lines. The loss-vs-size curves for different sparsity levels form "almost parallel lines, differing primarily in the intercept." On a log-log plot, parallel lines imply the same slope (exponent) but different multiplicative constants. This means the scaling exponent $b_N$ is approximately the same across sparsity levels—sparsity does not change how quickly loss decreases with more non-zero parameters, only the starting point (lower is better). In operational terms: if you go from 1M to 10M non-zero parameters, you get the same factor of loss reduction regardless of whether the model is dense or 75% sparse, but the 75% sparse model achieves lower absolute loss at every size.

Observation 2: Diminishing sparsity returns. "The higher the sparsity, the lower the loss, but gains are quickly diminishing." Going from 0% to 50% sparsity provides a substantial loss reduction. Going from 50% to 75% provides a smaller additional gain. Going from 75% to 87.5% provides an even smaller gain—the curves for 75% and 87.5% are nearly overlapping. This saturating behavior suggests a power-law relationship between sparsity (or its complement, density) and the capacity benefit: the marginal value of additional sparsity decreases.

Observation 3: Data independence. "The overall shape of all curves is very similar for each training duration, the y-axis just tends to shift a bit downwards with more training steps." The relationship between sparsity and loss does not appear to interact with training duration—more training lowers all curves roughly uniformly. This implies that sparsity and data scaling are mostly independent factors.

These three observations are the inductive basis for the functional form of the sparse scaling law.

The Core Sparse Scaling Law: Derivation and Fitting

Based on the preliminary observations, the authors construct a joint scaling law $L(S, N, D)$ that extends the dense form:

$L(S, N, D) = \left(a_S(1 - S)^{b_S} + c_S\right) \cdot \left(\frac{1}{N}\right)^{b_N} + \left(\frac{a_D}{D}\right)^{b_D} + c$

where $S \in [0, 1)$ is the sparsity fraction (0 = dense, 0.5 = half the weights are zero, 0.875 = 7/8 of weights are zero), $N$ is the number of non-zero parameters, $D$ is the amount of training data, $b_N$ and $b_D$ are the size and data scaling exponents (shared with the dense case), $a_S$ , $b_S$ , and $c_S$ are new coefficients governing the sparsity effect, $a_D$ is the data coefficient, and $c$ is the irreducible loss lower bound.

What each term means, operationally:

$(a_S(1 - S)^{b_S} + c_S)$ : This multiplicative factor captures how sparsity modifies the effective capacity. It depends only on $S$ , not on $N$ or $D$ . The term $(1-S)$ is the remaining density (fraction of weights that are non-zero). The power law $(1-S)^{b_S}$ captures the saturating benefit of sparsity: as density decreases (sparsity increases), this term decreases, reducing the capacity penalty. At $S=0$ (dense), this factor becomes $(a_S + c_S)$ , which—when multiplied by $(1/N)^{b_N}$ —subsumes the original dense coefficient $a_N^{b_N}$ (so $a_N$ is absorbed and not separately parameterized). The constant $c_S$ represents a floor on how much sparsity can help—even as $S \to 1$ (density $\to 0$ ), the factor approaches $c_S$ , not zero, meaning there is a limit to the capacity benefit from making the model extremely sparse.
$(1/N)^{b_N}$ : The standard size scaling, identical in form to the dense law but using non-zero parameter count. This captures the power-law decrease in loss as the number of retained parameters increases.
$(a_D/D)^{b_D}$ : The standard data scaling, identical to the dense law and independent of sparsity. This term is not multiplied by the sparsity factor—it stands alone.
$c$ : The irreducible loss, identical in interpretation to the dense law.

What the equation computes: Given a sparsity level $S$ , a non-zero parameter count $N$ , and a training data amount $D$ , the equation predicts the expected validation loss. The first multiplicative group $[(a_S(1-S)^{b_S} + c_S) \cdot (1/N)^{b_N}]$ is the capacity-limited component—this dominates when the model is small relative to the task complexity. The second additive group $(a_D/D)^{b_D}$ is the data-limited component—this dominates when training data is insufficient. The constant $c$ is the floor that remains even with infinite capacity and infinite data.

Why this form, based on the observations:

Parallel lines → multiplicative sparsity factor. Observation 1 shows that sparsity changes the intercept on a log-log plot without changing the slope. An intercept shift in log-log corresponds to multiplication by a constant in linear space. Hence the sparsity term multiplies the size term, and the exponent $b_N$ is shared across all sparsity levels (not a separate $b_N(S)$ per sparsity).
Diminishing returns → saturating power law. Observation 2 shows that the benefit of sparsity saturates. The form $(1-S)^{b_S}$ is a decreasing power law in density: as density goes from 1 to 0.5 (50% sparsity), the term drops from $1^{b_S}=1$ to $0.5^{b_S}$ , which can be a substantial change depending on $b_S$ . But going from 0.25 to 0.125 (75% to 87.5% sparsity) changes density by only 0.125, producing a much smaller change. The additive $c_S$ provides a floor so that the factor doesn't go to zero as density → 0 (which would imply infinite capacity benefit, physically impossible). The sum $a_S(1-S)^{b_S} + c_S$ captures both the power-law benefit and the saturation limit.
Data independence → separate additive data term. Observation 3 shows that sparsity and training duration don't interact. Hence the data term $(a_D/D)^{b_D}$ is not multiplied by the sparsity factor—it remains an independent additive bottleneck. This is a strong structural claim: it says that the primary benefit of sparsity is to increase model capacity (the ability to absorb information), not to change how efficiently the model uses each training example.

An important note on sparsity representation: The authors define the sparsity power-law in terms of the compression rate $1/(1-S)$ rather than sparsity $S$ directly. This ensures proper scaling behavior: "0.75 is twice as sparse as 0.5" in terms of the expanded dense model size (a 75% sparse model with N non-zeros was originally a 4N dense model; a 50% sparse model with N non-zeros was originally a 2N dense model—exactly half the original size). The density complement $(1-S)$ in the equation directly captures the fraction of retained weights.

The dense recovery property: At $S = 0$ , the sparsity factor becomes $(a_S + c_S)$ , and the full law reduces to:

$L(0, N, D) = (a_S + c_S) \cdot \left(\frac{1}{N}\right)^{b_N} + \left(\frac{a_D}{D}\right)^{b_D} + c$

which is exactly the dense form (Equation 2) with $a_N^{b_N} = a_S + c_S$ . So the sparse law nests the dense law as a special case, which is a necessary consistency condition.

The 7 free parameters: The law has exactly 7 parameters to fit: $a_S$ , $b_S$ , $c_S$ , $b_N$ , $a_D$ , $b_D$ , and $c$ . This is a relatively small number given the complexity of the phenomenon being modeled (a 3D surface over S, N, D). For the n:m sparsity extension, only the sparsity parameters $a_S$ , $b_S$ , $c_S$ are refit (3 parameters), with the other 4 inherited from the unstructured fit—exploiting the modularity of the law.

Fitting procedure (T5/C4). Following Hoffmann et al. (2022), the coefficients are fitted by minimizing the Huber loss between log-predicted and log-actual validation loss:

$\text{Loss}_{\text{fit}} = \text{Huber}_\delta(\log L_{\text{pred}} - \log L_{\text{measured}})$

with $\delta = 0.001$ . The Huber loss (quadratic for small errors, linear for large errors) is used for "robustness against outliers"—a few anomalous runs won't dominate the fit. The log transformation is standard for power-law fitting because it converts multiplicative relationships to additive ones and makes errors relative (a 0.1 absolute error matters differently at loss 1.0 vs. loss 10.0). Optimization uses BFGS (a quasi-Newton method) from multiple random starting points to avoid local minima.

Fitting procedure (ViT/JFT-4B). Here the authors use $\delta = 0.01$ and do not take the log of L. The rationale: the NLP-optimized settings from the T5 fit "exclude outliers too aggressively for ViT data (which gives a poor fit for smaller models)." This is an empirical adjustment—the ViT loss surface appears to have different outlier characteristics than the T5 loss surface—and it means the ViT fit is more sensitive to absolute loss magnitudes.

Fit quality assessment (Figure 1, Left). The paper evaluates fit quality on T5/C4 by plotting predicted vs. measured loss. The fitting data (points used in optimization) aligns closely with the diagonal (perfect prediction). The extrapolation point—a 2.3 billion parameter T5-XL model pruned to 75% sparsity, approximately 6.75× larger in non-zero parameters than the maximum in the fitting data—also lands close to the predicted value. This extrapolation test is methodologically important: scaling laws are most useful when they predict beyond the fitted range, and the T5-XL extrapolation validates that the law generalizes to larger scales.

ViT fit assessment (Figure 3). For ViT, the paper shows a qualitative comparison: measured loss-vs-data curves (points) overlaid with predicted curves (lines) from the single joint fit, organized by sparsity level. The predictions match the data well, with the main deviation occurring "at the lowest amount of training" where "some points are a bit off the prediction curve"—the authors suspect this is because very short training runs "only involve comparatively few training steps, which may be a slight bottleneck for the optimization process."

Coefficient values (Table 7). The fitted coefficients are:

ViT/JFT unstructured: $a_S = 2.94 \times 10^2$ , $b_S = 0.821$ , $c_S = 4.68 \times 10^2$ , $b_N = 0.392$ , $a_D = 2.37 \times 10^8$ , $b_D = 0.890$ , $c = 4.517$
T5/C4 unstructured: $a_S = 1.68 \times 10^1$ , $b_S = 0.722$ , $c_S = 4.50 \times 10^1$ , $b_N = 0.245$ , $a_D = 6.90 \times 10^8$ , $b_D = 0.203$ , $c = 0.651$

A key observation: $b_D = 0.890$ for ViT vs. $b_D = 0.203$ for T5. This means data scaling is much stronger for ViT on JFT (loss decreases rapidly with more images) than for T5 on C4 (loss decreases slowly with more tokens). Conversely, $b_N = 0.392$ for ViT vs. $b_N = 0.245$ for T5, so size scaling is also slightly stronger for ViT. These differences reflect the fundamental characteristics of the two tasks and datasets—JFT-4B classification benefits more from additional data than C4 language modeling does, possibly because language modeling has a higher irreducible entropy (more inherent uncertainty in predicting the next token).

Optimal Sparsity: From Description to Prescription

The scaling law predicts loss for any (S, N, D) combination. But the practical question is: given a fixed budget—I want a model with N non-zero parameters at inference time, and I have a total training compute budget C—what sparsity should I use? This is the optimal sparsity $S_{opt}(N, C)$ .

The cost model for sparse training. Computing $S_{opt}$ requires a model of how much training compute a sparse model consumes. The dense training FLOPs are approximated (Kaplan et al., 2020) as $6ND$ for decoder-only models or $3ND$ for encoder-decoder models (the factor of 2 difference is because encoder-decoder models process both input and output sequences, roughly halving the effective parameter count per FLOP). For the sparse case, the cost depends on when during training the sparsity is applied, because the model starts dense and becomes increasingly sparse. This is captured by the multiplier $c_{mul}(S)$ :

$c_{mul}(S) = (0.25 + 0.50 \cdot (1 - 0.75 \cdot S))/(1 - S) + 0.25$

where $S$ is the target sparsity and the terms correspond to the pruning schedule described in Section 3.1.

Derivation of $c_{mul}(S)$ : The pruning schedule divides training into three phases:

First 25% of training: The model is fully dense with size $1/(1-S)$ times the target non-zero count. Cost contribution: $0.25/(1-S)$ , because the dense model is $1/(1-S)$ times larger and trains for 25% of the total duration.
Middle 50% of training: The model is gradually pruned from density 1 to density $(1-S)$ following a cubic schedule $S \cdot (1 - (1-t)^3)$ where $t \in [0,1]$ spans the pruning interval. The average density over this period is $\int_0^1 [1 - S \cdot (1 - (1-t)^3)] dt$ . Computing this integral: the density at time $t$ is $1 - S(1 - (1-t)^3) = 1 - S + S(1-t)^3$ . The integral from 0 to 1 is $1 - S + S \cdot [-(1-t)^4/4]_0^1 = 1 - S + S \cdot (0 - (-1/4)) = 1 - S + S/4 = 1 - 0.75S$ . So the average density (fraction of weights present) is $1 - 0.75S$ . Multiplying by the 0.50 duration fraction gives cost contribution $0.50 \cdot (1 - 0.75S)$ , then dividing by $(1-S)$ to account for the initial model being $1/(1-S)$ times larger. Wait—the authors express this as $0.50 \cdot (1 - 0.75 \cdot S)/(1 - S)$ . Let me re-derive: during the middle phase, the model's size relative to the target non-zero count is $\text{density}/(1-S)$ . The integral of density over the 50% duration is $0.50 \cdot (1 - 0.75S)$ . Dividing by $(1-S)$ gives the cost relative to training a dense model of the target size. So the term is $0.50 \cdot (1 - 0.75S)/(1 - S)$ .
Last 25% of training: The model is at the target sparsity, with density $(1-S)$ relative to its expanded size, meaning its actual size equals the target non-zero count. Cost contribution: $0.25$ (same as training a dense model of the target size for 25% of the duration).

Summing all three phases gives Equation 4. The multiplier $c_{mul}(S)$ is always > 1 for $S > 0$ , reflecting that sparse training is more expensive than training a dense model of the target non-zero size, because you train a larger initial model and only gradually prune it.

Two definitions of training cost. The paper considers two ways to account for training FLOPs:

Dense FLOPs: The cost is $C = 6N \cdot D/(1-S)$ , treating the sparse model as if it were a dense model of size $N/(1-S)$ (the expanded size before pruning) trained for $D$ steps. This assumes sparsity provides NO training speedup—the pessimistic view.
Sparse FLOPs: The cost is $C = 6N \cdot D \cdot c_{mul}(S)$ , accounting for the actual FLOPs consumed by the gradual pruning schedule. This assumes sparsity CAN be exploited during training as soon as weights are removed—the optimistic view.

The distinction matters because it changes where $S_{opt}$ falls. With dense FLOP accounting, sparsity looks more expensive (you pay for the full dense model throughout training), so $S_{opt}$ will be lower. With sparse FLOP accounting, sparsity looks cheaper (you stop paying for pruned weights), so $S_{opt}$ will be higher.

From cost to data: Given a compute budget $C$ and sparsity $S$ , the effective data $D_S$ that can be processed is:

$D_S = \frac{C}{6N \cdot c_{mul}(S)}$

This simply rearranges $C = 6N \cdot D_S \cdot c_{mul}(S)$ to solve for $D_S$ . For dense FLOP accounting, replace $c_{mul}(S)$ with $1/(1-S)$ . The key: higher sparsity increases $c_{mul}(S)$ , which reduces $D_S$ for a fixed compute budget—you get less data because each training step costs more.

The optimal sparsity equation. To find $S_{opt}$ , substitute $D_S$ into the scaling law and differentiate with respect to $S$ . The validation loss at sparsity $S$ with compute budget $C$ and non-zero parameters $N$ is:

$L(S, N, C) = \left(a_S(1 - S)^{b_S} + c_S\right) \cdot \left(\frac{1}{N}\right)^{b_N} + \left(\frac{a_D}{D_S}\right)^{b_D} + c$

Substituting $D_S = (C/6N)/c_{mul}(S)$ :

$L(S, N, C) = \left(a_S(1 - S)^{b_S} + c_S\right) \cdot \left(\frac{1}{N}\right)^{b_N} + a_D^{b_D} \cdot \left(\frac{6N \cdot c_{mul}(S)}{C}\right)^{b_D} + c$

Differentiating with respect to $S$ and setting to zero yields the condition for optimal sparsity:

$a_D b_D \cdot \frac{c'_{mul}(S)}{c_{mul}(S)} \cdot (D_S/c_{mul}(S))^{-b_D} = a_S b_S \cdot (1 - S)^{b_S - 1} \cdot N^{-b_N}$

where $c'_{mul}(S)$ is the derivative of the cost multiplier with respect to sparsity.

What this equation computes: The left side represents the marginal cost of higher sparsity—how much the data term increases because you get less training data for your compute budget (since sparse training is more expensive). The right side represents the marginal benefit—how much the capacity term decreases because sparsity increases effective model capacity. At $S_{opt}$ , these marginal costs and benefits exactly balance. If the left side (cost) exceeds the right side (benefit), you should sparsify less; if the right side exceeds the left, you should sparsify more.

Why this form: The derivative separates cleanly into the data-cost effect (left) and the capacity-benefit effect (right). The cost effect depends on $c'_{mul}(S)/c_{mul}(S)$ —the relative rate of change of the training cost with respect to sparsity—multiplied by the data scaling exponent $b_D$ (how much loss changes with data) and the data term itself (which depends on the compute budget $C$ ). The benefit effect depends on $a_S b_S$ (the sparsity scaling parameters), $(1-S)^{b_S-1}$ (the derivative of the density power law), and $N^{-b_N}$ (how much sparsity matters depends on model size—larger models have smaller $N^{-b_N}$ , so the absolute benefit of sparsity is smaller).

An important property: The contour equation implies $D_S = O(N^{b_N/b_D})$ . This means that if data scaling is stronger than size scaling ( $b_D > b_N$ ), then the same optimal sparsity occurs at a smaller data-to-size ratio for larger models. Intuitively: if performance is more bottlenecked by capacity than by data (which is true for large models in the overtraining regime), adding sparsity (which increases capacity) is more beneficial, so $S_{opt}$ is higher.

Closed-form solution for dense FLOPs: For the special case of dense FLOP accounting, $c_{mul}(S) = 1/(1-S)$ and $c'_{mul}(S) = 1/(1-S)^2$ . The ratio $c'_{mul}(S)/c_{mul}(S) = 1/(1-S)$ . Substituting and solving for $S$ yields:

$S_{opt}(N, C) = \max\left\{1 - \exp\left(\left[\log\frac{a_D b_D}{a_S b_S} + b_N \log N\right]/(b_D + b_S)\right) \cdot \left(\frac{C}{6N}\right)^{-b_D/(b_D + b_S)}, 0\right\}$

What this computes explicitly: Given non-zero parameter count $N$ and total compute budget $C$ (in FLOPs), this formula returns the sparsity level that minimizes validation loss. The $\max\{\cdot, 0\}$ clamp ensures the result is non-negative—if the formula returns a negative value, the optimal sparsity is 0 (dense).

How to read this: The term $(C/6N)^{-b_D/(b_D + b_S)}$ decreases as the compute-per-parameter ratio increases—more compute per parameter means you're training longer relative to model size, which makes sparsity more beneficial (since dense models start to saturate). The log term inside the exponential captures the relative strength of data vs. sparsity scaling—if $a_D b_D$ (data scaling strength) is large relative to $a_S b_S$ (sparsity scaling strength), the optimal sparsity is lower because gaining training data matters more than gaining capacity.

Empirical results for $S_{opt}$ (Figures 1 Right, 4). The paper plots optimal sparsity iso-contours on the (non-zero parameters, pretraining FLOPs) plane. Key findings:

All sparsity contours are parallel to the dense Chinchilla compute-optimal line. This is by construction—Equation 5 has this property—but it's an elegant result: the line that separates regions where different sparsities are optimal runs parallel to the line that defines the optimal dense size-data tradeoff.
For sparse FLOP accounting on ViT (Figure 1 Right), 50% sparsity becomes optimal at less than 2× Chinchilla training duration. On T5 (Figure 4), it becomes optimal at less than 3× Chinchilla. With dense FLOP accounting (pessimistic training cost), the thresholds are higher: approximately 5× Chinchilla for ViT and approximately 70× Chinchilla for T5.
The practical implication: "as one trains significantly longer than Chinchilla, more and more sparse models start to become optimal." Since many production models ARE trained far beyond Chinchilla (the authors cite LLaMA 2 7B at ~14× Chinchilla and the largest LMs at >100× Chinchilla), sparsity is already optimal in these regimes under either cost model.
An important structural difference from dense scaling: "while optimal dense models define a line on the parameter-FLOPs surface, optimal sparse models form a half-plane (with different sparsities unlocking multiple optimal sizes for a fixed training cost)." This means that for a given compute budget, there isn't a single optimal model size—there's a range of sizes, each with its own optimal sparsity, all achieving similar loss.

Figure 5 concrete example: The paper shows loss vs. sparse pretraining FLOPs for ViT models at three sizes (0.66M, 1.33M, 2.65M non-zeros) with different sparsities. At each size, the sparse models (50%, 75%, 87.5%) achieve lower loss than the dense model across a range of compute budgets. Critically, the benefit of sparsity is larger when the compute budget is larger (the curves fan out)—this directly shows that "the optimal sparsity increases with longer training."

Limit Performance: How Much Larger Must a Dense Model Be?

The optimal sparsity analysis tells you when to use sparsity. The limit performance analysis answers a different question: ignoring training cost, in the infinite-data limit, how much larger does a dense model need to be to match the loss of a sparse model? This quantifies the pure capacity benefit of sparsity.

In the infinite-data limit ( $D \to \infty$ ), the data term $(a_D/D)^{b_D} \to 0$ , and the loss reduces to:

$L_{\infty}(S, N) = \left(a_S(1 - S)^{b_S} + c_S\right) \cdot \left(\frac{1}{N}\right)^{b_N} + c$

The question: for a sparse model with sparsity $S$ and non-zero parameters $N$ , what dense model size $N_{dense}$ achieves the same loss? Setting $L_{\infty}(S, N) = L_{\infty}(0, N_{dense})$ and solving:

$\left(a_S(1 - S)^{b_S} + c_S\right) \cdot \left(\frac{1}{N}\right)^{b_N} = (a_S + c_S) \cdot \left(\frac{1}{N_{dense}}\right)^{b_N}$

$\frac{N_{dense}}{N} = \left(\frac{a_S + c_S}{a_S(1 - S)^{b_S} + c_S}\right)^{1/b_N}$

The gain factor is:

$\text{gain}(S) = \left(\frac{a_S(1 - S)^{b_S} + c_S}{a_S + c_S}\right)^{-1/b_N}$

What this computes: For a given sparsity $S$ , $\text{gain}(S)$ is the multiplier on non-zero parameters—a sparse model with $N$ non-zero parameters performs equivalently to a dense model with $\text{gain}(S) \times N$ parameters, assuming both are trained on infinite data. This is the effective capacity multiplier of sparsity.

Why it's independent of N and D: Because sparsity acts multiplicatively on the size term and the size scaling shares the same exponent $b_N$ across all sparsity levels, the relative gain $N_{dense}/N$ does not depend on $N$ itself—only on $S$ and the fitted coefficients. This is a direct consequence of Observation 1 (parallel lines on log-log plots).

Empirical gains (Table 2):

Sparsity	ViT/JFT	T5/C4
50%	1.60×	1.59×
75%	2.17×	2.16×
87.5%	2.63×	2.63×

These numbers are remarkably consistent across vision and language. A 75% sparse model with N non-zero parameters matches a dense model with approximately 2.16N parameters. This means the effective capacity gain from sparsity is less than the compression rate: at 75% sparsity (4× compression), you only get 2.17× effective capacity gain, not 4×. The "lost" capacity represents the imperfect utilization of the additional parameters—the zero pattern can encode information, but less efficiently than if those parameters were non-zero and fully trained.

Diminishing returns are clear: Going from 50% to 75% sparsity (doubling compression from 2× to 4×) adds only 0.57-0.58× additional gain (1.60→2.17). Going from 75% to 87.5% (doubling compression again from 4× to 8×) adds only 0.46-0.47× (2.17→2.63). The marginal capacity benefit per unit of sparsity decreases substantially.

Practical interpretation: A 75% sparse model with 100M non-zero parameters, trained on infinite data, achieves the same loss as a 217M-parameter dense model. At inference time, the sparse model costs the same as a 100M-parameter dense model (assuming ideal sparsity acceleration). The "waste" relative to the 4× compression is 183M parameters (400M original - 100M retained = 300M pruned, but only 117M effective gain). The pruned parameters do contribute something—just not as much as if they had been non-zero.

Why this matters: The $\text{gain}(S)$ values provide the theoretical ceiling on sparsity's practical benefit. If you're considering deploying a sparse model instead of a dense one, and your training budget is large enough to reach the capacity-limited regime, you can expect at most a 2.17× parameter-equivalent gain at 75% sparsity. Any deployment decision involving sparsity should use these numbers (or ones refit to the specific model/dataset) as upper bounds on expected improvement.

4. Key Insights and Innovations

Innovation 1: Sparsity as a Multiplicative Capacity Modifier That Is Independent of Data Scaling

The foundational conceptual contribution of this paper is the discovery and empirical validation that sparsity enters the scaling law as a separable, multiplicative modifier of model capacity that does not interact with data scaling. This is not an assumption—it emerges from the data (Figure 2) and is then formalized, fitted, and validated across two domains.

Prior to this work, the default assumption in the sparsity literature was essentially agnosticism on this question: nobody knew whether sparsity would change the fundamental shape of scaling, whether it would interact with training duration, or whether its benefits would be larger or smaller at different model sizes. The Rosenfeld et al. (2021) study on CIFAR-10 had examined width-depth-density relationships but in a regime where data scaling was essentially irrelevant (small datasets, quick saturation), so it could not speak to the data dimension that is central to foundation model training. The Gopher appendix (Rae et al., 2021) and Cerebras (2022) sparsity results were suggestive but confounded sparsity effects with additional training—they showed sparse models could beat dense ones, but could not disentangle whether the gain came from sparsity per se or simply from training longer on more data.

The structural finding—that the loss-vs-size curves for different sparsity levels are parallel on log-log axes (same exponent $b_N$ , different intercept), and that sparsity and training duration do not interact—is what makes the separable form $(a_S(1-S)^{b_S} + c_S) \cdot (1/N)^{b_N} + (a_D/D)^{b_D} + c$ possible. This is not merely a convenient functional form; it is a claim about the mechanism by which sparsity improves performance. If sparsity modified how efficiently the model uses data—for instance, if sparse models learned faster per example, or if high sparsity became more beneficial when data was scarce—then the sparsity term would need to multiply the data term, or there would need to be a cross-term, and the law would be more complex. The fact that the simple separable form fits well (with only 7 parameters across a 112-run ViT sweep) is evidence that sparsity primarily increases representational capacity (the ability to store and express more complex functions for a given non-zero parameter count) rather than changing the efficiency of learning from data.

This insight has immediate practical and theoretical implications. On the practical side, it means the capacity gain from sparsity—the $\text{gain}(S)$ values in Table 2—is a universal property of the sparsity level, independent of how much you train or how large the model is. A 75% sparse model provides approximately 2.16× effective capacity gain whether it has 1M or 100M non-zero parameters, and whether it sees 100M or 1B training images. This predictive stability is what makes the scaling law actually useful for decision-making: you don't need to re-fit the law at every scale. On the theoretical side, it suggests that the zero pattern in a sparse weight matrix encodes information in a way that is analogous to having additional parameters, but with a reduced efficiency (the gain is less than the compression rate). The factor $(a_S(1-S)^{b_S} + c_S)$ quantifies exactly how much less: the saturating power-law form captures that early sparsity (going from 0% to 50%) provides substantial relative gain, while later sparsity (75% to 87.5%) provides diminishing returns because useful zero patterns become harder to find.

The cross-domain consistency of this finding—the same functional form fits both ViT/JFT-4B (multi-label classification, $b_D = 0.890$ ) and T5/C4 (masked language modeling, $b_D = 0.203$ )—elevates it from an empirical observation about one setup to a candidate universal property of weight-sparse Transformers. The fact that the $\text{gain}(S)$ values are nearly identical across vision and language (1.60× vs. 1.59× at 50%, 2.17× vs. 2.16× at 75%) despite very different tasks, architectures, and scaling exponents is striking and suggests the sparsity-capacity relationship may be determined more by fundamental properties of gradient-based optimization in overparameterized networks than by domain-specific factors. This is a significant conceptual advance because it means sparsity research can potentially focus on improving the efficiency of the capacity encoding (raising the gain closer to the compression rate) rather than worrying about task-specific interactions.

Innovation 2: Optimal Sparsity as an Analytic Concept with Training-Duration-Dependent Thresholds

The paper introduces the concept of optimal sparsity $S_{opt}(N, C)$ —the sparsity level that minimizes validation loss for a given inference size and training compute budget—and provides an analytic framework for computing it. This transforms sparsity from a heuristic choice ("try 50% and 75% and see what works") into a principled, derivable quantity that emerges from the same scaling principles that govern model sizing and data allocation.

What makes this novel is not merely the definition—"optimal" anything is a standard optimization framing—but the structure of the answer. The key finding is that optimal sparsity increases with training duration relative to the Chinchilla point: "as one trains significantly longer than Chinchilla (dense compute optimal), more and more sparse models start to become optimal in terms of loss for the same number of non-zero parameters." This is non-obvious because it means sparsity is not universally beneficial or universally harmful—its value is conditional on the training regime. In the compute-optimal (Chinchilla) regime where model size and training data are perfectly balanced, dense models are optimal or nearly optimal. But as training extends beyond Chinchilla into the overtraining regime that characterizes most production foundation models (LLaMA 2 7B at ~14× Chinchilla, the largest LMs at >100× Chinchilla), sparsity becomes increasingly favorable.

The mechanism behind this is a tradeoff that the scaling law makes explicit: sparsity increases effective capacity (reducing the capacity-limited component of the loss) but also increases training cost (reducing the amount of data you can process for a fixed compute budget, which increases the data-limited component). Near Chinchilla optimality, the data-limited component is still significant, so the training-cost penalty of sparsity outweighs the capacity benefit. Far beyond Chinchilla, the data-limited component has already saturated (you have abundant data relative to model capacity), so the capacity benefit dominates. This is why the $S_{opt}$ contours (Figures 1 and 4) run parallel to the Chinchilla line—the balance point between capacity and data bottlenecks shifts systematically with scale, and sparsity slides along with it.

A particularly elegant structural result is the distinction between the dense and sparse views: "while optimal dense models define a line on the parameter-FLOPs surface, optimal sparse models form a half-plane (with different sparsities unlocking multiple optimal sizes for a fixed training cost)." In dense scaling, for a given compute budget, there is exactly one optimal model size (the Chinchilla point). With sparsity, for a given compute budget, there is a range of model sizes, each with its own optimal sparsity, that all achieve similar loss. This fundamentally changes how one thinks about model design: instead of a single optimal point, you have a frontier of (size, sparsity) combinations to choose from based on deployment constraints (inference latency, memory, hardware sparsity support) rather than training constraints alone.

The practical significance of this framing is that it provides a decision tool for practitioners. Given a target inference size (e.g., "I need a model with 1B non-zero parameters that fits on one GPU") and a training budget, the $S_{opt}$ formula tells you whether to train a 1B dense model, a 2B model pruned to 50%, a 4B model pruned to 75%, etc. The answer depends on the training budget: if you're training at 2× Chinchilla, go dense or 50% sparse; if you're training at 10× Chinchilla, 75% sparse becomes optimal. This is the sparsity analog of the Chinchilla compute-optimal recipe (Hoffmann et al., 2022)—instead of telling you "for C FLOPs, train a model of size N on D tokens," it tells you "for C FLOPs with inference target N, train at sparsity S."

The quantitative thresholds the paper finds—50% sparsity becomes optimal at <2× Chinchilla for ViT under sparse-FLOP accounting, 75% at some higher multiple—are specific to the fitted coefficients and should not be taken as universal constants. But the qualitative shape of the relationship— $S_{opt}$ increasing monotonically with training duration beyond Chinchilla—is the durable insight that would likely persist across model families and datasets even as coefficient values change.

Innovation 3: Sparsity Gains Are Universal but Sub-Compression-Rate—A Fundamental Efficiency Bound

The paper's limit performance analysis (Table 2) reveals a finding that is simultaneously encouraging and sobering: sparsity provides a genuine, cross-domain capacity benefit, but the benefit is substantially less than the compression rate would naively suggest. This is not a failure of pruning methodology—the authors explicitly tested alternatives and found GMP to be competitive—but rather appears to be a fundamental property of how zero patterns encode information in neural networks.

The specific numbers are striking in their consistency: 75% sparsity (4× compression) yields approximately 2.16× effective capacity gain across both ViT and T5, and 87.5% sparsity (8× compression) yields approximately 2.63× gain. This means the efficiency of sparsity—the ratio of effective gain to compression rate—is approximately 2.17/4 ≈ 54% at 75% sparsity and drops to 2.63/8 ≈ 33% at 87.5% sparsity. The remaining capacity (the "wasted" parameters that are pruned but don't contribute to effective gain) represents the inherent limitation of representing information through zero patterns rather than through trained non-zero weights.

Why is this a conceptual innovation rather than just a measurement? Because it quantifies a fundamental bound that prior work could only gesture at. The sparsity literature has long acknowledged that sparsity provides diminishing returns at very high levels, but the reasons were confounded: is it because pruning algorithms are imperfect at finding the right weights to remove? Because highly sparse networks are harder to optimize? Because the architecture fundamentally can't utilize extreme sparsity? By fitting the saturating power-law form $(a_S(1-S)^{b_S} + c_S)$ and extracting the asymptotic constant $c_S$ , the paper separates the "diminishing returns within the power-law regime" (captured by $b_S$ ) from the "hard floor on sparsity benefit" (captured by $c_S$ ). The fitted values show that $c_S$ is approximately $a_S + c_S$ (the dense baseline) times 0.16 for ViT and 0.13 for T5—meaning that even at extreme sparsity, the capacity penalty factor cannot drop below ~13-16% of the dense baseline. This is the irreducible capacity cost of sparsity: no amount of additional zero patterns can reduce the effective capacity multiplier below this floor.

This finding has a direct bearing on the practical value proposition of sparsity. If sparsity's effective gain were close to the compression rate (e.g., 4× gain at 4× compression), then sparse models would be unambiguously superior to dense models for any deployment—you'd get smaller, faster models with no accuracy penalty. The fact that the gain is sub-compression-rate means there is a genuine accuracy-efficiency tradeoff: a 75% sparse model with N non-zeros achieves lower loss than a dense N-parameter model (2.17× gain), but higher loss than a dense 4N-parameter model (because 2.17 < 4). So sparsity is not a free lunch—it buys you inference efficiency at a modest accuracy cost relative to the original uncompressed model. The paper's framework lets you quantify exactly how modest.

The cross-domain consistency of the gain values also has methodological implications. It suggests that the sparsity-capacity relationship may be more fundamental to the optimization geometry of Transformers than to the specific task or dataset. If the gain at 75% sparsity had been 1.5× for ViT and 3.0× for T5, one would conclude that sparsity's value is highly task-dependent and that each application requires its own scaling law. The near-identity of the gains (within 0.01) across vision and language instead points toward a more universal phenomenon—perhaps related to the information-theoretic capacity of sparse random matrices, or to the properties of stochastic gradient descent in overparameterized networks. This opens the door to theoretical work that could predict the $\text{gain}(S)$ curve from first principles rather than requiring expensive empirical fitting.

Innovation 4: A Rigorous Evaluation Framework That Exposes the Confounds in Prior Sparse Foundation Model Work

Section 2 of the paper is not merely a "related work" discussion—it is a methodological critique that identifies specific, systematic flaws in how prior work evaluated sparse large models, and then constructs an evaluation protocol that eliminates those flaws. This is an intellectual contribution in its own right because it changes the standards by which sparse model claims should be judged.

The critique identifies three confounds that have historically inflated apparent sparsity benefits when applied to foundation models:

The training duration confound. In classic pruning benchmarks (ResNet50/ImageNet), dense models overfit quickly, so extending training during pruning has no meaningful dense baseline—there's no dense model that also trained for that extended duration. But in the foundation model regime, dense models continue to improve with more data (Zhai et al., 2022), so a sparse model that started from a pretrained checkpoint and then trained further during pruning has consumed more total data than the original dense checkpoint. The fair comparison is against a dense model trained on the same total data—and that dense model is substantially better than the original checkpoint. Prior work (Rae et al., 2021; Cerebras, 2022) did not make this comparison.

The architectural efficiency confound. In the ResNet era, simply scaling down a model produced poor architectures—a 2× smaller ResNet was not competitive with a 2× compressed ResNet, so sparse models could win without facing a legitimate small-dense baseline. In the Transformer era, width/depth scaling produces competitive models at any size (Kaplan et al., 2020), so a dense model with the same number of non-zero parameters as a 50% sparse model is a strong baseline that must be included. The paper's choice to scale only hidden dimension (not all architectural parameters) is specifically designed to ensure this baseline is fair and competitive.

The compute accounting confound. Training a sparse model consumes more FLOPs than training a dense model of the target non-zero size, because you start from a larger dense model and only gradually prune. If this extra compute is not accounted for, sparse models can win simply because they consumed more resources—not because sparsity is inherently beneficial. The paper's $c_{mul}(S)$ derivation (Equation 4) and the use of both "dense FLOPs" and "sparse FLOPs" accounting frameworks are designed to make this explicit and to show how the conclusions depend on one's assumptions about sparsity acceleration during training.

By insisting on equal non-zero parameters, equal total data/FLOPs, and equal architectural competitiveness, the paper constructs what is essentially the hardest possible evaluation setting for sparsity—one where sparsity gets no credit for training longer, using more parameters, or exploiting architectural weaknesses in the dense baseline. The fact that sparsity still shows gains in this setting (the 2.17× capacity multiplier, the optimality beyond Chinchilla) is therefore a stronger statement than it would be in a laxer evaluation—the gains that survive this protocol are gains that can be attributed to sparsity per se, not to confounds.

This framework has implications beyond this paper. It provides a template for future sparse model evaluations in the foundation model era: always compare to a same-nonzero-parameter dense model, always equalize total data, always account for training FLOP differences. The paper's own analysis of prior results (Section 6) demonstrates the value of this template by showing that the seemingly impressive 5× gain reported by Cerebras (2022) is actually consistent with the paper's scaling law once the confounds are properly modeled—it's not that Cerebras's results are wrong, but that they conflate sparsity benefits with the benefits of training on 5× more data. This kind of diagnostic decomposition is only possible with a rigorous evaluation framework and a calibrated scaling law.

The intellectual contribution here is not a new pruning method or a new scaling law—it's a new standard of evidence for claims about sparse foundation models. In a field where reproducibility and fair comparison have been persistent challenges (the pruning literature has a particularly checkered history of incomparable baselines), this is a significant step toward methodological maturity.

5. Experimental Analysis

Evaluation Methodology

Dataset. The experiments use two datasets: JFT-4B (Dehghani et al., 2023), consisting of 4 billion images for multi-label image classification, and C4 (Raffel et al., 2020a), consisting of 150+ billion tokens for masked language modeling. For ViT, the full 4B-image dataset is used with the longest training runs processing approximately 1.8 billion images (well within a single epoch). For T5, the full C4 dataset is used with the longest runs processing approximately 65 billion tokens, also within a single pass. The paper uses the standard validation splits from each dataset to record final loss values for every run in the sweep grid.
Base model(s). Two model families are used: Vision Transformers (ViT) (Dosovitskiy et al., 2021) and encoder-decoder T5 models (version 1.1) (Raffel et al., 2020b; Google, 2023b). For ViT, the base architecture is ViT-B/16, scaled by varying only the hidden dimension while keeping all other shape parameters (number of layers, attention head ratio, MLP expansion ratio) constant, producing 7 target sizes from 0.66M to 42.47M non-zero parameters in 2× increments. For T5, the base architecture is t5-base, similarly scaled by hidden dimension only, producing 4 target sizes from 1.3M to 85M non-zero parameters in 4× increments. The models are chosen to be "representative of the capabilities of many contemporary" Transformer architectures and to span both vision and language domains with different pretraining objectives (multi-label classification vs. masked language modeling), enabling cross-domain validation of the scaling law's generality.
Metrics. The primary metric is validation loss: multi-label classification loss for ViT/JFT-4B and masked language modeling cross-entropy loss for T5/C4. All losses are recorded at the end of training for each run in the sweep grid. The paper uses these raw loss values for fitting, with the lower bound $c$ subtracted in visualizations to account for power-law saturation against the inherent uncertainty limit. When comparing models quantitatively, loss differences are interpreted on the native scale (lower is better). For the T5 fit, the Huber loss of $\log L$ is minimized ( $\delta = 0.001$ ), following Hoffmann et al. (2022), which treats errors in relative terms. For ViT, the Huber loss on raw $L$ is used ( $\delta = 0.01$ ) because the log transformation was found to "exclude outliers too aggressively for ViT data (which gives a poor fit for smaller models)."
Baselines. The paper uses several baselines, though the nature of the study means there is no single "competitor method"—rather, comparisons are between different sparsity levels and against the dense baseline:
- Dense models ( $S = 0$ ) : The primary reference point at every (size, training duration) combination. The dense model provides the baseline against which sparsity gains (or losses) are measured. Critically, this comparison equalizes non-zero parameter count: a 50% sparse model with N non-zeros is compared to a dense model with N parameters (not the original N/0.5 = 2N dense model it was pruned from), because the inference cost scales with non-zero parameters.
- Different sparsity levels ( $S = 0.5, 0.75, 0.875$ ) : Comparisons across sparsity levels at the same non-zero parameter count and training budget reveal which sparsity is optimal for a given regime.
- Dense Chinchilla compute-optimal line (Hoffmann et al., 2022): Used as a reference for understanding when sparsity becomes optimal relative to the standard dense compute-optimal training point. The Chinchilla line denotes ideal utilization of training FLOPs for fully dense models.
- Prior large-model sparsity results: The paper compares its findings against the Gopher appendix (Rae et al., 2021), Cerebras (2022), and SparseGPT (Frantar & Alistarh, 2023) in the discussion (Section 6), using the fitted scaling law to diagnose whether reported sparse gains are consistent with the law's predictions once confounds (extra training data, unequal FLOP accounting) are modeled.
Generation budget / compute accounting. Compute is measured as pretraining FLOPs, following the standard approximation $6ND$ for decoder-only models and half that for encoder-decoder models (Kaplan et al., 2020). The paper introduces two distinct accounting frameworks:
- Dense FLOPs: Cost is calculated as if the sparse model were a dense model of size $N/(1-S)$ (the expanded pre-pruning size) trained for $D$ steps, i.e., $C = 6N \cdot D/(1-S)$ . This is the pessimistic view—sparsity provides no training speedup.
- Sparse FLOPs: Cost accounts for the actual FLOPs consumed during gradual pruning, using the multiplier $c_{mul}(S) = (0.25 + 0.50 \cdot (1-0.75S))/(1-S) + 0.25$ (Equation 4). This is the optimistic view—sparsity can be exploited during training as soon as weights are removed.
The distinction matters because $S_{opt}$ differs between the two frameworks (sparsity becomes optimal at lower training multiples under sparse-FLOP accounting). All FLOPs calculations assume the standard $6N$ per token per parameter approximation and factor in the architectural differences between ViT (encoder-only) and T5 (encoder-decoder, roughly half the FLOPs per parameter-token).
Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional ML sense for model selection—there is no held-out set for hyperparameter tuning because there are no hyperparameters to tune (the scaling law is fitted to all data points simultaneously). Instead, extrapolation serves as the validation mechanism: for T5, the law is fitted on all runs in the sweep grid (maximum 85M non-zero parameters) and then tested on a 2.3 billion parameter T5-XL model pruned to 75% sparsity—approximately 6.75× larger in non-zero parameters than any point in the fitting set. This extrapolation test (Figure 1, Left) verifies that the fitted law generalizes beyond the interpolation range. For ViT, the full 112-run grid is used for fitting, and fit quality is assessed qualitatively by overlaying predicted and measured curves (Figure 3). The authors do not report formal confidence intervals on the fitted coefficients, nor do they assess uncertainty in the $S_{opt}$ contours; the 48-run T5 grid has approximately 7 datapoints per free parameter, which is relatively sparse for nonlinear fitting and means individual coefficient estimates may have substantial variance even if the overall law captures the trends correctly.

Main Quantitative Results

Scaling Law Fit Quality and Structure

The central quantitative result is that the proposed sparse scaling law $L(S, N, D) = (a_S(1-S)^{b_S} + c_S) \cdot (1/N)^{b_N} + (a_D/D)^{b_D} + c$ fits the sweep data with high fidelity across both domains, despite having only 7 free parameters to capture a 3D surface over sparsity, non-zero parameters, and training data.

T5/C4 fit quality. Figure 1 (Left) shows predicted vs. measured loss for all fitting datapoints plus the extrapolation point. The fitting data clusters tightly around the diagonal identity line, indicating good interpolation quality. The extrapolation point—a 2.3B-parameter T5-XL model at 75% sparsity, trained for 250K steps with batch size 256—lands close to the predicted value despite being approximately 6.75× larger than the maximum fitting point (85M non-zeros) and using different architectural and training hyperparameters (T5-XL architecture rather than a simply rescaled T5-base, batch size 256 rather than 128). The final Huber loss of the log fit is $7.60 \times 10^{-6}$ (Table 7), indicating very tight fit on the log scale. The fitted coefficients (Table 7) are: $a_S = 16.8$ , $b_S = 0.722$ , $c_S = 45.0$ , $b_N = 0.245$ , $a_D = 6.90 \times 10^8$ , $b_D = 0.203$ , $c = 0.651$ . The data exponent $b_D = 0.203$ is notably small, indicating that T5 on C4 benefits relatively slowly from additional training data—the data-scaling curve is shallow. The size exponent $b_N = 0.245$ is similarly modest, suggesting that both capacity and data bottlenecks ease slowly with scale for this task.

ViT/JFT-4B fit quality. Figure 3 shows measured validation loss vs. pretraining data (points) overlaid with the jointly-fitted law's predictions (lines), organized by sparsity level with multiple model sizes per subplot. The fit quality is visually strong across most of the range, with predictions tracking the measured points through curvature and across model sizes. The main deviation occurs "at the lowest amount of training" where "some points are a bit off the prediction curve." The authors attribute this to the fact that these runs "only involve comparatively few training steps, which may be a slight bottleneck for the optimization process"—the models may not be fully converged at the earliest checkpoint, violating the scaling law's assumption of near-optimal training. The total Huber loss (raw scale, $\delta = 0.01$ ) is $4.93 \times 10^{-4}$ (Table 7), which is not directly comparable to the T5 log-scale error but indicates good absolute fit. The fitted coefficients (Table 7) are: $a_S = 294$ , $b_S = 0.821$ , $c_S = 468$ , $b_N = 0.392$ , $a_D = 2.37 \times 10^8$ , $b_D = 0.890$ , $c = 4.517$ . The data exponent $b_D = 0.890$ is dramatically larger than T5's $0.203$ , meaning ViT on JFT-4B benefits much more strongly from additional training data—the loss drops much faster per added image than T5's loss drops per added token. The size exponent $b_N = 0.392$ is also larger than T5's $0.245$ , indicating stronger size scaling. The irreducible loss $c = 4.517$ reflects the multi-label classification task's inherent difficulty floor.

Structural validation of the separable form. The paper does not explicitly ablate the multiplicative-sparsity, additive-data structure against alternative forms (e.g., sparsity multiplying the data term, or a cross-term between sparsity and size). The validation is therefore inductive: Figure 2 shows the parallel-lines property that motivates the separable form, and the good overall fit (Figures 1 and 3) is consistent with the structure being correct. However, with only 48 (T5) to 112 (ViT) datapoints and 7 parameters, the fitted law likely has enough flexibility to mask moderate misspecification—alternative functional forms might fit equally well. The extrapolation test provides stronger evidence because misspecification tends to manifest at larger scales, and the T5-XL point's accuracy argues against severe structural error.

Optimal Sparsity Results

The fitted scaling laws enable computing $S_{opt}(N, C)$ , the sparsity level that minimizes validation loss for a given inference size and training compute budget. Figures 1 (Right, ViT) and 4 (T5) show the optimal sparsity contours on the (non-zero parameters, pretraining FLOPs) plane.

Contour structure. Both figures display iso-contours labeled by the sparsity level that is optimal in each region: 0% (dense), 50%, 75%, and 87.5%. By construction from Equation 5, all contours are parallel to the dense Chinchilla compute-optimal line (the dashed line in each figure). The Chinchilla line itself approximately corresponds to the $S = 0$ boundary under dense-FLOP accounting, though it does not necessarily correspond to $S = 0$ under sparse-FLOP accounting because non-zero sparsity may already be optimal in the Chinchilla regime when training-cost savings from sparsity are credited.

Sparse-FLOP accounting thresholds. For ViT/JFT-4B (Figure 1, Right), the 50% sparsity contour is reached at less than 2× the training duration of the Chinchilla optimal point—meaning that if you train approximately 2× longer than what would be compute-optimal for a dense model, 50% sparsity becomes the best choice for a model with the same non-zero parameters. The 75% contour is reached at a higher multiple (visually approximately 3-4× Chinchilla in the figure), and the 87.5% contour at an even higher multiple. For T5/C4 (Figure 4), the 50% contour under sparse FLOPs is reached at less than 3× Chinchilla. Under dense-FLOP accounting (where sparsity receives no training-speedup credit), the thresholds shift substantially upward: approximately 5× Chinchilla for ViT 50% and approximately 70× Chinchilla for T5 50%.

The half-plane property. A key structural observation visible in both figures: "while optimal dense models define a line on the parameter-FLOPs surface, optimal sparse models form a half-plane (with different sparsities unlocking multiple optimal sizes for a fixed training cost)." For a given compute budget (vertical slice in the contour plots), a dense model has a unique optimal size (the Chinchilla intersection). But with sparsity, there is a range of model sizes—each with its own optimal sparsity—that achieve similar loss. This is visible in the figures as the contour regions being unbounded to the right: for a fixed FLOPs value, as you move rightward (larger model), the optimal sparsity increases, and the loss remains approximately constant along the contour boundaries. This provides flexibility in deployment: you can choose from multiple (N, S) combinations based on inference hardware constraints rather than being locked into a single optimal dense size.

Figure 5 demonstration. For concrete ViT sizes (0.66M, 1.33M, 2.65M non-zeros), the paper plots loss vs. sparse pretraining FLOPs for different sparsity levels. At each size, the sparse models (50%, 75%, 87.5%) achieve lower loss than the dense baseline across a range of compute budgets, with the gap widening at higher budgets (the curves fan out). For the 0.66M non-zero model, the 75% sparse variant achieves loss below 8.5 at approximately $10^{19}$ sparse FLOPs, while the dense model requires substantially more compute to reach the same loss. For the 1.33M model, the 87.5% sparse variant shows only marginal improvement over 75%—the diminishing returns from higher sparsity are clearly visible as the 75% and 87.5% curves nearly overlap. For the 2.65M model, the ordering is similar but the absolute losses are lower across all sparsity levels, consistent with the size scaling captured by $b_N$ .

Practical interpretation. The paper frames these results in terms of current training practices: language models of the sizes studied "are already typically trained for >100× longer than Chinchilla" (Brown et al., 2020), and "Llama2-7B with ≈14× Chinchilla" (Touvron et al., 2023b). Under sparse-FLOP accounting, 50% sparsity is already optimal at 2-3× Chinchilla, meaning it is already optimal for essentially all production foundation models trained today. Even under the pessimistic dense-FLOP accounting, 50% sparsity at ~5× Chinchilla is within range of current practice for vision models, though the ~70× threshold for T5 under dense FLOPs suggests language models would need to be trained even longer than current norms to justify sparsity if no training speedup is realized.

Limit Performance Gains

Table 2 reports the effective capacity multiplier $\text{gain}(S)$ for each sparsity level, computed from the fitted coefficients using Equation 7. This represents the factor by which a dense model must be larger to match the loss of a sparse model with the same number of non-zero parameters, in the infinite-data limit.

Sparsity	ViT/JFT gain	T5/C4 gain
50%	1.60×	1.59×
75%	2.17×	2.16×
87.5%	2.63×	2.63×

Key observations from Table 2:

Cross-domain consistency. The gain values are nearly identical between ViT and T5 (within 0.01 at all three sparsity levels). This is striking given the very different tasks, architectures, datasets, and scaling exponents ( $b_D = 0.890$ vs. $0.203$ , $b_N = 0.392$ vs. $0.245$ ). The authors note that "the fact that the numbers are within 0.01 of each other is likely a coincidence," but the qualitative closeness suggests the sparsity-capacity relationship may be more universal than task-specific factors would predict.
Sub-compression-rate gains. At 75% sparsity (4× compression), the effective gain is only 2.17×—approximately 54% of the naive compression ratio. At 87.5% sparsity (8× compression), the gain is 2.63×—approximately 33% of the compression ratio. The efficiency of sparsity (gain divided by compression rate) drops substantially as sparsity increases.
Diminishing marginal returns. The incremental gain from 50% to 75% sparsity is 0.57-0.58× (1.60 → 2.17), and from 75% to 87.5% is 0.46-0.47× (2.17 → 2.63). Doubling the compression rate produces less than half the additional gain at higher sparsity levels.

What these numbers mean operationally. A 75% sparse ViT with 100M non-zero parameters, when trained on enough data to reach the capacity-limited regime, will achieve the same validation loss as a 217M-parameter dense ViT trained on the same amount of data. At inference time, the sparse model costs as much as a 100M-parameter dense model (assuming ideal sparsity acceleration), so the net effect is a 2.17× reduction in inference cost for the same accuracy, or equivalently, 2.17× higher accuracy for the same inference cost. The remaining 1.83× of the 4× compression (the difference between 4× and 2.17×) represents capacity that was present in the original unpruned 400M model but is not effectively utilized in the sparse version—the zero pattern cannot encode information as efficiently as trained non-zero weights.

Ablation Studies and Robustness Checks

Pruning hyperparameter sensitivity (Table 5): The paper ablates gradual magnitude pruning hyperparameters by sparsifying ViT-B/16 for 900M images to S = 0.9375 (a higher sparsity than the main sweep, chosen to amplify differences). The authors vary the start point (0.125, 0.250, 0.375, 0.475), end point (0.625, 0.750, 0.875), and mask update frequency (50, 100, 200 steps), always using a cubic schedule. The resulting accuracies are tightly clustered: 45.04–45.33 for all configurations except the one with a shortened pruning interval (start 0.475, end 0.625, frequency 100), which drops to 44.65. The key finding: GMP is robust to start/end/frequency choices as long as the pruning interval is not too short. The paper ultimately selects the 0.25-0.75/100 setup to ensure "sufficient time for training a decent model before starting pruning, as well as for properly finetuning the final sparse version."

Alternative sparsification methods (Table 6): GMP is compared against AC/DC (Peste et al., 2021) on T5 models at two sizes (1.3M and 42M non-zeros) and three sparsities (50%, 75%, 87.5%), trained for 250K steps. AC/DC uses alternating sparse and dense cycles of 20K steps (10K each phase) applied during the same training window as GMP. For the 1.3M model, GMP and AC/DC produce nearly identical perplexities at 50% sparsity (both 17.11), while AC/DC slightly outperforms GMP at 75% (15.96 vs. 15.64) and 87.5% (14.59 vs. 14.73). For the 42M model, GMP and AC/DC are identical at 50% (both 6.11), and AC/DC performs slightly worse at 75% (6.05 vs. 5.87) and at 87.5% (6.11 vs. 5.81). The differences are small and inconsistent across scales. The paper also tested STE (Lin et al., 2020) and RigL (Evci et al., 2020) but reports they "saw similar or worse performance, while being more sensitive to hyper-parameters as we scale." A second-order diagonal variant of GMP (Kurtic et al., 2022) performed "almost identically." The finding is that no tested method clearly and consistently outperforms simple GMP at these sparsity levels and scales, justifying the choice of GMP for the scaling law sweeps.

Optimizer modification (sparsity-aware RMS): The paper identifies a subtle interaction between AdaFactor's relative learning rate scaling and high sparsity: "a tensor with many zeros tends to have a lower RMS, resulting in a smaller learning rate, which is especially problematic as high levels of sparsification require more recovery during the pruning process." The solution—computing RMS only over unpruned weights—is applied consistently in all T5 runs and is described as effectively resolving the issue. No quantitative ablation of this choice is reported (e.g., loss with vs. without the correction at different sparsity levels), so the magnitude of the improvement cannot be assessed from the paper.

Fitting procedure sensitivity (ViT log vs. raw loss): The paper notes that the T5-optimized fitting procedure (Huber loss of $\log L$ with $\delta = 0.001$ ) "exclude[s] outliers too aggressively for ViT data (which gives a poor fit for smaller models)." The ViT fit instead uses raw-loss Huber with $\delta = 0.01$ . This is an important robustness observation: the optimal fitting procedure is not universal across tasks, and using log-transformed loss can mask poor fits for smaller models where relative errors may be large but absolute errors are small. The paper does not systematically compare the two fitting approaches on the same data or quantify how much the fitted coefficients change under each transformation. For the n:m sparsity extension (T5/C4), $\delta = 0.01$ (raw loss) is also used because "0.75 patterns will otherwise be treated as an outlier" under the log-transformed Huber, suggesting that the 75% n:m results deviate enough from the power-law pattern that the robust loss function would downweight them excessively.

Extrapolation robustness (T5-XL): The T5-XL extrapolation test (Figure 1, Left) validates that the scaling law generalizes to a model approximately 6.75× larger than the maximum fitting point. The extrapolation point's accuracy is notable because the setup differs from the fitting data in several ways: different architecture (full T5-XL rather than scaled T5-base), different batch size (256 vs. 128), and different training duration (250K steps at batch 256 vs. the fitting grid's maximum of 1M steps at batch 128). Despite these changes, the prediction is close to the measured loss, suggesting the law captures fundamental scaling relationships that are robust to moderate process variations—consistent with the scaling laws literature's finding that architectural and optimization changes primarily affect multiplicative coefficients rather than functional forms (Kaplan et al., 2020).

Cost multiplier sensitivity (Equation 4 derivation): The $c_{mul}(S)$ formula assumes a specific pruning schedule (cubic, starting at 25%, ending at 75%) and that sparsity can be exploited during training as soon as weights are removed. The paper does not ablate alternative schedules (e.g., linear pruning, later start, earlier end) to test how much $S_{opt}$ contours shift under different cost models. The derivation also assumes perfect sparsity acceleration during training (for the sparse-FLOP case) and zero acceleration (for the dense-FLOP case)—these represent bounds rather than realistic intermediate cases. The dependence of conclusions on these assumptions is partially addressed by reporting both accounting frameworks, but the gap between them is large (50% sparsity becomes optimal at ~2× vs. ~5-70× Chinchilla), meaning the practical recommendation is highly sensitive to the actual training speedup achievable.

Data repetition / single-epoch assumption: All sweep runs operate in the single-epoch regime (no data is repeated), which the paper states is the "ideal training setup" for scaling law formulation. However, the paper's implications about overtraining (e.g., LLaMA 2 7B at ~14× Chinchilla) involve regimes where data IS repeated multiple times. Muennighoff et al. (2023) showed that data repetition changes scaling exponents, so it is unclear whether the fitted law (calibrated on single-epoch data) accurately predicts behavior in the repeated-data regime that characterizes actual overtraining practice. The paper does not include any multi-epoch runs to test this.

n:m sparsity extension (Table 3, Figure 6): The paper extends the scaling law to structured n:m sparsity patterns by refitting only the sparsity parameters $(a_S, b_S, c_S)$ while inheriting the size and data parameters from the unstructured T5 fit. The n:m sweep is substantially smaller than the main sweep—only runs involving the smallest model or the least training steps are executed for 2:4, 1:4, 4:8, and 2:8 patterns. The fitted gains (Table 3) show that 2:4 and 4:8 (50%-equivalent patterns) achieve 1.56× and 1.67× gain respectively—close to but slightly below the unstructured 50% gain of 1.59×. The 1:4 pattern (75%-equivalent) provides only 1.62× gain, substantially below the unstructured 75% gain of 2.16×, and the 2:8 pattern (also 75%-equivalent) provides 1.81×. The authors note these estimates are "likely slightly more noisy due to less data used in fitting the curves" but conclude that "the 75% [n:m] patterns may simply be too stringent to significantly increase capacity beyond their 50% variants." Figure 6 shows a subset of the n:m data and confirms that the general scaling law shape also holds for structured sparsity—the loss-vs-size curves are approximately parallel across patterns, consistent with the multiplicative-sparsity-factor structure.

Critical Assessment

The paper makes three central claims that the experimental section must support:

Claim 1: The sparse scaling law of Equation 3 accurately predicts validation loss for Transformers across model sizes, sparsity levels, and training durations. The evidence for this claim is substantial but has specific limitations. The within-grid fit quality is strong: Figure 3 (ViT) and the T5 fitting data in Figure 1 (Left) show close alignment between predicted and measured loss. The T5-XL extrapolation is the strongest test—predicting loss for a model 6.75× larger than any fitting point—and it succeeds. However, the extrapolation is a single point, not a systematic test across a range of sizes and sparsities. A skeptic could argue that this point happened to land near the prediction by chance, though the paper notes the setup also changed architecture and batch size, making accidental agreement less likely. The ViT law is not extrapolation-tested at all—there is no equivalent "ViT-XL" out-of-grid validation. This is a significant gap because the ViT law is the more extensively fitted one (112 vs. 48 points) and is used for the optimal sparsity analysis in Figure 1 (Right). Without an extrapolation test, we cannot assess whether the ViT law generalizes beyond the 42.47M non-zero parameter maximum of the fitting grid. Furthermore, the sweep grid itself is relatively sparse for nonlinear fitting: 48 points for 7 parameters in the T5 case means approximately 7 points per parameter, which is adequate but not generous. The fitted coefficients (Table 7) likely have substantial uncertainty that is not quantified—no confidence intervals are reported, and the paper does not discuss coefficient stability under bootstrap or cross-validation. The law's functional form is also not ablated against alternatives; the paper builds the law inductively from observations and then validates it on the same data that motivated the form, which provides confirmatory but not falsifying evidence.

Claim 2: Optimal sparsity increases with training duration beyond Chinchilla, and sparse models can match or exceed dense ones of equivalent non-zero parameter count. This claim is derived analytically from the fitted scaling law (Equations 5-6) rather than tested through direct experiment. The paper does NOT, for example, take a fixed compute budget, train models at several sparsities, and verify that the $S_{opt}$ predicted by the law is indeed the one that achieves the lowest loss. Instead, the $S_{opt}$ contours in Figures 1 and 4 are model predictions—they show what the fitted law says should be optimal, not what was experimentally observed to be optimal. The Figure 5 plots are consistent with the claim (showing sparse models outperforming dense ones at the same non-zero parameter count across a range of compute budgets), but they do not test the specific prediction that $S = 0.75$ is optimal at some threshold while $S = 0.50$ is optimal at another. The $S_{opt}$ analysis is therefore a derived implication of the law, and its validity depends entirely on the law's correctness. If the law is misspecified at the tails (e.g., if the multiplicative-sparsity-factor structure breaks down at very long training durations, or if the cost model $c_{mul}(S)$ is wrong), the $S_{opt}$ contours could be misleading. The paper's claim that sparsity "provides a means to further improve model performance" in overtraining regimes is thus better described as predicted by the model rather than experimentally demonstrated. A direct experimental validation—taking one model size, training it at multiple sparsities for, say, 10× Chinchilla duration, and confirming the law's $S_{opt}$ prediction—would substantially strengthen this claim.

Claim 3: Sparsity provides a universal but sub-compression-rate capacity gain (Table 2), with 75% sparsity yielding approximately 2.16× effective parameter-equivalent gain. This claim is the most directly experimentally supported of the three. The gain values in Table 2 are computed from the fitted coefficients, which are in turn derived from the sweep data showing consistent loss reductions from sparsity across all model sizes and training durations. The cross-domain consistency (1.60× vs. 1.59×, 2.17× vs. 2.16×, 2.63× vs. 2.63×) is striking and not forced by the fitting procedure—the ViT and T5 fits are entirely independent, yet produce nearly identical gain ratios. However, the claim of "universality" is supported by only two domains (both Transformer-based, both with massive datasets), and the near-identity of the numbers may partially reflect the specific sparsity levels chosen (0.5, 0.75, 0.875). The effective capacity multiplier depends on the fitted coefficients $a_S$ , $b_S$ , $c_S$ , and $b_N$ , which differ substantially between ViT and T5 (e.g., $a_S = 294$ vs. $16.8$ , $b_S = 0.821$ vs. $0.722$ , $b_N = 0.392$ vs. $0.245$ ). The fact that these different coefficients combine to produce nearly identical $\text{gain}(S)$ values is either a remarkable coincidence or evidence that the gain function is robust to substantial coefficient variation. The latter interpretation would strengthen the universality claim, but the paper doesn't explore this analytically (e.g., by showing that $\text{gain}(S)$ is relatively insensitive to coefficient perturbations).

Weaknesses and missing experiments. Several experiments would have significantly strengthened the paper's claims:

Systematic extrapolation testing. For both ViT and T5, testing the law on models 2×, 4×, and 8× larger than the maximum fitting point (not just a single T5-XL point) would provide a much stronger validation of generalization. The T5-XL point is encouraging but insufficient alone.
Direct $S_{opt}$ experimental validation. Taking a fixed compute budget (e.g., a specific total FLOPs value) and training models at several sparsities to verify that the $S_{opt}$ predicted by the law actually achieves the lowest loss. This would transform $S_{opt}$ from an analytic prediction to an experimentally demonstrated phenomenon.
Ablation of the cost model $c_{mul}(S)$ . Testing alternative pruning schedules (linear, early-start, late-start) to quantify how much $S_{opt}$ contours shift under different training cost assumptions. The large gap between dense-FLOP and sparse-FLOP $S_{opt}$ thresholds (2× vs. 70× for T5 50%) shows that the practical recommendation is highly sensitive to this assumption, yet it is never experimentally varied.
Multi-epoch / data repetition runs. Since the paper's practical conclusions center on the overtraining regime (LLaMA at 14× Chinchilla, largest LMs at >100×), and overtraining inherently involves data repetition, validating that the single-epoch-fitted law holds under repeated data would be important. Muennighoff et al. (2023) showed that data repetition changes scaling exponents, so the law may not transfer cleanly.
Comparisons against stronger baselines. The GMP vs. AC/DC comparison (Table 6) is limited to two model sizes and one training duration. A full sweep comparing GMP against the best alternative method at all sparsities and durations would clarify whether the $\text{gain}(S)$ values are specific to GMP or represent a more fundamental bound. The paper's claim that "the overall shape of our scaling results will remain consistent" while "specific coefficient values can be improved significantly with more extensive per-run tuning" (Section 6, Limitations) suggests that $\text{gain}(S)$ might increase with better pruning methods, but this is untested.
Uncertainty quantification for the fitted law. The paper does not report confidence intervals for fitted coefficients, prediction intervals for loss estimates, or uncertainty bounds for $S_{opt}$ contours. Given the relatively small number of datapoints per parameter (7:1 for T5, 16:1 for ViT), the fitted surface likely has non-trivial uncertainty, especially at extrapolated scales. Practitioners using this law to make deployment decisions would need to know whether the predicted $S_{opt}$ of 75% at some budget might actually be 50% or 87.5% within reasonable confidence.
Additional model families. The two-domain validation (ViT + T5) is good but both are Transformer-based and both use massive datasets. Testing on a convolutional architecture (e.g., ResNet at scale) or a different Transformer variant (decoder-only GPT-style) would test the claimed universality of the multiplicative-sparsity-factor structure.

Where the claims hold conditionally. The paper is explicit about boundary conditions: the law is fitted in the single-epoch, non-bottlenecked regime with no data repetition; the pruning method is GMP with a specific schedule; the architectures are Transformer-based; and the tasks are pretraining loss (not downstream accuracy). The $S_{opt}$ conclusions depend on the cost model—sparsity becomes optimal at much lower training multiples if sparsity can be exploited during training (sparse FLOPs) than if it cannot (dense FLOPs). The $\text{gain}(S)$ values are computed in the infinite-data limit; at finite training budgets, the effective gain is lower because sparse training consumes more FLOPs, reducing the data available for a fixed compute budget. The paper does not overclaim on these points—the Discussion section (Section 6) explicitly notes that "specific coefficient values can be improved significantly" and that extending to different target metrics is "a very interesting topic for future work."

Overall assessment. The experimental evidence convincingly demonstrates that the separable scaling law form (sparsity as a multiplicative capacity modifier, independent of data scaling) is a good empirical description of the observed relationship across two domains and a range of scales. The evidence supports but does not prove the optimal sparsity predictions, which are analytic consequences of the fitted law rather than directly validated phenomena. The evidence strongly supports the sub-compression-rate capacity gain at the tested sparsity levels, with the cross-domain consistency being the most compelling aspect. The main vulnerabilities are: lack of systematic extrapolation testing for ViT, absence of direct $S_{opt}$ experimental validation, sensitivity of conclusions to untested cost-model assumptions, and the fundamentally confirmatory nature of the analysis (the law is built from and tested on the same data, with the single T5-XL point as the only out-of-sample test). Despite these limitations, the paper achieves its stated goal of providing "the first scaling law for characterizing the impact of sparsity on the performance of Transformers trained on massive datasets," and the empirical regularities it documents—parallel loss-vs-size curves across sparsity levels, saturating sparsity benefit, data-independence of the sparsity effect—are robust findings that would likely persist under improved pruning methods and different model scales.

6. Limitations and Trade-offs

The ViT Scaling Law Is Never Extrapolation-Tested, and the T5 Extrapolation Is a Single Point

The assumption or constraint. Scaling laws are most useful when they predict performance beyond the range of the fitted data. The paper fits the ViT law (Equation 3) on a sweep spanning 0.66M to 42.47M non-zero parameters and uses this law to compute optimal sparsity contours (Figure 1, Right) and limit gains (Table 2). However, unlike the T5 law—which is validated on a single T5-XL model approximately 6.75× larger than the maximum fitting point—the ViT law receives no extrapolation testing at all. The paper acknowledges this implicitly by only showing the T5 extrapolation result and not mentioning any ViT equivalent.

The consequence. Without extrapolation validation, we cannot assess whether the ViT law's predictions at scales beyond 42M non-zero parameters are reliable. The optimal sparsity contours in Figure 1 (Right) extend to models with 10^9 non-zero parameters—more than 20× larger than the maximum fitting point—and the S_opt values in that region depend on the law continuing to hold at those scales. If the multiplicative-sparsity-factor structure breaks down for larger models (e.g., if the size exponent b_N shifts, or if the sparsity saturation constant c_S changes at scale), the predicted optimal sparsities could be substantially wrong. Given that the paper's practical recommendations target exactly these larger models (LLaMA 2 7B at ~14× Chinchilla, the largest LMs at >100×), the ViT extrapolation gap is directly relevant to the paper's stated use case.

What evidence exists in the paper. The T5-XL extrapolation point (Figure 1, Left) provides indirect evidence—if the same functional form holds for T5 at 6.75× scale, perhaps it holds for ViT at similar multiples. But this is an argument from analogy across domains, not a direct validation. The ViT and T5 tasks differ substantially (multi-label classification vs. masked language modeling, b_D = 0.890 vs. 0.203), so the structural regularities that produce accurate extrapolation in one domain may not transfer. The paper provides no ViT-specific extrapolation evidence—no "ViT-Large" or "ViT-Huge" experiments outside the fitting grid.

Mitigation status. The paper does not address this gap. The Discussion (Section 6) notes that "our results still have limitations, which we hope will be addressed in future work," but this generic statement does not specifically identify the ViT extrapolation missing validation. A systematic extrapolation test—training ViT models at, say, 2×, 4×, and 8× the maximum fitting size and checking the law's predictions—would be straightforward to add and would substantially strengthen confidence in the ViT-derived S_opt contours.

Difficulty Estimation Cost Is Unaccounted for in the Compute-Optimal Framework

The assumption or constraint. The paper's optimal sparsity framework requires knowing the fitted scaling law coefficients (a_S, b_S, c_S, b_N, a_D, b_D, c) to compute S_opt(N, C). These coefficients are obtained by fitting the law to a large sweep grid: 112 runs for ViT, 48 runs for T5. The cost of executing this sweep—training dozens of models at multiple sizes, sparsities, and training durations—is not included in any of the paper's compute-efficiency calculations. A practitioner wanting to determine S_opt for their own model family, dataset, and pruning recipe would need to replicate a sweep of comparable scale before the law could be applied to their setting. The paper provides no method for cheaply transferring coefficients from one setting to another.

The consequence. The paper's framing—that S_opt provides a principled tool for deciding when to use sparsity—is technically correct but practically incomplete. The cost of acquiring the scaling law (the sweep itself) may exceed the cost savings from using sparsity optimally, especially for smaller-scale deployments where the absolute FLOPs budget is modest. For example, the T5 sweep trains 48 models, some for up to 1M steps at batch size 128, consuming substantial compute. If a team's total training budget for a production model is, say, 100× the cost of the sweep, the sweep overhead is 1% and probably worthwhile. But if the budget is only 5× the sweep cost, the overhead is 20%—a significant fraction. The paper's headline claim that 50% sparsity becomes optimal at <2-3× Chinchilla does not amortize the sweep cost, so the net benefit of the framework (as opposed to just guessing a sparsity level) depends on the ratio of production training cost to sweep cost, which the paper never discusses.

What evidence exists in the paper. The paper reports the total number of runs (112 ViT, 48 T5) and the training durations for each (Table 1), allowing a rough estimate of the sweep cost. The authors do not frame this as a limitation—the sweep is treated as the cost of doing the science, not as a practical barrier to adoption. The Discussion (Section 6) does not mention sweep cost as a limitation. The paper's suggestion that "good pruning performance in less expensive settings should generalize to performance at scale, which will hopefully accelerate research on new sparsification recipes" (Section 6, Practical Consequences) hints at transferability but provides no evidence that coefficients fitted at small scale accurately predict S_opt at production scale.

Mitigation status. The paper does not attempt to reduce the sweep cost or provide coefficient transfer methods. It does not test whether the law fitted on a subset of the grid (e.g., only 2 sparsities, only 2 training durations) would yield similar S_opt predictions as the full grid. It does not test whether coefficients transfer from ViT to T5 (the \text{gain}(S) values coincidentally match, but the raw coefficients differ by orders of magnitude—a_S = 294 vs. 16.8—so direct transfer is not possible). The authors do not suggest a lightweight procedure for practitioners to estimate their own coefficients (e.g., fitting on 3-4 strategically chosen points rather than a full grid). This limitation is unaddressed in the current work.

The Optimal Sparsity Predictions Are Analytic Consequences of the Fitted Law, Not Directly Experimentally Validated

The assumption or constraint. The S_opt analysis in Section 3.3 derives optimal sparsity by differentiating the fitted scaling law (Equation 5) and plotting the resulting contours (Figures 1 Right, 4). This is a model prediction, not an experimental result. The paper does not take a fixed compute budget, train models at multiple sparsities, and verify that the S_opt identified by the law actually achieves the lowest loss in practice. The Figure 5 plots show that sparse models can outperform dense ones across a range of budgets, consistent with S_opt > 0, but they do not test the specific quantitative predictions: that 50% sparsity becomes optimal at exactly <2× Chinchilla for ViT, or that 75% sparsity overtakes 50% at some higher threshold.

The consequence. The paper's central prescriptive claim—"as one trains significantly longer than Chinchilla, more and more sparse models start to become optimal"—is derived from a model that was fitted to the same data that motivated its functional form. This creates a circularity risk: the law was built to capture the observed pattern (parallel lines, saturating sparsity benefit), so it will necessarily produce S_opt predictions consistent with that pattern. If the true relationship between sparsity and loss at very long training durations deviates from the fitted power-law form—for instance, if sparse models hit an optimization wall that dense models don't, or if the sparsity benefit begins to decrease beyond some training threshold—the fitted law (which has no mechanism to represent such effects) would continue to predict increasing S_opt regardless. The single T5-XL extrapolation point provides some reassurance that the law holds at larger scale, but it tests only one sparsity level (75%) at one training duration—it does not test the S_opt ranking (does 75% actually beat 50% at that scale?).

What evidence exists in the paper. All evidence for S_opt is analytic. The derivation in Equations 4-6 is mathematically correct given the fitted law and cost model, but the validity of the conclusions depends entirely on the law's correctness at the specific (N, C) points where different sparsities cross over. The paper does not report any experiment where, for example, a 1.33M non-zero ViT model is trained at 0%, 50%, 75%, and 87.5% sparsity for a compute budget where the law predicts 50% is optimal, and the measured losses confirm that 50% indeed achieves the lowest loss. The Discussion (Section 6) does not identify this as a limitation.

Mitigation status. Unaddressed. The paper treats the fitted law as sufficiently validated (by within-grid fit quality and the single T5-XL extrapolation) to support the S_opt derivation, but the specific claim about optimal sparsity thresholds would be substantially stronger with even one direct experimental test. A practical validation could take the form: for a fixed model size, allocate a total FLOPs budget, train at S = 0, 0.5, 0.75, 0.875 (adjusting training duration to equalize cost via c_{mul}(S)), and verify that the sparsity achieving the lowest measured loss matches the law's S_opt prediction. The absence of this experiment—which would not require any new sweep runs beyond what the paper already executes, only a re-analysis of existing data through the cost-equalization lens—is a significant gap between the paper's analytic claims and its experimental evidence.

The Cost Model `c_{mul}(S)` Is Specific to One Pruning Schedule and Makes Extreme Assumptions About Training Speedup

The assumption or constraint. The optimal sparsity analysis depends critically on c_{mul}(S) (Equation 4), which models how much more expensive sparse training is compared to training a dense model of the target non-zero size. This formula assumes a specific pruning schedule: cubic, starting at 25% of training, ending at 75% of training. It then offers two extreme accounting frameworks: dense FLOPs (sparsity provides zero training speedup, cost = 6N \cdot D/(1-S) as if training the full dense model throughout) and sparse FLOPs (sparsity provides perfect speedup from the moment weights are removed, cost computed via Equation 4). The paper acknowledges the distinction between these two frameworks but treats them as bounds rather than testing intermediate, more realistic scenarios.

The consequence. The practical recommendation is highly sensitive to which cost model is used. Under sparse-FLOP accounting, 50% sparsity becomes optimal for ViT at <2× Chinchilla—a threshold easily exceeded in practice. Under dense-FLOP accounting, the threshold shifts to ~5× for ViT and ~70× for T5. The 35× gap between these estimates for T5 (2× vs. 70×) means that the decision to use sparsity could flip entirely depending on one's assumption about training-time sparsity acceleration. Current hardware and software support for sparse training is limited and rarely achieves the ideal speedup assumed by the sparse-FLOP model, but it is also not zero (sparse matrix multiplication libraries do provide some acceleration). Realistic deployments would fall somewhere between the two extremes, but the paper provides no guidance on how to interpolate or how S_opt behaves in the intermediate regime.

Furthermore, the c_{mul}(S) formula assumes the specific GMP schedule used in the sweeps. A practitioner using a different pruning method (e.g., one-shot pruning with retraining, or AC/DC with alternating dense/sparse phases) would have a different cost profile, and the S_opt contours would shift. The paper does not provide a general method for computing c_{mul}(S) for arbitrary pruning schedules, nor does it test whether S_opt is robust to moderate schedule variations (the pruning hyperparameter ablation in Table 5 only tests final accuracy, not training cost implications).

What evidence exists in the paper. The paper reports both accounting frameworks for all S_opt figures (Figures 1 Right, 4), which is transparent and allows readers to assess the sensitivity. The derivation of Equation 4 is included in Appendix D. The pruning hyperparameter ablation (Table 5, Appendix B) shows that GMP accuracy is robust to start/end/frequency changes, but does not compute how c_{mul}(S)—and thus S_opt—would differ under those alternative schedules. The AC/DC comparison (Table 6) reports final perplexities but not training FLOPs, so the cost-efficiency tradeoff of AC/DC vs. GMP cannot be assessed.

Mitigation status. The paper partially addresses the sensitivity by presenting both extreme cost models, which is better than reporting only one. However, it does not: (1) experimentally measure actual sparse training speedup on real hardware (which would ground the sparse-FLOP assumption), (2) test intermediate cost models (e.g., 50% speedup rather than 0% or 100%), (3) provide a formula for c_{mul}(S) that generalizes to other pruning schedules, or (4) discuss how practitioners should choose between the two cost frameworks for their specific hardware/software stack. The Discussion (Section 6) notes that "sparsity acceleration can be quite complex: current software/hardware may not provide ideal speedups," implicitly acknowledging the issue but without quantifying its impact on the S_opt analysis.

The Law Is Fitted and Validated Only on Pretraining Loss, Not Downstream Task Performance

The assumption or constraint. All experiments measure validation loss on the pretraining task: multi-label classification loss for ViT/JFT-4B and masked language modeling loss for T5/C4. The scaling law is fitted to these losses, and all conclusions—S_opt, \text{gain}(S), optimal sparsity thresholds—are stated in terms of pretraining loss. The paper explicitly justifies this choice: "we deliberately consider the pretraining loss and infinite data setting to assess the effectiveness of sparsity in its most challenging (one essentially needs to fit the data as well as possible) yet also most useful application (all further post-processing would directly benefit from a compressed base model)." The implicit assumption is that improvements in pretraining loss translate proportionally to improvements in downstream task performance after fine-tuning.

The consequence. If the relationship between pretraining loss and downstream accuracy is not monotonic or not uniform across sparsity levels, then the paper's prescriptive claims (S_opt for a given compute budget) may not hold for the metric that practitioners actually care about. For example, it is possible that sparse models, while achieving lower pretraining loss than equivalently-sized dense models, exhibit worse transfer learning—perhaps because the zero pattern, which helps memorize the pretraining distribution, provides less generalizable features. Or conversely, sparsity might act as an implicit regularizer that improves downstream performance beyond what pretraining loss would predict. The paper provides no evidence either way. The claim that "all further post-processing would directly benefit from a compressed base model" is plausible but untested—downstream fine-tuning could interact with sparsity in ways that the pretraining loss does not capture.

What evidence exists in the paper. None. The paper does not fine-tune any of the sparse models on downstream tasks (e.g., ImageNet classification for ViT, GLUE/SuperGLUE for T5) and does not report any downstream metrics. The discussion of practical applications (Section 6) focuses entirely on pretraining loss and inference cost. The authors acknowledge in the Limitations that "sparsity rates can probably be improved significantly when pruning is performed directly for more specialized applications that require only a subset of the base model's full capabilities," which suggests that the pretraining-loss setting may be pessimistic for downstream use—sparsity might look even better when evaluated on specific tasks. But this is speculation without experimental support, and the reverse could also be true.

Mitigation status. The paper does not address this gap beyond the brief acknowledgment quoted above. The choice to focus on pretraining loss is methodologically defensible (it provides the cleanest signal for scaling law fitting without the confounding factors of task-specific fine-tuning hyperparameters), but the absence of any downstream validation limits the practical applicability of the S_opt recommendations. A moderate extension—taking a few points from the sweep, fine-tuning them on a standard benchmark, and checking whether pretraining loss rank-ordering is preserved—would have substantially strengthened the claim that the scaling law is relevant for real-world deployment decisions.

The Sweep Uses Only Transformer Architectures and Two Specific Tasks, Limiting Claims of Generality

The assumption or constraint. All experiments use Transformer-based architectures: Vision Transformers (encoder-only with patch embedding) and T5 (encoder-decoder). Both are trained on standard pretraining objectives (multi-label classification, masked language modeling) using massive, single-epoch datasets. The paper claims cross-domain validation because it tests both vision and language, but both domains share the same underlying architecture family (self-attention with feed-forward layers), the same optimization approach (AdaFactor with minor modifications), and the same pruning method (GMP with cubic schedule). The paper does not test convolutional architectures, decoder-only language models (GPT-style), mixture-of-experts models, or models trained with data repetition. The Discussion (Section 6) states: "We think that our findings can be interpreted as providing practical motivation for further developing sparsity support," implying generality beyond the tested settings.

The consequence. The paper's key structural finding—that sparsity acts as a multiplicative capacity modifier independent of data scaling—might be specific to the Transformer architecture's particular scaling properties. Transformers are known for clean power-law scaling across many orders of magnitude; other architectures (CNNs, RNNs, MLP-Mixers) may exhibit different relationships between sparsity and capacity. Even within the Transformer family, decoder-only models (which now dominate language modeling practice) have different parameter-sharing patterns and depth-to-width ratios than encoder-decoder T5, potentially affecting how sparsity interacts with capacity. The cross-domain consistency of \text{gain}(S) values (Table 2) is striking but might be specific to the particular combination of Transformer architectures and massive single-epoch datasets tested. A practitioner using a convolutional network on a medium-scale dataset, or a decoder-only LLM trained with data repetition, cannot confidently apply the paper's S_opt thresholds or \text{gain}(S) values.

What evidence exists in the paper. The paper tests two architectures within the Transformer family. The ViT and T5 coefficients differ substantially (Table 7), indicating that the raw scaling parameters are task- and architecture-specific, even though the derived \text{gain}(S) values coincidentally match. This suggests that the functional form of the law—not the specific coefficient values—may be the transferable insight. However, with only two architectural variants tested, there is no basis for claiming that the functional form itself is universal. The n:m sparsity extension (Section 4.1) tests structured sparsity patterns within the same T5 architecture, finding that the same law form fits but with different sparsity coefficients—this demonstrates robustness to sparsity pattern variation but not architectural variation.

Mitigation status. The paper acknowledges the scope limitation implicitly by choosing to study "Transformer-family models" and by framing contributions as applying to "foundation models" (which are predominantly Transformers). The Discussion (Section 6) does not identify the narrow architectural scope as a limitation, instead focusing on pruning method improvements and extension to different target metrics. The authors do not suggest testing on alternative architectures as future work. Given that the paper's goal is to establish a first scaling law for sparse foundation models, limiting scope to Transformers is reasonable for an initial study—but the generality claims should be interpreted as applying to Transformer-based models trained on massive single-epoch datasets until validated more broadly.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a foundational methodological contribution rather than a paradigm shift: it provides the field with the first predictive, quantitative framework for reasoning about sparsity in the foundation model regime. Before this work, sparsity was evaluated heuristically—prune a model, measure accuracy, compare against an often-incomparable baseline—with no way to forecast whether sparsity would help or hurt at a given scale and training budget. The paper replaces this with a principled scaling law that treats sparsity as a first-class variable alongside parameter count and data volume, enabling the same kind of rational resource allocation that Chinchilla (Hoffmann et al., 2022) enabled for dense model sizing. The magnitude of the shift is comparable to the introduction of dense scaling laws themselves: it does not invent a new technique, but it transforms an empirical art into a predictive science.

The most important conceptual reframing is the decomposition of sparsity's effect into two separable components: a multiplicative capacity modifier (the (a_S(1-S)^{b_S} + c_S) term that multiplies the size-scaling component) and a training-cost penalty (the c_mul(S) multiplier that reduces effective data for a fixed compute budget). This decomposition resolves what was previously a muddled debate: is sparsity "good" or "bad"? The answer is conditional on the training regime. Near the Chinchilla optimum, the training-cost penalty dominates because data is still the binding constraint—sparsity looks like a bad trade. Far beyond Chinchilla, where dense models saturate and capacity becomes the binding constraint, the capacity benefit dominates—sparsity looks like a good trade. This explains why prior work produced seemingly contradictory results: Rae et al. (2021) and Cerebras (2022) found sparse models winning in overtrained regimes (consistent with the law), while standard pruning benchmarks on saturated tasks also showed sparsity benefits (because overfitting provided a natural ceiling that mimicked the capacity-bound regime). The paper does not overturn these findings but provides the unified framework that makes them all predictable from a single equation.

The methodological contribution around fair evaluation (Section 2) is likely to have lasting impact independent of the specific scaling law coefficients. By identifying three confounds—training duration (sparse models getting extra data), architectural efficiency (dense baselines not being competitive at small scales), and compute accounting (sparse training consuming more FLOPs)—the paper sets a new standard for what constitutes a valid comparison between sparse and dense foundation models. Future sparsity papers that omit equal-nonzero-parameter dense baselines, fail to equalize total training data, or ignore the FLOPs cost of the sparsification process itself will now be judged against this standard. This is less flashy than a new scaling law but arguably more important for the health of the field: it prevents the kind of confounded benchmarks that have periodically plagued the pruning literature.

The paper also redirects research attention in a specific way: away from optimizing pruning algorithms for marginal accuracy gains at fixed sparsity, and toward understanding the fundamental capacity-efficiency tradeoff that sparsity represents. The finding that 75% sparsity yields approximately 2.16× effective capacity gain (not 4×, the compression rate) across both vision and language (Table 2) suggests there is a hard efficiency ceiling determined by something more fundamental than pruning algorithm choice—perhaps the information-theoretic limits of encoding knowledge in zero patterns, or the optimization geometry of sparse networks. If this ceiling is real (and the cross-domain consistency argues it might be), then the most impactful research is not on incrementally better pruning schedules but on raising the ceiling itself—finding ways to make zero patterns encode information more efficiently, perhaps through learned sparsity patterns, dynamic sparse connectivity, or fundamentally different compression schemes.

Finally, the paper makes over-training more attractive as a research investment. By showing that sparsity's benefits are largest precisely in the regime where dense models begin to saturate, the paper provides a constructive answer to the question "what do we do when more data stops helping?" The standard response has been "train a bigger model"—but that increases inference cost. Sparsity offers an alternative: train an even bigger model, prune it, and get higher capacity at the same inference cost as the smaller dense model. This reframes sparsity from a compression-afterthought to a capacity-scaling strategy—a way to keep improving model quality in the overtraining regime without paying the inference penalty of larger dense models. In an era where LLaMA-class models are routinely trained at 10-100× Chinchilla, this reframing is immediately actionable.

Follow-Up Research This Work Enables

Direct experimental validation of S_opt predictions. The paper's optimal sparsity contours (Figures 1 Right, 4) are analytic consequences of the fitted scaling law—they are model predictions, not experimentally verified phenomena. A critical follow-up would take a single model size (e.g., 10M non-zero parameters for ViT), fix a total FLOPs budget, train models at S = 0, 0.5, 0.75, 0.875 (adjusting training steps to equalize cost via c_mul(S) for each sparsity), and verify that the sparsity achieving the lowest measured loss matches the law's S_opt prediction. The test should cover multiple budget levels: one near Chinchilla (where S_opt should be 0 or low), one at 5× Chinchilla (where 50% should become optimal under sparse FLOPs), and one at 20× Chinchilla (where 75% might be optimal). This experiment would transform S_opt from an analytic derivation to an experimentally demonstrated phenomenon. If the law's predictions hold, it strongly validates the scaling law framework; if they fail—for instance, if 75% sparsity never actually beats 50% at the predicted threshold—it would reveal a structural misspecification in the law that the current within-grid fitting cannot detect.

Systematic ViT extrapolation testing with multiple scale jumps. The paper validates the T5 law on a single T5-XL point (~6.75× beyond the fitting maximum) but performs no extrapolation testing for the ViT law, which is the more extensively fitted one (112 runs) and the basis for the optimal sparsity analysis in Figure 1 (Right). A direct follow-up would train ViT models at 2×, 4×, and 8× the maximum fitting non-zero parameter count (42.5M → 85M, 170M, 340M) at multiple sparsity levels and compare measured losses against the fitted law's predictions. Critically, these extrapolation models should be trained for at least one training duration that was in the fitting grid (e.g., 220K steps) to isolate the size extrapolation from data extrapolation. If the law's predictions hold across all scale jumps, it provides strong evidence that the multiplicative-sparsity-factor structure is scale-invariant for ViT. If predictions degrade systematically with scale (e.g., the law consistently underpredicts loss for larger sparse models), it suggests that the capacity benefit of sparsity saturates more aggressively at larger scales than the power-law form captures, requiring a law modification (perhaps a c_S that depends on N).

Characterize the sparsity efficiency ceiling with better pruning methods. The paper deliberately uses simple GMP to maximize robustness across the sweep, and the gain values in Table 2 (~2.16× at 75% sparsity) represent what GMP achieves. A natural extension is to refit the scaling law using the best available pruning method—perhaps a modern second-order approach (SparseGPT-style, Frantar & Alistarh, 2023, but with retraining), or iterative AC/DC (Peste et al., 2021) with tuned hyperparameters—and measure whether the \text{gain}(S) values increase. The specific hypothesis to test: does a stronger pruning method raise the sparsity efficiency (gain per unit compression), or does it primarily shift the irreducible floor c_S? If \text{gain}(0.75) increases from 2.16× to, say, 2.8× under a better method, it suggests the efficiency ceiling is partly algorithmic and that better pruning is a high-impact research direction. If it stays near 2.16× regardless of method, it strengthens the case that the ceiling is fundamental to how weight sparsity encodes information, and research should pivot toward alternative efficiency strategies (structured sparsity patterns, quantization, pruning during architecture design rather than post-hoc). The experiment requires a moderate sweep (perhaps 2-3 sizes, 2-3 sparsities, 2 training durations) with the improved method, fitting the sparsity coefficients, and comparing \text{gain}(S) values against Table 2.

Test the law under data repetition (multi-epoch training). The paper fits the scaling law in the single-epoch regime (no data repetition), but its practical conclusions target the overtraining regime where data IS repeated multiple times—LLaMA 2 7B at ~14× Chinchilla necessarily repeats data. Muennighoff et al. (2023) showed that data repetition changes scaling exponents for dense models: loss scales less favorably with repeated data than with fresh data. An important open question is whether sparsity interacts with data repetition: does the independence of sparsity and data scaling (Observation 3) hold when data is repeated, or does sparsity change the efficiency of re-using examples? The experiment would take a fixed total data budget (e.g., 1 billion tokens), train dense and sparse models at different repetition factors (1 epoch, 2 epochs, 4 epochs, 8 epochs on a smaller dataset), and test whether the sparsity benefit remains multiplicative and data-independent. If the sparsity factor multiplies the capacity term identically regardless of whether data is fresh or repeated, the current law extends directly to the overtraining regime. If sparsity provides larger (or smaller) relative benefit under repetition—perhaps because sparse models have higher capacity and thus benefit more from fresh data than from repeated examples—the law needs a modification for the multi-epoch case. This experiment connects the paper's framework to the practical reality of how most large models are actually trained.

Combine sparsity with quantization in a joint scaling law. The paper studies weight sparsity alone, but notes that sparsity and quantization are "complementary techniques that may be stacked for compound gains" (Section 5). A natural extension is to develop a joint scaling law L(S, Q, N, D) where Q represents quantization level (e.g., bits per weight). The paper's framework suggests a hypothesis: both sparsity and quantization likely act as multiplicative capacity modifiers with saturating benefits, but they may interact—a 4-bit model with 75% sparsity might achieve different effective capacity than the product of their individual gains. The experiment would require a sweep over sparsity levels, bit-widths (e.g., 16, 8, 4, 2 bits), model sizes, and training durations—a larger grid than the current paper but feasible if the sweep is designed efficiently (perhaps using the insight from Section 4.1 that refitting only the sparsity/quantization coefficients is sufficient once the base scaling is known). The practical value would be enormous: a single joint law telling practitioners "for inference target N non-zero parameters at Q bits, use sparsity S" would effectively automate the model compression design process.

Apply the fair-evaluation framework to re-assess published sparse LLM results. The paper's critique in Section 2 and the diagnostic analysis of Cerebras (2022) and Rae et al. (2021) in Section 6 demonstrate a methodology: use the fitted scaling law to decompose reported sparse model gains into (a) the genuine capacity benefit from sparsity and (b) the benefit from additional training data or compute that the sparse model consumed. A valuable meta-analysis would apply this decomposition systematically to the sparse LLM literature: for each paper reporting a sparse model that beats a dense baseline, compute what the scaling law predicts the sparse model would achieve if it had the same non-zero parameters, same data, and same FLOPs as the dense baseline. This would quantify how much of the field's claimed sparsity gains are attributable to sparsity per se versus expanded training budgets. If most gains are explained by the latter (which the Cerebras analysis suggests), it would shift the conversation from "sparse models beat dense models" to "sparsity lets us trade off training compute for inference efficiency, with a predictable exchange rate" — a more precise and actionable framing.

Practical Applications and Downstream Use Cases

Optimizing sparsity levels for overtrained production LLMs. Current production language models (LLaMA 2 7B at ~14× Chinchilla, the largest LMs at >100× Chinchilla) are trained far beyond the dense compute-optimal point. The paper's S_opt analysis predicts that at these training multiples, substantial sparsity is already optimal: 50% sparsity at <2-3× Chinchilla under sparse-FLOP accounting, meaning today's models would likely benefit from being trained as larger sparse models and then pruned to their target inference size. Concretely: instead of training a 7B-parameter LLaMA model for 2T tokens (which is ~14× Chinchilla), a team could train a 14B-parameter model, prune it to 50% sparsity (yielding 7B non-zero parameters), and—based on the 1.60× capacity gain in Table 2—achieve loss equivalent to a 7B × 1.60 = 11.2B-parameter dense model at the same inference cost. The scaling law provides the tool to quantitatively evaluate this tradeoff before committing to the larger training run. The practical barrier is the training cost: training a 14B model costs more than training a 7B model, and the c_mul(S) analysis shows that even with ideal sparse training acceleration, the 50% sparse 14B → 7B model consumes more total FLOPs than the dense 7B baseline. Whether the 1.60× effective capacity gain justifies the increased training cost depends on the relative value of inference efficiency versus training compute in the specific deployment—a calculation the paper's framework now enables but that the paper itself does not perform for a concrete production scenario.

Inference-cost reduction for vision models deployed at scale. For organizations serving vision models at high query volumes (e.g., image classification or object detection APIs), inference FLOPs directly determine serving costs. The paper's finding that a 75% sparse ViT achieves the same loss as a ~2.17× larger dense model (Table 2) translates to a ~2.17× reduction in inference cost for equivalent accuracy—or, keeping the same inference budget, a model that achieves lower loss. For a deployment processing 1 billion images per day, the cost difference between a 100M-parameter dense model and a 100M-nonzero-parameter 75% sparse model (which matches a 217M dense model in accuracy) could be millions of dollars annually in cloud compute. The practical path to realizing this benefit requires: (1) training the larger dense model (400M parameters) for sufficient data that the sparse version reaches the capacity-limited regime where Table 2 gains apply, (2) pruning using GMP or a similar robust method, and (3) deploying on hardware with sparsity support (e.g., NVIDIA GPUs with 2:4 sparsity, though the paper's n:m results in Table 3 show lower gains for structured patterns—1.62× for 1:4 vs. 2.16× for unstructured 75%, meaning the practical gain depends on hardware sparsity support matching the pruning pattern).

Guiding architecture decisions in model family design. When designing a family of models at different size tiers (e.g., a "small," "medium," and "large" variant of a foundation model), the S_opt framework provides a principled way to decide which tiers should be sparse and at what levels. Without the framework, the default is to make all tiers dense and scale width/depth uniformly. With the framework, a team can evaluate: for the "small" tier (targeting edge deployment with strict inference constraints), should we train a dense model or a larger sparse model pruned to the target size? The answer depends on how much training compute is allocated to that tier relative to its size. If the small tier is trained modestly (near Chinchilla), the law predicts that density is optimal or nearly optimal—sparsity's training-cost penalty outweighs its capacity benefit. If the small tier is overtrained (say, 10× Chinchilla, possible if training data is abundant and the goal is to push small-model quality as high as possible), the law predicts that 50-75% sparsity becomes optimal, meaning the small tier should be trained as a larger model and pruned. The framework thus rationalizes what might otherwise be ad-hoc design choices, and the specific coefficients from Table 7 (or refitted to the team's own model family) provide the quantitative thresholds.

When to Prefer This Method

The paper does not position sparsity against named alternative efficiency techniques (quantization, distillation, mixture-of-experts) with explicit tradeoff curves—it acknowledges these as "complementary techniques that may be stacked for compound gains" (Section 5) but provides no joint optimization framework. The paper's contribution is a scaling law for sparsity specifically, and the "method" it provides is the predictive framework for deciding when and at what level to apply sparsity, not sparsity itself as a method that competes with named alternatives. The decision rule is therefore about whether to sparsify under what training conditions, rather than "sparsity vs. alternative X."

The paper does articulate a clear tradeoff internal to its own framework: the choice between training a dense model of size N and training a larger model pruned to sparsity S with N non-zeros. The decision rule emerging from the work is:

Prefer sparsity (train a larger model and prune to N non-zeros) when:
- The training budget substantially exceeds the Chinchilla optimum (roughly >2-3× Chinchilla for 50% sparsity under sparse-FLOP accounting, >5-70× under dense-FLOP accounting, per Figures 1/4). This is the regime where capacity bottlenecks dominate data bottlenecks, and sparsity's capacity benefit outweighs its training-cost penalty.
- Inference efficiency (FLOPs, memory, latency at deployment) is the binding constraint, and you are willing to spend additional training FLOPs to reduce inference cost. The 2.17× effective capacity gain at 75% sparsity (Table 2) quantifies the inference savings per unit of training investment.
- The model family architecture scales well with width (so the larger pre-pruning model is genuinely higher-capacity) and the pruning method is robust at scale (GMP was shown to be, in Tables 5-6).
- Training-time sparsity acceleration is available (hardware/software supports sparse matrix operations), making the sparse-FLOP cost model closer to reality and reducing the training-cost penalty of sparsity.
Prefer dense training (train a dense model of size N directly) when:
- The training budget is near or below the Chinchilla optimum, where data is the binding constraint and sparsity's training-cost penalty dominates any capacity benefit (S = 0 is optimal or near-optimal in this regime).
- The target sparsity level exceeds the sweet spot where marginal gains become small—Table 2 shows diminishing returns beyond 75% (2.17× → 2.63× for doubling compression from 4× to 8×), and the additional training cost may not justify the incremental inference savings.
- Hardware sparsity support is unavailable, so inference acceleration cannot be realized even if the model is sparse—the 2.17× capacity gain exists mathematically (the model achieves the loss of a larger dense model) but provides no wall-clock benefit if the sparse model runs at dense-model speed.
- The existing ecosystem assumes dense checkpoints (fine-tuning recipes, inference servers, model hubs), and the overhead of maintaining sparse-format artifacts outweighs the efficiency gains.

These rules are conditional on the specific coefficients in Table 7 and the GMP pruning recipe—refitting the law for a different model family, dataset, or pruning method would shift the thresholds, though the paper argues (without direct evidence) that the qualitative shape should persist.