ArXiv: 2309.04564

🎯 Pitch

Models pretrained on only 30–50% of the data can match or beat full-data performance, but only if you keep the moderately easy examples—the simplest ones actually hurt. Incredibly, a basic perplexity score from a reference model outperforms far more expensive quality estimators, and the gains hold across model sizes up to 1.5B parameters.

1. Executive Summary

This work analyzes whether scalable, metric-based data pruning can improve LLM pretraining by systematically comparing three quality estimators—perplexity, Error L2-Norm (EL2N) (a measure of early-training loss that identifies samples the model learns quickly), and memorization factor (a measure of how much the reference model can reproduce a sequence verbatim from a prefix)—on CommonCrawl data using models from 124M to 1.5B parameters. The central finding is that the simple technique of ranking examples by their perplexity from a large reference model and retaining the middle of the score distribution outperforms the no-pruning baseline and all more computationally expensive methods, achieving a ~1% improvement in test set perplexity using only 50% of the original training data and a ~1.5% improvement at the 1.5B scale. Critically, the paper establishes that performance degrades when retaining the lowest-scoring (“easiest”) examples—regardless of the metric—and that reduced-data gains are most pronounced when the reference model used to compute pruning scores is itself large and trained on clean data, while the hardest-bin problems remain essentially unsolvable regardless of the pruning budget.

2. Context and Motivation

The Core Problem: We Don't Know How to Measure Data Quality at Scale

The fundamental question this paper tackles is deceptively simple: given a massive web-scraped pretraining corpus, can we systematically identify which individual training examples are "high quality" and which are "low quality" for the purposes of language model pretraining? Despite the enormous importance of this question—pretraining data is the single most expensive and influential ingredient in building LLMs—the field has no established, rigorous answer.

This gap is striking. The standard narrative in machine learning is that more data leads to better performance, a belief strongly reinforced by scaling laws work (Kaplan et al., 2020) showing predictable improvements in loss as data quantity increases. This narrative has driven the creation of ever-larger web-scraped datasets: C4 (Raffel et al., 2020), RefinedWeb (Penedo et al., 2023), The Pile (Gao et al., 2021). These datasets are compiled by crawling the internet, leading to what the paper diplomatically calls "a substantial portion of the text being noisy and of low quality" (Section 1). The underlying reality is that raw web text contains machine-generated spam, pornographic content, boilerplate legal disclaimers, broken markup, repetitive filler, and countless other forms of degradation that provide little useful signal for language learning (Dodge et al., 2021; Kreutzer et al., 2022; Luccioni & Viviano, 2021).

Yet the way we currently handle this problem is surprisingly primitive.

Where Existing Approaches Fall Short

Rule-based heuristics are the dominant paradigm—and they're not enough. The paper meticulously documents the standard filtering practices that practitioners rely on (Section 1, Section 5.1): removing documents with repetitive text (Zhang et al., 2022; Raffel et al., 2020), filtering out sequences containing special characters or non-English text (Wenzek et al., 2020), excluding data from a manually curated list of "blocklist" websites (Dodge et al., 2021; Rae et al., 2022), and applying length thresholds to documents. These are all hand-crafted, binary rules—a document either passes the filter or it doesn't, with no gradation of quality beyond these coarse checks.

The paper identifies several specific failures of this approach:

  • Controversial effectiveness. The literature is genuinely split on whether rule-based filtering even helps. Some works report improvements in language modeling from applying such filters (Penedo et al., 2023; Raffel et al., 2020), while others find no meaningful benefit (Black et al., 2022; Biderman et al., 2023b). When the community cannot agree on whether a technique works at all, it suggests the underlying heuristics are not capturing what actually matters for learning.

  • Unintended consequences. Rule-based filters can produce harmful side effects precisely because of their simplicity. Dodge et al. (2021) demonstrated that removing blocklisted words from the C4 dataset disproportionately eliminated text from and about minority individuals. A heuristic designed to clean data inadvertently introduced systematic representational bias—exactly the kind of failure mode that a more nuanced quality measure might avoid.

  • No notion of individual example quality. This is the paper's key critique: rule-based heuristics "are not a substitute for a measure of 'quality' for individual training examples" (Section 1). They can catch egregious cases (a document that is 90% repeated text) but provide no signal about whether a syntactically well-formed document actually contributes to the model's learning. Is a boilerplate terms-of-service agreement "high quality" because it's grammatically correct? Is a technical specification document with dense terminology "low quality" because it has high perplexity? Rule-based filters have no way to discriminate these cases.

Existing metric-based approaches don't address the pretraining setting. The paper acknowledges that some work has gone beyond rule-based heuristics to actually score data quality using model signals, but these efforts have been almost entirely confined to the fine-tuning stage of LLM training (Section 5.2). This is understandable—fine-tuning datasets are orders of magnitude smaller than pretraining corpora, making them computationally tractable to score and prune. Attendu & Corbeil (2023) performed dynamic pruning during fine-tuning using EL2N scores. Cao et al. (2023) selected high-quality instruction data for fine-tuning. But the pretraining stage, where data volume is largest and the potential impact of pruning is greatest, has remained largely unexplored because of the prohibitive computational cost of scoring billions of tokens.

The few works that do address pretraining data selection (Section 5.2) either rely on hand-picking "known good" corpora—Xie et al. (2023a) and Wenzek et al. (2020) use reference corpora like Wikipedia as quality benchmarks—or focus on specific sub-problems like semantic deduplication (Abbas et al., 2023; SemDeDup). No prior work has provided a systematic, head-to-head comparison of different quality metrics applied to the same pretraining corpus at the same scale.

Where perplexity has been used, it's been applied incorrectly. Several prior works have used perplexity to filter datasets, but with a specific—and, this paper argues, wrong—assumption (Section 5.2). Laurençon et al. (2023) and Muennighoff et al. (2023) filtered out high-perplexity samples from their corpora, framing such samples as "unnatural language" that is harmful for performance relative to their reference domain (typically Wikipedia). The implicit assumption is that low perplexity = good quality. This paper directly challenges that assumption—indeed, one of its central findings is that the lowest-perplexity samples (the "easiest" data, from the model's perspective) are the least useful for training. Prior work took an intuitive but untested position on what constitutes quality data; this paper does the systematic comparison needed to actually answer the question.

The computer vision community has the right idea, but the NLP setting is fundamentally different. The most relevant intellectual precedent comes from computer vision, where data pruning using model-based scoring has been studied extensively in supervised settings (Section 5.3). Sorscher et al. (2023) demonstrated that with abundant data, training only on the "hardest" examples (as scored by a teacher model's margin) yields better performance, while when data is scarce, training on the "easiest" examples is preferable. Paul et al. (2023) introduced EL2N scores as an early-training signal for identifying important samples. But the paper emphasizes a crucial distinction: the CV setting is supervised—there are well-defined class labels and loss signals that directly capture whether a model is learning the task. Pretraining an LLM is unsupervised—the objective is next-token prediction, and there are no "mistakes" in the traditional sense, just sequences that are more or less probable under the model's learned distribution. Translating quality metrics from the supervised to the unsupervised setting is non-trivial, and the paper notes that this difference makes direct comparison difficult.

Why This Problem Matters

The paper's framing of the problem's importance rests on several pillars:

1. The scale of waste in current pretraining. If the paper's findings hold—that 50% or even 70% of pretraining data can be removed with no performance degradation—then the status quo of training on ever-larger, unfiltered web corpora represents a massive computational inefficiency. Training a single large language model can cost millions of dollars in compute; reducing the training data by half could roughly halve training time and cost, or alternatively, allow training longer on the remaining high-quality data for better final performance at the same budget.

2. The quality of models depends fundamentally on data quality. The paper positions itself as part of a growing recognition that data quality, not just data quantity, is the primary driver of model capabilities. This is implicit in the success of carefully curated datasets and explicit in works like LIMA (Zhou et al., 2023), which showed that a small amount of high-quality fine-tuning data can outperform much larger but noisier datasets. Extending this insight to the pretraining stage—where data curation has been dominated by crude heuristics—represents an important frontier.

3. A principled alternative to the "more data" default. The paper pushes back against the reflexive assumption that scaling up data collection is always the right approach. As Mitchell et al. (2023) argued, the field lacks established best practices for measuring data quality. Without such practices, the default response to any model deficiency is "collect more data" rather than "better understand the data we already have." This paper provides an empirical framework for breaking that cycle.

4. Foundation for future data-centric research. By establishing which metrics work (perplexity), which don't (memorization), and how selection criteria interact with metric choice (the "middle" subset is optimal across metrics), the paper creates a foundation for future work on automated corpus curation. The finding that reference model quality matters—larger models trained on cleaner data produce better pruning signals—suggests a scalable path forward where improved data curation is bootstrapped from improved models.

How This Paper Positions Itself

The paper explicitly frames its contribution as a systematic comparison rather than the proposal of a new pruning method (Section 1). It identifies three metrics that "all rely solely on model outputs and do not require a preselected high-quality dataset" (Section 1)—a deliberate choice to avoid the circularity of using a "good" dataset to identify "good" data. The metrics span a spectrum from simple (perplexity) to complex (EL2N, memorization), allowing the paper to test whether sophistication buys meaningful improvement.

The key intellectual move is to treat data pruning as a ranking and selection problem rather than a filtering problem. Traditional heuristics produce binary decisions (include/exclude), but the paper's metrics produce continuous scores that can be used to select arbitrary percentiles of the distribution. This enables the exploration of which subset of the distribution to retain—bottom, middle, or top—which turns out to be the critical variable. The paper's most counterintuitive finding—that the "easiest" examples (lowest perplexity, lowest EL2N, highest memorization) are the least useful for training—cannot be discovered by any binary filtering approach. It requires the continuous ranking framework that the paper introduces.

The paper also positions itself as a static pruning study (Section 2), where all pruning decisions are made once before training begins. This contrasts with dynamic or adaptive pruning approaches where data is re-evaluated during training (Fayyaz et al., 2022; Park et al., 2022). The choice is pragmatic: dynamic pruning would require continuously scoring the entire dataset throughout training, which is computationally infeasible at pretraining scale. By demonstrating that even static pruning can yield substantial improvements, the paper establishes a strong baseline that future dynamic approaches would need to beat.

Finally, the paper's scope is deliberately narrow in some dimensions and broad in others. It focuses exclusively on language model pretraining (not fine-tuning, not vision), on a single dataset source (CommonCrawl), and on three specific metrics. But within this scope, it explores a wide range of experimental axes: seven different reference model configurations (varying size, training data, and training steps), four pruning ratios (10, 30, 50, 70% retained), three selection criteria (bottom, middle, top), and two trained model scales (124M and 1.5B parameters). This combinatorial thoroughness is the paper's methodological contribution—by holding the experimental framework constant and varying one axis at a time, it can make credible causal claims about what drives pruning effectiveness that would be impossible in a less systematic study.

3. Technical Approach

3.1 Reader Orientation

The "system" in this paper is not a model architecture or a training algorithm, but rather a systematic evaluation pipeline for comparing data pruning strategies in LLM pretraining. It takes a massive web-scraped corpus, scores every training example according to one of several quality metrics computed from a reference model, selects a subset of the data by retaining specific percentiles of the score distribution, and then trains a new model from scratch on this pruned dataset to measure whether the pruning helped or hurt. The core problem it solves is: how do we identify which individual training examples are worth keeping, without access to human quality labels, in a way that actually improves model performance rather than degrading it? The "shape" of the solution is a ranking-and-thresholding framework where the choice of metric, the choice of which portion of the ranking to retain (bottom/middle/top), and the choice of reference model all interact to determine whether pruning succeeds or fails.

3.2 Big-Picture Architecture (Diagram in Words)

The pipeline has five major stages, executed sequentially:

  1. Tokenization and Sequence Assembly. The raw text corpus is tokenized, split into fixed-length sequences matching the model's context window, and organized into a flat collection of training instances. Each instance is a self-contained unit for both scoring and training.

  2. Reference Model Scoring. A separate, pre-existing "reference model" (which may differ in size, training data, and training duration from the model that will eventually be trained) processes every training instance and assigns it a scalar quality score. Three alternative scoring functions are compared: perplexity, Error L2-Norm (EL2N), and memorization factor.

  3. Score Distribution Analysis and Subset Selection. For a given scoring metric, the pipeline computes the full distribution of scores across the entire training corpus. It then selects a subset by applying a percentile-based criterion: retain either the bottom X% of scores, the middle X% (centered on the median), or the top X% of scores. The percentage X is swept across 10%, 30%, 50%, and 70%.

  4. Model Training from Scratch. A new model (the "trained model" or "student model") is initialized randomly and trained exclusively on the pruned subset using a standard autoregressive language modeling objective. The trained model may have a different parameter count than the reference model.

  5. Evaluation. The trained model's performance is measured via perplexity on a held-out test set from the same distribution as the training data, and (for a subset of configurations) via fine-tuning on GLUE classification tasks.

Information flows strictly forward through these stages: raw text → tokenized sequences → scored sequences → filtered subset → trained model → evaluation metrics. The only feedback loop is conceptual—results from one pruning configuration inform which configurations are worth exploring at larger scale.

3.3 Roadmap for the Deep Dive

  • First, the formal pruning framework (Equations 1–3), which defines the mathematical objective and the notation that unifies all three pruning methods under a common selection formalism. This is the conceptual backbone—understanding it is prerequisite to understanding why "middle" vs. "bottom" selection matters.

  • Second, the three pruning metrics in full detail—perplexity, EL2N, and memorization—including how each is computed, what property of the data it captures, and what design decisions went into adapting it for the unsupervised pretraining setting.

  • Third, the reference model ablation space, which is the most experimentally rich component: seven reference model configurations varying size, training data, and training duration, each producing a different scoring distribution whose downstream effects are measured.

  • Fourth, the subset selection logic and the "bottom / middle / top" trichotomy, which is the paper's key methodological innovation—no prior work systematically compares which portion of the score distribution is optimal, and this choice turns out to dominate the choice of metric.

  • Fifth, the trained model configurations and training hyperparameters, including the scaling experiments from 124M to 1.5B parameters.

  • Sixth, the evaluation protocol, covering both perplexity-based and downstream task evaluation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical ablation study whose core idea is that data pruning metrics must be evaluated not just by which metric is used, but by which portion of the metric's score distribution is retained for training—and that the optimal choice (middle subset, avoiding both the "easiest" and "hardest" extremes) is both counterintuitive and consistent across metrics.

Formal Pruning Framework

The paper defines data pruning as a subset selection problem operating on sequences of fixed length. The framework has three equations that together specify what pruning does and what success looks like.

Step 1: Dataset preparation. The raw corpus $D$ is first tokenized and all documents are appended with a special <eod> (end-of-document) token. Documents are then concatenated and split into $n$ sequences $z_i$, each of fixed length $t$ equal to the model's context length (2048 tokens throughout the paper):

D={z1,z2,…,zn}D = \{z_1, z_2, \ldots, z_n\}

where $D$ is the complete pretraining dataset after prefiltering but before pruning, $z_i$ is the $i$-th sequence of exactly $t tokens, and $n$ is the total number of sequences (approximately 3.7 million sequences for the 7.6B token dataset with $t = 2048$).

What this does: it converts a variable-length document corpus into a uniform grid of fixed-length training instances, each of which can be independently scored and independently included or excluded from training. This is necessary because the model's context window creates a natural atomic unit for both scoring (the reference model processes one full context at a time) and training (each forward pass consumes one full context).

Why fixed-length sequences rather than document-level pruning: the authors choose to prune at the sequence level rather than the document level because perplexity and other model-based scores are computed over fixed-length contexts—a 10,000-token document would need to be split into multiple sequences anyway, and each subsequence might have very different quality characteristics. Pruning at the sequence level gives finer granularity and avoids throwing out an entire long document because of one noisy paragraph.

Step 2: Score computation and subset selection. For each sequence $z_i$, a pruning algorithm $\xi$ computes a scalar score $\text{Score}_\xi(z_i)$. The subset $P_\xi$ of instances to be removed is defined by a selection criterion applied to these scores:

Pξ={zi∈D∣Criteria(Scoreξ(zi))}P_\xi = \{z_i \in D \mid \text{Criteria}(\text{Score}_\xi(z_i))\}

where $\xi \in \{\text{PPL}, \text{EL2N}, \text{Mem}\}$ identifies which pruning metric is used, $\text{Score}_\xi(z_i) \in \mathbb{R}$ is the scalar quality score assigned to sequence $z_i$ by metric $\xi$, and $\text{Criteria}(\cdot)$ is a boolean predicate that returns true for sequences that should be removed (not kept).

What this computes: given the distribution of scores across all $n$ sequences, the criteria function thresholds the distribution to select a specific subset for removal. The remaining data is:

D^ξ=D∖Pξ\hat{D}_\xi = D \setminus P_\xi

where $\hat{D}_\xi$ is the pruned dataset that will actually be used for training.

Why define it as removal rather than retention: this is a notational convention—the paper consistently talks about "pruning" as removal ($P_\xi$ is what gets removed) and the trained-on data ($\hat{D}_\xi$) as what remains. This matters for interpreting the percentile-based criteria: "retaining the bottom 30%" means removing the top 70%, i.e., $\text{Criteria}$ returns true for sequences with scores above the 30th percentile.

Step 3: Success criterion. The goal of pruning is formally stated as a performance preservation constraint:

Pτ(MD^ξ)≥Pτ(MD)\mathcal{P}_\tau(M_{\hat{D}_\xi}) \geq \mathcal{P}_\tau(M_D)

where $M_{\hat{D}_\xi}$ is the model trained on the pruned dataset $\hat{D}_\xi$, $M_D$ is the model trained on the full (unpruned) dataset, $\mathcal{P}_\tau$ is the performance metric on task $\tau$, and the inequality requires that pruning does not degrade performance.

What this states: the pruned model should be at least as good as the unpruned model. This is a deliberately conservative objective—the paper does not require that pruning improves performance, only that it doesn't hurt. In practice, the best configurations exceed this constraint (achieving $\mathcal{P}_\tau(M_{\hat{D}_\xi}) > \mathcal{P}_\tau(M_D)$ with less data), but the formal objective sets a minimum bar.

Why this form is important: it defines pruning success in terms of model performance after retraining, not in terms of any intrinsic property of the kept data. A pruning metric is judged solely by its downstream effect—does a model trained on the surviving data perform well? This avoids the circularity of defining "quality" and then "validating" that the definition is correct by checking whether the data looks high-quality. The paper is measuring whether pruning works, not whether it finds data that matches some preconceived notion of quality.

Perplexity-Based Pruning

Perplexity is the simplest of the three metrics and, paradoxically, the most effective. It measures how "surprised" a reference model is by a given sequence—lower perplexity means the model assigns higher probability to the text, suggesting it is more predictable or more consistent with the model's learned distribution.

Computation. For each sequence $z_i$ of length $|z_i|$ tokens, the perplexity is:

PPL(zi)=exp⁡(1∣zi∣∑tj∈ziNLL(tj))\text{PPL}(z_i) = \exp\left(\frac{1}{|z_i|}\sum_{t_j \in z_i} \text{NLL}(t_j)\right)

where $|z_i|$ is the number of tokens in the sequence (typically the full context length of 2048, unless the sequence is a partial sequence at the end of the dataset), $t_j$ is the $j$-th token in the sequence, and $\text{NLL}(t_j)$ is the negative log-likelihood of token $t_j$ given all preceding tokens in the sequence:

NLL(tj)=−log⁡P(tj∣t<j;θref)\text{NLL}(t_j) = -\log P(t_j \mid t_{<j}; \theta_{\text{ref}})

where $P(t_j \mid t_{<j}; \theta_{\text{ref}})$ is the reference model's predicted probability for token $t_j$ conditioned on the prefix $t_{<j}$ (all tokens before position $j$ in the sequence), and $\theta_{\text{ref}}$ are the parameters of the reference model.

What this computes: for a single sequence, the reference model processes it token by token, computing the negative log-likelihood at each position. These are averaged across all tokens in the sequence, exponentiated, to produce a single number. The exponentiation converts the average NLL (measured in nats) to perplexity, which can be interpreted as the "effective branching factor"—a perplexity of 100 means the model is as uncertain, on average, as if it were choosing uniformly among 100 equally likely options at each token.

Why perplexity rather than raw NLL: perplexity is the standard metric for language model evaluation, making it directly interpretable by practitioners. But more importantly, the exponentiation amplifies differences in the tail—a sequence with NLL 5.0 has perplexity ~148, while one with NLL 6.0 has perplexity ~403. This matters for the distribution of scores: exponentiation stretches the high end, which affects where percentile cutoffs fall and thus which sequences get classified as "low," "middle," or "high" perplexity. The paper does not explore whether using raw NLL would change results, but the choice of perplexity has a specific distributional effect: it makes the high-perplexity tail even more extreme.

What low vs. high perplexity means. A sequence with low perplexity is one where the reference model confidently predicts each next token—the text is highly predictable given the model's training. Examples from the paper's Appendix B (Tables 3-5) make this concrete: low-perplexity text tends to be formulaic legal boilerplate ("Submissions, you hereby grant Company a license to translate, modify..."), standardized disclaimers, and repetitive templates. High-perplexity text includes dense technical specifications, lists of product features with unusual formatting, and broken or ill-formed text. The middle-perplexity examples include descriptive prose about local landmarks, narrative text about daily life, and informational content—the kind of varied, well-formed text that likely teaches the model about the world.

Key design choice: separate reference model. The paper emphasizes that the reference model used to compute perplexity scores is separate from the model that will be trained on the pruned data. This avoids a dangerous circularity: if the same model both scores the data and trains on the scored data, the scores reflect that model's particular strengths and weaknesses, and pruning based on those scores might amplify the model's biases rather than selecting genuinely useful data. Using a separate reference model (which may be larger, trained on different data, or trained for different durations) breaks this circularity and makes the pruning signal independent of the training process.

The reference model ablation (Section 3.3) is extensive:

  • Size: 124M, 6B, 13B, and 52B parameter models, all fully trained.
  • Training data: models trained on CommonCrawl vs. models trained on Wikipedia.
  • Training duration: early checkpoints at 14% and 55% of full training steps vs. fully trained models.

This ablation is designed to answer: does a better reference model (larger, trained on cleaner data, trained longer) produce a more effective pruning signal? The answer, explored in Sections 4.3-4.5, is generally yes—larger models and cleaner training data yield better pruning outcomes.

EL2N-Based Pruning

The Error L2-Norm (EL2N) score was originally developed by Paul et al. (2023) for supervised computer vision, where it measures how much a model's prediction deviates from the ground-truth label early in training. The intuition is that examples with low EL2N scores are "easy"—the model learns them quickly—while examples with high EL2N scores are "hard" and require more training iterations to master. The paper adapts this concept to the unsupervised language modeling setting by treating next-token prediction as a per-token classification task with a one-hot target.

Computation. For a sequence $z_i$ of length $t$ (where $t$ is the context length, 2048), the EL2N score is:

EL2N(zi)=1t∑i=1t∥y^i−yi∥2\text{EL2N}(z_i) = \frac{1}{t} \sum_{i=1}^{t} \|\hat{y}_i - y_i\|_2

where $t$ is the sequence length (nominally 2048), $\hat{y}_i \in \mathbb{R}^{|V|}$ is the reference model's predicted probability distribution over the vocabulary of size $|V| = 51,200$ at position $i$, $y_i \in \{0, 1\}^{|V|}$ is the one-hot encoded representation of the ground-truth next token at position $i$, and $\|\cdot\|_2$ is the Euclidean (L2) norm.

What this computes, step by step: at each token position $i$, the reference model produces a probability vector $\hat{y}_i$ over all 51,200 tokens in the vocabulary—this is the model's predicted distribution of what token comes next. The ground truth $y_i$ is a one-hot vector with a 1 at the index of the actual next token and 0 everywhere else. The L2 norm of their difference, $\|\hat{y}_i - y_i\|_2$, measures the Euclidean distance between the predicted distribution and the one-hot truth. If the model assigns probability 1.0 to the correct token (and 0.0 to all others), the error vector is zero and the norm is 0. If the model assigns probability near 0.0 to the correct token, the error vector is large (roughly $\sqrt{1^2 + \text{(other errors)}^2} \approx \sqrt{2}$ in the worst case where all probability mass is on a single wrong token). These per-token norms are averaged across all $t$ positions in the sequence to produce a single sequence-level score.

Why the L2 norm rather than cross-entropy or L1: Paul et al. (2023) originally used L2 norm in the supervised setting because it empirically provided better pruning signals than cross-entropy for identifying important examples. The L2 norm is more sensitive to the model's confidence on incorrect classes than cross-entropy—if the model spreads probability across many incorrect tokens, the L2 norm penalizes this more heavily than cross-entropy, which primarily cares about the probability assigned to the correct token. In the unsupervised setting, this property means EL2N captures not just whether the model gets the token right, but how much probability mass is misallocated.

What low vs. high EL2N means. A sequence with low EL2N score is one where the reference model's predicted distribution closely matches the one-hot target at most positions—the model is confident and mostly correct on a per-token basis. The paper hypothesizes that these are sequences the model "learns in its early stages of training, likely because they are relatively easier" (Section 2.1.2). Conversely, high EL2N sequences are those where the model continues to incur significant loss. The examples in Appendix B (Table 6) show that low-EL2N text includes well-structured but formulaic content (governmental descriptions, medical trial protocol language), while high-EL2N text includes dense business descriptions with unusual formatting and run-on lists.

Single-checkpoint vs. ensemble scoring. The paper experiments with two variants of EL2N scoring:

  • Single reference model at two checkpoints: one after 250 training steps (~14% of total training) and one after 1000 steps (~55% of total). This tests whether an early signal is sufficient.
  • Ensemble of 10 reference models: the EL2N scores from 10 independently trained models (different random seeds, same architecture and data) are averaged to produce a single score per sequence. The hypothesis is that averaging across seeds reduces noise and produces a more reliable signal.

The ensemble approach is computationally expensive—it requires pretraining 10 separate reference models—and the paper finds that it "did not outperform the best pruned models trained on only one reference model's EL2N score" (Section 4.5), though it did produce more consistent behavior across subset selection criteria.

Why the 55% checkpoint is important. The 14% checkpoint models show "minimal variance across percentages and subset selection criteria" (Section 4.5), meaning early-checkpoint EL2N scores are too noisy to discriminate useful from useless data. The 55% checkpoint models behave more like fully trained models, suggesting that a pruning signal emerges after roughly half of training is complete. This has practical implications: practitioners don't need to fully train a reference model to get useful EL2N scores, but they do need to go substantially beyond the very early stages where the model's predictions are still dominated by initialization noise.

Memorization-Based Pruning

Memorization, as studied in language models (Carlini et al., 2023; Biderman et al., 2023a), refers to a model's ability to reproduce training data verbatim when prompted with a prefix. The paper explores whether a high degree of memorization signals that an example is "easy" for the model (because the model has fully absorbed it) and therefore potentially less useful for further training.

Computation. The memorization score is adapted from Biderman et al. (2023a) and is defined as:

score(M,N)=1N∑i=1N1(zM+i=z^M+i)\text{score}(M, N) = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(z_{M+i} = \hat{z}_{M+i})

where $M$ is the number of prefix tokens provided to the model (set to 32 in all experiments), $N$ is the number of tokens the model generates (also 32), $z_{M+i}$ is the actual $(M+i)$-th token in the original sequence from the training data, $\hat{z}_{M+i}$ is the token greedily generated by the reference model at generation step $i$ (conditioned on the prefix and all previously generated tokens), and $\mathbf{1}(\cdot)$ is the indicator function that equals 1 when the generated token matches the original token exactly, and 0 otherwise.

What this computes, operationally: a 32-token prefix is extracted from the beginning of a training sequence and fed into the reference model. The model then generates 32 tokens greedily (always selecting the highest-probability next token, with no sampling). The fraction of these 32 generated tokens that match the original 32 tokens in the training data is the memorization score. A score of 1.0 means the model reproduced all 32 tokens exactly; a score of 0.0 means none matched. The paper notes that "the authors did not originally propose this as a data pruning metric" (Section 2.1.3)—this is an exploratory application of an existing measurement tool.

Why 32/32 prefix/generation lengths: the choice is not explicitly justified in the paper beyond citing Biderman et al. (2023a). In the original memorization work, these lengths were chosen to balance statistical reliability (longer sequences provide more tokens to match, reducing variance) against computational cost and the natural context length of common n-gram patterns. In the pruning context, 32 tokens of exact match is a relatively stringent criterion—it requires the model to reproduce a substantial span of text verbatim, not just a few words.

What high vs. low memorization means. A sequence with a high memorization score (close to 1.0) is one that the reference model can reproduce nearly verbatim from a short prefix. This typically indicates that the sequence is highly predictable or has been seen many times during training—it may be a duplicated passage, a common template, or a formulaic pattern. A sequence with a low memorization score (close to 0.0) is one where the model cannot reproduce the continuation accurately, suggesting the content is novel relative to the model's training or requires reasoning beyond simple pattern completion.

Reference model requirement. The paper specifies that reference models used for memorization scoring must be "guaranteed to have seen the full training set" (Section 2.1.3). This is because memorization can only be measured if the model was actually exposed to the text during training—you can't measure whether a model memorized text it never saw. This constraint limits which reference model configurations can be used for memorization scoring.

Interpretation for pruning. The paper hypothesizes that "a high memorization score indicates the model reproduces more of the text verbatim" and that such examples "require additional learning" (Section 2.1.3). The intuition is inverted compared to the other metrics: high memorization (which corresponds to "easy" text that the model has fully absorbed) is hypothesized to be less useful, while low memorization (text the model hasn't memorized) may contain more novel information.

Subset Selection Criteria: The "Bottom / Middle / Top" Trichotomy

This is the paper's key methodological innovation and the axis that produces its most counterintuitive results. For each metric, the paper does not simply threshold the score distribution to keep "good" examples and discard "bad" ones—instead, it systematically tests three regions of the distribution independently.

How the criteria are implemented. Given a pruning percentage $p$ (e.g., 30% retention) and a criterion (bottom/middle/top), the paper computes the relevant percentiles of the score distribution across all $n$ sequences and removes all sequences whose scores fall outside the specified region:

  • Bottom $p\%$: keep sequences with scores at or below the $p$-th percentile. Remove all sequences with scores above the $p$-th percentile. This retains the "easiest" examples according to the metric (lowest perplexity, lowest EL2N) or the "most memorized" (highest memorization).
  • Middle $p\%$: keep sequences with scores between the $\frac{100-p}{2}$-th percentile and the $\frac{100+p}{2}$-th percentile. For example, middle 30% means keeping sequences between the 35th and 65th percentiles. This retains examples of "medium" difficulty.
  • Top $p\%$: keep sequences with scores at or above the $(100-p)$-th percentile. This retains the "hardest" examples (highest perplexity, highest EL2N) or the "least memorized" (lowest memorization).

What this achieves: for each combination of metric, retention percentage, and selection criterion, the paper trains a completely separate model from scratch. At minimum nine models per experimental variant (Section 3.3). With three metrics, four retention percentages, three criteria, and various reference model configurations, this represents a substantial computational undertaking—each 124M parameter model is trained for 8000 steps on up to 7.6B tokens of data.

Why test all three regions independently: the standard assumption in prior work (explicitly called out in Section 5.2) was that low perplexity = high quality, i.e., one should retain the bottom of the distribution. The paper tests this assumption rather than accepting it, and discovers that the middle subset consistently outperforms both extremes. This finding—that the "easiest" examples are actually the least useful for training—cannot be discovered without the three-way comparison. It also explains why prior results on perplexity filtering were mixed: if researchers only compared "keep low perplexity" to "keep random," they might see small improvements on some tasks and not others, missing the larger effect available from keeping medium-perplexity data.

Relationship between criteria and data diversity. The authors note (Section 4.1) that "as the middle subset grows, it begins to overlap with the easiest examples, degrading performance." This is important: at 70% retention, the "middle" subset is very wide (roughly the 15th to 85th percentiles) and necessarily includes both easy and hard examples. At 30%, the middle subset is narrower and more selective. The optimal retention percentage represents a tradeoff between data quantity and data quality, with the middle 30-50% being the sweet spot for perplexity-based pruning.

Reference Model Ablation Space

The quality of a pruning signal depends critically on the reference model that produces it. The paper systematically varies three aspects of the reference model:

Model size. Four sizes are compared for perplexity scoring: 124M, 6B, 13B, and 52B parameters. The 6B, 13B, and 52B models are fully trained Cohere models "trained on full web-scale datasets" (Table 1), meaning they were trained on substantially more data than the 124M models. The 124M model is trained in-house on a non-overlapping subset of CommonCrawl. The hypothesis is that larger models produce better-calibrated probability estimates, leading to more discriminative pruning signals—and the results (Section 4.3, Figure 2) confirm that "increasing reference model size improves trained model performance over the no-pruning baseline when either the middle or top subsets are used."

Training data. Two 124M parameter reference models are compared: one trained on CommonCrawl and one trained on English Wikipedia (5.3M tokens). Wikipedia represents a "clean, noise-free corpus" (Section 4.4) deliberately chosen as an example of carefully curated data. The hypothesis is that a model trained on cleaner data is better at distinguishing useful from useless text in a noisy corpus—and the results confirm that the Wikipedia-trained reference model "consistently yields lower validation perplexity compared to a model trained on CommonCrawl" across the middle and top selection criteria (Section 4.4, Figure 5).

Training duration. For both perplexity and EL2N, scores are computed from reference model checkpoints at 14% of total training steps (250 steps for the 124M model) and 55% of total training steps (1000 steps). The 14% checkpoint tests whether very early training signals are sufficient—the results show they are not, with "minimal variance across percentages and subset selection criteria" (Section 4.5). The 55% checkpoint performs similarly to fully trained models, establishing that roughly half the training compute is sufficient.

Key finding (Section 4.5): "Fully training the reference model is shown not to be necessary to uphold comparable performance. Halving the reference model training steps proves effective, enabling the utilization of early checkpoints." This has significant practical implications: in many real-world scenarios, practitioners would use off-the-shelf models for computing perplexity and would not need to bear the cost of pretraining a reference model from scratch. An early checkpoint of a partially trained model already provides a useful pruning signal.

Trained Model Configurations and Hyperparameters

The models trained on pruned data—called "trained models" or "pruned models" to distinguish them from reference models—are decoder-only Transformers following a standard GPT-style architecture.

124M parameter models. The primary experimental scale:

  • Architecture: GPT-style autoregressive decoder-only Transformer (Vaswani et al., 2023; Radford et al., 2018).
  • Training steps: 8000 steps with a batch size of 2048, totaling "33B tokens" (Section 3.1). This is approximately 4.4 epochs over the 7.6B-token unpruned dataset—for more aggressively pruned datasets (e.g., 30% retention), the effective number of epochs increases proportionally.
  • Learning rate: linear warmup from 0 to $1.5 \times 10^{-4}$ over the course of training (not specified whether this is over all 8000 steps or a shorter warmup period followed by decay).
  • Optimizer: AdamW (Loshchilov & Hutter, 2019) with linear cosine scaling. Specific AdamW hyperparameters (betas, epsilon, weight decay) are not reported in the paper.
  • Tokenizer: Byte Pair Encoding (Sennrich et al., 2016) with a vocabulary size of 51,200 tokens.
  • Context window: 2048 tokens.

1.5B parameter models. Scaling experiments use a different training configuration due to computational constraints:

  • Batch size: 512 (reduced from 2048 due to memory constraints).
  • Training steps: 14,568 steps, totaling only "7.6B tokens, equivalent to a single epoch of our unpruned dataset" (Section 3.1).
  • Learning rate: linear warmup from 0 to $1.2 \times 10^{-4}$ over the course of training.

Important caveat on 1.5B results: the 1.5B models see dramatically fewer tokens than the 124M models (7.6B vs. 33B) and only one epoch of the unpruned data. This means the scaling results in Section 4.6 (Figure 7) compare models trained with very different data budgets and may not isolate the effect of model scale from the effect of training duration. The paper acknowledges this distinction but does not control for it—the 1.5B experiments demonstrate that the pruning benefit persists at larger scale, but do not provide a clean scaling law for how the benefit changes with model size at fixed data budget.

Static vs. dynamic pruning. The paper explicitly states (Section 2) that it focuses on "static pruning, in which data is pruned once before training. This is in contrast to adaptive pruning, in which data is pruned as training is happening." The choice is pragmatic: adaptive pruning would require recomputing pruning scores throughout training, which is computationally prohibitive at pretraining scale. Static pruning is feasible because scores are computed once (using the reference model) and reused. The paper acknowledges that dynamic pruning (Park et al., 2022; Fayyaz et al., 2022) might produce better results, but establishing the static baseline is the prerequisite for future comparison.

Evaluation Protocol

Primary metric: test set perplexity. The main evaluation metric is perplexity on a held-out test set from the same CommonCrawl snapshot, processed with identical prefiltering as the training data. The test set contains 266M tokens, approximately 3.5% of the training set size. Using in-distribution test data—rather than a different corpus—measures whether pruning improves the model's ability to model the target data distribution, which is the direct objective of pretraining.

Evaluation timing. Models are evaluated after 8000 steps (for 124M models) or 14,568 steps (for 1.5B models). The paper states this is "chosen to compare performance after models have saturated their capacity by training enough steps to plateau on validation metrics" (Section 3.4). This is important because it means the comparison is between models that have each converged as much as their data allows—a model trained on less data (30% retention) will plateau at a different performance level than one trained on more data (100% retention), and the evaluation captures this asymptotic difference rather than a transient training-speed effect.

Downstream evaluation on GLUE. A subset of pruned models is fine-tuned on six GLUE classification tasks: SST-2 (sentiment analysis), MRPC (paraphrase detection), QQP (question paraphrase), QNLI (question answering), RTE (textual entailment), and WNLI (coreference). The fine-tuning uses "3 epochs with a learning rate of $1 \times 10^{-5}$" (Table 2 caption), with 5 runs per model to compute mean and standard deviation. The task datasets are not pruned—the goal is to measure how pretraining data quality affects downstream transfer, not to study pruning of task-specific data.

Why GLUE tasks specifically: these are standard benchmarks for evaluating the quality of pretrained representations. If pruning degrades performance on downstream tasks even when perplexity is maintained, that would suggest the pruning metric is optimizing for a superficial property of the data that doesn't support general-purpose language understanding. The results (Section 4.7) show that pruned models generally outperform the baseline on downstream tasks, alleviating this concern.

Cross-validation and statistical rigor. The paper does not use cross-validation for model selection—each pruned configuration is trained and evaluated independently on the fixed test set. This is a practical necessity given the computational cost of training each model. The paper reports only point estimates (no confidence intervals) for most perplexity results, and standard deviations only for the GLUE experiments (where 5 runs per configuration provide variance estimates). This limits the ability to assess whether the observed differences (e.g., 0.97% improvement with 50% perplexity pruning) are statistically significant, though the consistent pattern across multiple reference model sizes and pruning percentages supports the qualitative conclusions.

Random Pruning Baseline

The paper includes models trained on randomly pruned subsets as a "lower bound of expected performance" (Section 2.1.4) at each retention percentage (10%, 30%, 50%, 70%). This serves a specific purpose: it distinguishes the effect of removing data (which reduces training signal regardless of which data is removed) from the effect of selective removal based on a quality metric. If perplexity-based pruning outperforms random pruning, the benefit cannot be attributed to simply training on less data—the selection is doing meaningful work.

The random pruning baseline also reveals a subtle point: in some configurations (e.g., the 1.5B model experiments shown in Figure 7), random pruning "performs considerably well, even reaching levels below the no-pruning run" (Section 4.6). This suggests that for models undertrained relative to their capacity (the 1.5B model sees only one epoch), simply reducing the dataset size can sometimes be beneficial even without intelligent selection—perhaps because it forces the model to see each example more times (effectively increasing the number of epochs on the retained data). The fact that perplexity-based pruning still outperforms random pruning in these settings demonstrates that the pruning signal provides genuine value beyond mere downsampling.

4. Key Insights and Innovations

Innovation 1: The "Middle Subset" as a Universal Pruning Principle

The paper's most striking and counterintuitive finding is that, across all three pruning metrics tested, the optimal data to retain is neither the "easiest" examples (lowest perplexity, lowest EL2N, highest memorization) nor the "hardest" examples (highest perplexity, highest EL2N, lowest memorization), but rather the middle of the score distribution. This is not a subtle preference—the paper demonstrates that retaining the bottom (easiest) subset consistently degrades performance, often substantially, relative to both the middle subset and even random pruning at the same retention percentage.

This finding constitutes a fundamental reframing of how the field should think about data quality for pretraining. The dominant assumption in prior work—both explicitly in papers that used perplexity filtering and implicitly in the intuition that "clean, predictable text" should be good training data—was that low perplexity equals high quality. Laurençon et al. (2023) and Muennighoff et al. (2023) filtered out high-perplexity samples under exactly this assumption, treating them as "unnatural language" that was harmful relative to a Wikipedia reference. This paper does not refine that assumption or add nuance to it—it rejects it entirely. The data that looks "best" to the reference model (highly predictable, formulaic, easily scored) turns out to be the data that is least useful for training. The examples in Appendix B make this concrete: low-perplexity text is dominated by legal boilerplate, standardized disclaimers, and repetitive templates—text that is syntactically well-formed but semantically vacuous. The middle-perplexity examples include descriptive prose, narrative content, and informational text that likely teaches the model about the world.

What makes this insight intellectually distinctive is that it cannot be discovered by any binary filtering approach. If you only test "keep low perplexity" vs. "keep all data," you might see small, inconsistent effects (which explains the mixed results in prior literature). It is only by testing all three regions independently—bottom, middle, and top—that the consistent superiority of the middle subset becomes visible. The paper's three-way selection criteria (Section 2) are not just a thorough ablation; they are a diagnostic tool that reveals the relationship between score distribution and training utility. The finding that "as the middle subset grows [to 70% retention], it begins to overlap with the easiest examples, degrading performance" (Section 4.1) further refines the insight: there is a "sweet spot" between data diversity and data quality, and including too many extreme examples—whether too easy or too hard—dilutes the training signal.

The evidence is strongest for perplexity (Figure 2, Figure 4) but the pattern generalizes: for EL2N, "the middle subset is also the best variant" (Section 4.1, Figure 3a), and for memorization, keeping the least-memorized examples (which is the conceptual equivalent of the "hard" subset, not the easiest) performs best (Figure 3b). The universality of the pattern—independent of the specific metric—suggests this is a structural property of pretraining data distributions, not an artifact of any particular scoring method. Easy examples are easy precisely because they contain little novel signal; hard examples may be hard because they are genuinely noisy or malformed. The middle contains examples that are neither trivially predictable nor pathologically difficult—and it is precisely this "Goldilocks zone" of learnable difficulty that drives effective pretraining.

Innovation 2: Perplexity as a Sufficient Statistic for Data Quality

The paper's second major contribution is the empirical demonstration that a remarkably simple metric—sequence-level perplexity computed from a frozen reference model—outperforms significantly more complex and computationally expensive alternatives for the purpose of data pruning. This is not just a claim about computational efficiency (though it is that too); it is a claim about what information is actually relevant for identifying useful training data.

Consider what the alternatives attempt to capture. EL2N (Paul et al., 2023) measures the L2 norm of the error vector at each token position, capturing not just whether the model predicts the correct token but how its probability mass is distributed across the entire vocabulary. Computing EL2N requires storing and processing full probability distributions over 51,200 tokens at every position—a substantial memory and compute burden. The ensemble version requires training 10 separate reference models and averaging their scores. Memorization scores (Biderman et al., 2023a) require the reference model to generate text greedily and compare it to the original, measuring exact token matches over 32-token spans—a process that is both computationally intensive and sensitive to the choice of prefix and generation lengths. Both of these methods are motivated by sophisticated theories of learning dynamics: EL2N identifies samples that the model finds difficult early in training, while memorization identifies samples the model has fully absorbed.

Perplexity requires only a single forward pass through a reference model—the same computation that is already performed during training and evaluation. It does not require storing distributions, averaging across ensembles, or generating text. And yet, Figure 4 shows perplexity-based pruning outperforming both EL2N and memorization across all retention percentages, with the gap being particularly stark at aggressive pruning ratios: at 30% retention, perplexity achieves a 2.1% improvement over the best EL2N variant and a 1.6% improvement over the best memorization variant.

The intellectual significance of this finding is that it calls into question whether more "principled" quality metrics—those grounded in theories of learning dynamics or memorization—actually capture anything beyond what a well-calibrated probability estimate already provides. The paper suggests, implicitly, that the reference model's uncertainty about a sequence is a sufficient statistic for its training utility: sequences that are moderately surprising to a capable model contain learnable signal; sequences that are trivially predictable contain nothing new; sequences that are wildly surprising are likely malformed. This is a deeply Occam's-razor result. The field has been searching for sophisticated data quality metrics, and the answer may be staring us in the face: just ask a good model how confused it is.

This finding is not purely a matter of convenience. It has structural implications for how data pruning can be deployed in practice. Perplexity scoring can be done with any off-the-shelf language model, requires no specialized infrastructure, and scales linearly with dataset size. The paper's finding that early reference model checkpoints (55% of full training) are sufficient (Section 4.5) further reduces the barrier. This transforms data pruning from a research prototype—requiring custom model training and complex scoring pipelines—into something a practitioner could implement with a single script and a pretrained model.

Innovation 3: Reference Model Quality as a Bootstrap for Data Quality

The paper's third key insight is that the quality of the pruning signal is directly limited by the quality of the reference model that produces it—and, crucially, that this creates a bootstrap mechanism where improvements in model capability (through scale, cleaner training data, or better training) translate directly into improved data selection for the next generation of models.

This is not just the unsurprising observation that better models produce better scores. The paper demonstrates a graded, systematic relationship between reference model characteristics and pruning effectiveness that has specific, actionable components:

Scale matters monotonically. Figure 2 shows that increasing reference model size from 124M to 6B to 13B to 52B produces consistent improvements in trained model performance when using the middle or top subsets. The gap between the 52B and 124M reference models is substantial—a 2.2% improvement in perplexity for the trained model—and the effect is monotonic: each step up in model size yields better results. This is evidence that larger models are not just better at language modeling in general, but specifically better at discriminating useful from useless training examples—their probability estimates carry more information about data quality.

Training data quality matters independently of scale. The comparison between reference models trained on Wikipedia (a small, carefully curated corpus) and CommonCrawl (a large, noisy corpus) in Figure 5 reveals that, even at the same 124M parameter count, the Wikipedia-trained reference model produces better pruning signals—a 0.69% improvement over the CommonCrawl-trained reference model's best variant. This isolates the effect of training data quality from model scale: a smaller model trained on cleaner data outperforms an identically-sized model trained on noisier data. The practical implication is that investing in a clean reference corpus, even a small one, amplifies the downstream benefits of pruning.

The bootstrap implication is what makes this finding fundamental rather than incremental. The paper's results suggest a virtuous cycle: train a model on the best currently available data → use that model to score and prune a larger, noisier corpus → train a better model on the pruned data → use that better model to score and prune an even larger corpus. Each generation of model improves the pruning signal for the next, enabling progressively better data selection from progressively larger raw corpora. This is conceptually similar to the self-improvement loops explored in other contexts (e.g., STaR, ReST), but applied to the data selection problem rather than the model's output distribution.

Innovation 4: Pruning as a Superior Alternative to Scaling Data Volume

The paper's fourth contribution is a direct empirical challenge to the "more data is better" scaling paradigm, demonstrating that carefully pruned subsets can outperform the full dataset even when the pruned subset is dramatically smaller—and that this result scales to models in the billion-parameter range.

This is not a claim about data efficiency in the sense of achieving the same performance with less compute (though that is a corollary). It is a stronger claim: that a model trained on half (or even 30%) of the data, when that data is intelligently selected, achieves strictly better performance than a model trained on the full dataset, evaluated on the same test distribution. The 124M model trained on the middle 50% of the perplexity distribution achieves a 0.97% improvement in test perplexity over the fully-trained baseline (Section 4.2). At 30% retention, the improvement is 0.80%. At the 1.5B scale, the perplexity-based pruning achieves a 1.5% improvement over the no-pruning baseline of the same size (Section 4.6, Figure 7).

The intellectual significance of this finding is that it inverts the default relationship between data quantity and data quality. The standard response to suboptimal model performance—motivated by scaling laws (Kaplan et al., 2020) and validated by the success of ever-larger training corpora—has been to collect more data, not less. This paper demonstrates that more data can be actively harmful if the additional data falls into the "easy" or "hard" extremes of the quality distribution. The no-pruning baseline includes all the low-perplexity boilerplate, all the high-perplexity noise, and everything in between. Pruning removes the extremes and retains the informative middle, and the resulting model outperforms the one trained on everything. This is evidence that low-quality data does not merely contribute less signal per token—it may actively interfere with learning by crowding out more useful examples or by teaching the model to model patterns that are not representative of the target distribution.

The scaling result (Figure 7) is particularly important because it addresses the natural objection that pruning benefits might be a small-model phenomenon that disappears at scale. The 1.5B parameter experiments show not only that perplexity-based pruning continues to outperform both the no-pruning baseline and random pruning, but that the pattern of improvement over random pruning "follows a consistent pattern for both the 124M and 1.5B models" (Section 4.6). This suggests the finding is robust to model scale, though the caveat about different training durations (single epoch for 1.5B vs. 4.4 epochs for 124M) limits the strength of this claim.

This insight has direct economic implications. Training data is not free—storage, preprocessing, and I/O all scale with dataset size. If 50% of the data can be discarded with no performance loss (and even a gain), the cost savings are substantial. More importantly, if training budget is fixed, the choice between "train on all data for N steps" and "train on the best 50% of data for 2N steps" becomes a real optimization problem. The paper's results suggest the latter is often preferable, because the additional epochs on high-quality data more than compensate for the reduced data diversity.

Innovation 5: Difficulty as a Distributional Property, Not an Example-Level Absolute

The paper's final conceptual contribution is more subtle but pervades its experimental design: the recognition that data "difficulty" or "quality" is not an intrinsic property of a text, but a property of the relationship between that text and a specific model at a specific stage of training. This reframing is what enables the paper's systematic approach and explains several otherwise puzzling results.

Consider what it means for a sequence to have "low perplexity." The score is computed by a specific reference model with a specific architecture, trained on a specific corpus, at a specific checkpoint. Change the reference model (as the paper does, sweeping from 124M to 52B parameters, from CommonCrawl to Wikipedia training data, from early to late checkpoints) and the scores change—sometimes dramatically, as seen in the distribution shifts in Figure 8a. A sequence that appears "easy" (low perplexity) to a small model might appear "medium difficulty" to a larger one. A sequence that appears "hard" to a model trained on noisy data might appear "easy" to a model trained on clean data.

This is not a weakness of the perplexity metric—it is the paper's central insight about the nature of data quality. Quality is relational: it depends on who is doing the scoring and what they already know. The bootstrap finding (Innovation 3) makes sense precisely because of this relational property—a better reference model produces better scores because it has a richer model of what constitutes "surprising" text. The finding that early-checkpoint perplexity and EL2N are poor pruning signals (Section 4.5) makes sense because an undertrained model hasn't yet developed a useful representation of difficulty. The finding that "easy" text is bad for training makes sense because text that a capable model finds trivially predictable contains no new information relative to what the model already knows—it is "easy" precisely because it is redundant.

Prior work largely treated data quality as an example-level absolute: a document either is or isn't high quality, independent of the model that will consume it. Hand-curated corpora like Wikipedia represent this view—certain sources are "known good" regardless of the model. Rule-based heuristics like blocklist filtering or length thresholds encode the same absolutist assumption. This paper demonstrates that such absolute measures capture at best a coarse proxy for what actually matters: whether an example contributes to a specific model's learning. The examples in Appendix B reinforce this: many low-perplexity sequences (legal boilerplate, product listings) would pass rule-based quality filters—they are grammatically correct, well-formed English text—but are revealed by model-based scoring to be among the least useful training examples.

This relational view of data quality has practical implications that go beyond the specific pruning methods studied. It suggests that the optimal training data for a 124M model may differ from the optimal data for a 1.5B model, even on the same task. It suggests that data pruning schedules might benefit from being adaptive—re-pruning the dataset as the model's capabilities evolve during training. And it suggests that the field's search for a universal "data quality score" is misguided; what matters is the gap between what the model already knows and what a training example can teach it.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use a random sample of the May 2022 snapshot of CommonCrawl, downsampled to 7.6B tokens (about 20% of the full snapshot) due to the computational cost of training many models from scratch. This dataset is prefiltered using a combination of automatic and hand-crafted filters—excluding repetitive documents, documents with high percentages of special characters, and documents containing explicit or toxic text—similar to deduplication steps in Taylor et al. (2022) and Kocetkov et al. (2022). The test set is drawn from the same CommonCrawl snapshot with identical prefiltering and contains 266M tokens, approximately 3.5% of the training set size. A separate English Wikipedia dataset (5.3M tokens) is used only for training one of the reference models, never as training data for the pruned models.

  • Base model(s). All trained models are autoregressive decoder-only Transformers following a standard GPT-style architecture (Radford et al., 2018; Vaswani et al., 2023). The primary experiments use 124M parameter models trained for 8000 steps with batch size 2048, consuming approximately 33B tokens total (roughly 4.4 epochs over the unpruned 7.6B-token dataset). Scaling experiments use 1.5B parameter models trained for 14,568 steps with batch size 512, consuming 7.6B tokens (approximately one epoch over the unpruned dataset). Reference models—the models used to compute pruning scores—range from 124M to 52B parameters and are either trained in-house on non-overlapping CommonCrawl subsets (124M models) or are fully trained Cohere models trained on web-scale datasets (6B, 13B, 52B). All models, both reference and trained, use context windows of 2048 tokens and Byte Pair Encoding tokenization with a vocabulary of 51,200 tokens.

  • Metrics. The primary metric is test set perplexity on the 266M-token held-out CommonCrawl test set, computed as the exponential of the average negative log-likelihood per token. Perplexity directly measures how well the trained model predicts the target data distribution—the same objective used during pretraining. For downstream evaluation, a subset of trained models is fine-tuned and evaluated on six GLUE classification tasks (SST-2, MRPC, QQP, QNLI, RTE, WNLI) using task-specific accuracy metrics. The fine-tuning uses 5 runs per configuration with 3 epochs at learning rate 1e−5, reporting mean accuracy and standard deviation.

  • Baselines. The paper uses four baseline configurations:

    • No pruning baseline: A model trained on the full 7.6B-token prefiltered CommonCrawl dataset without any pruning applied. This represents the status quo and establishes the performance floor that pruning must match or exceed.
    • Random pruning: At each retention percentage (10%, 30%, 50%, 70%), a model is trained on a randomly selected subset of the corresponding size. This is described as a "lower bound of expected performance" (Section 2.1.4) and serves to distinguish the effect of simply training on less data from the effect of intelligent selection.
    • EL2N-based pruning: Using the Error L2-Norm scores from Paul et al. (2023), adapted to the unsupervised language modeling setting by computing the per-token L2 distance between the reference model's predicted probability distribution and the one-hot ground truth.
    • Memorization-based pruning: Using the memorization factor from Biderman et al. (2023a), measuring the fraction of 32 greedily generated tokens that exactly match the original continuation given a 32-token prefix.

    For clarity, the paper compares variants within each of the non-random pruning methods (bottom/middle/top selection criteria) and selects the single best-performing variant of EL2N and memorization to compare against perplexity in the main results (Figure 4). The best EL2N variant is the middle subset at the 55% checkpoint; the best memorization variant is the bottom subset (least memorized examples).

  • Generation budget / compute accounting. The paper does not use "generation budget" in the sense of a test-time compute allocation. Instead, the unit of accounting is the training data budget: the percentage of the original 7.6B-token dataset retained after pruning (10%, 30%, 50%, 70%, or 100% for the no-pruning baseline). All trained models at a given scale (124M or 1.5B) see the same number of training steps regardless of the pruning percentage—for the 124M models, this is always 8000 steps at batch size 2048, meaning models trained on pruned subsets see their data more times (more epochs) than models trained on the full dataset. The paper does not normalize for total tokens seen, which means the comparison is between "same training steps, different data subsets" rather than "same tokens seen, different data subsets." This choice is noted in Section 3.1: the 124M models see approximately 4.4 epochs over the unpruned data, while models trained on 30% retention see approximately 14.7 epochs over their pruned subset. For the 1.5B models, training is 14,568 steps at batch size 512 (7.6B tokens total), which is one epoch over the unpruned data or more epochs over pruned subsets.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection—each pruning configuration is trained once and evaluated on the fixed held-out test set. This is a practical necessity given the computational cost: each of the minimum nine models per experimental variant (Section 3.3) requires training a 124M parameter model from scratch for 8000 steps. The paper reports standard deviations only for the GLUE fine-tuning experiments (where 5 fine-tuning runs per model provide variance estimates, reported in Table 2). For perplexity results, only point estimates are provided. The paper specifies that reference models trained on CommonCrawl are trained on "a non-overlapping subset from the CommonCrawl dataset that is pruned and used to train the student model" (Section 3.3), partially addressing concerns about data leakage between scoring and training.

Main Quantitative Results

Subset Selection: Easy Examples Degrade Performance Across All Metrics

The paper's most fundamental result is not about which metric works best, but about which portion of the score distribution should be retained, and this finding is consistent across all three pruning methods. Section 4.1 establishes that "the highest performant variants are not the subsets that correspond to the 'easier' data"—where "easy" is defined differently for each metric: lowest perplexity, lowest EL2N, or highest memorization.

For perplexity-based pruning, Figure 2 shows this pattern with striking clarity. Across all reference model sizes (124M through 52B) and all pruning percentages (10% through 70%), the bottom subset (lowest perplexity, i.e., the "easiest" examples) consistently produces the highest test perplexity—meaning the worst trained model performance—and shows "much less variance in results between reference models of varying sizes, indicating the bottom subset may not be suitable for training" (Section 4.1). The middle subset "achieves consistently low test set perplexities for various reference model sizes and pruning ratios." The top subset (highest perplexity, i.e., the "hardest" examples) shows intermediate performance with higher variance. For the best reference models (52B parameters), the middle subset demonstrates a counterintuitive non-monotonicity: "retaining only 50% and even 30% of the dataset outperforms retaining 70% of the dataset" (Section 4.1). This means that, at least for high-quality reference models, there exists a point where removing more data actually improves performance—the additional data being removed is harmful.

For EL2N-based pruning, Figure 3a shows the same qualitative pattern: "the middle subset is also the best variant for EL2N" (Section 4.1). The best-performing EL2N configuration achieves comparable performance to the no-pruning baseline (but does not surpass it) when retaining 50% of the middle subset—notably, this outperforms the model trained on 70% of the dataset, mirroring the non-monotonicity observed with perplexity. The paper notes that "as the middle subset grows [to 70%], it begins to overlap with the easiest examples, degrading performance" (Section 4.1), which explains why the wider 70% middle band underperforms the narrower 50% band: the wider band includes too many "easy" examples that dilute the training signal.

For memorization-based pruning, the pattern inverts in a way that actually confirms the same underlying principle. Figure 3b shows that "keeping the least memorized samples (bottom subset) generally performs best" (Section 4.1). This is the logical equivalent of the "hard" subset under the memorization metric—low memorization means the model cannot reproduce the text, suggesting it contains novel information. So even though the "best" subset is called "bottom" for memorization, it corresponds conceptually to the "hard" examples, not the "easy" ones. The most competitive variant is the bottom 70% of the memorization distribution, but memorization "never outperforms the no-pruning baseline" (Section 4.1).

The universality of this finding—across perplexity, EL2N, and memorization, all three metrics show that retaining the "easiest" examples according to that metric degrades performance—is the paper's central empirical insight and the justification for the three-way selection criteria framework. It cannot be discovered by any binary "keep good/discard bad" approach and represents a genuine reframing of how to think about data quality scores.

Perplexity Outperforms More Complex Metrics

Section 4.2 presents the head-to-head comparison of the best variant of each pruning method. Figure 4 is the key result figure, plotting test set perplexity against the percentage of original data retained for:

  1. Perplexity (best variant: middle subset, 52B reference model)
  2. EL2N (best variant: middle subset, 1000-step checkpoint)
  3. Memorization (best variant: bottom subset, least memorized)
  4. Random pruning
  5. No pruning (100% data)

The headline numbers: perplexity-based pruning outperforms all other methods at every retention percentage tested. At 50% retention, perplexity achieves a 0.97% improvement in test perplexity over the no-pruning baseline—the only method to beat the baseline by a meaningful margin. At 30% retention, it achieves a 0.80% improvement. The comparison with alternative metrics is even more stark: "a model trained on 50% of the dataset pruned based on perplexity achieves 1.33% and 1.77% improvement over the most performant models pruned to 50% of the dataset with EL2N and memorization factor respectively. A model trained on 30% of the dataset pruned with perplexity achieves a 2.1% and 1.6% improvement over the most performant models pruned to 30% of the dataset with EL2N and memorization factor" (Section 1, Contribution 1).

Compared with random selection, perplexity-based pruning "results in significantly higher model performance than random pruning across all data ratios" (Section 4.2). This is critical: it demonstrates that the benefit is not simply from training on less data, but from selectively removing specific examples. For memorization and EL2N, "both achieve similar performances to random pruning despite being far more computationally expensive" (Section 4.2)—a negative result that is itself important because it shows that computational cost is not a proxy for pruning effectiveness.

The shape of the perplexity curve in Figure 4 is also informative. Performance from the 52B reference model peaks at 50% retention (slightly better than the no-pruning baseline), is nearly identical at 30% retention (0.80% improvement), and drops below baseline at 10% retention. This suggests there is a genuine optimal retention ratio between 30% and 50% where the tradeoff between data diversity and data quality is maximized—below this range, the model loses too much useful signal, and above it, the inclusion of low-quality examples begins to hurt.

Reference Model Properties Matter Systematically

Section 4.3 (Figure 2) examines how reference model size affects pruning quality for perplexity-based pruning. The key finding: "increasing reference model size improves trained model performance over the no-pruning baseline when either the middle or top subsets are used" (Section 4.3). The 52B parameter reference model achieves a 2.2% improvement in perplexity over the best-performing trained model from the 124M parameter reference model experiments. The benefit is monotonic—6B, 13B, and 52B each outperform the next smaller model—and qualitatively changes the optimal retention percentage: for the 13B and 52B reference models, "retaining the middle 30% and 50% of the training data produces pruned models that outperform the pruned models trained on the middle 70% of the training set," whereas for the 124M model, 70% retention is still optimal. This means larger reference models not only produce better pruning signals but also enable more aggressive pruning—you can discard more data without hurting performance.

An interesting amplifier effect emerges with the bottom subset: "the larger reference models' bottom subset training runs perform even worse than their smaller counterparts when retaining the same percentage of the training set" (Section 4.3). Larger models are better at identifying the truly easiest examples, and these examples are even less useful for training than the ones a smaller model would identify as easy. The paper interprets this as evidence that "larger models are better calibrated at computing a useful data pruning ranking" (Section 4.3).

Section 4.4 (Figure 5) examines how reference model training data affects pruning quality. Two 124M parameter reference models are compared: one trained on CommonCrawl and one trained on English Wikipedia (5.3M tokens). The Wikipedia-trained model "consistently yields lower validation perplexity compared to a model trained on CommonCrawl" in the two optimal selection variants (middle and top). The best Wikipedia variant (middle 70%) outperforms the best CommonCrawl variant (also middle 70%) by 0.69%. This gap is notable because the Wikipedia-trained model is trained on dramatically less data (5.3M vs. billions of tokens), yet produces a better pruning signal. The paper frames this as evidence that "investing in a high quality reference model to generate rankings results in more effective data pruning. Reference models trained on higher quality data are better at identifying a subset of data points most conducive to model performance" (Section 4.4). The implication is that reference model quality—both in terms of scale and training data cleanliness—is a first-class variable in pruning system design.

Section 4.5 (Figure 6) examines how reference model training duration affects pruning quality for both perplexity and EL2N. Two early checkpoints are tested: at 14% of total training steps (250 steps) and 55% (1000 steps). At the 14% checkpoint, both perplexity and EL2N show "minimal variance across percentages and subset selection criteria. Performance across subsets changes considerably less than either the 55% checkpoint or the fully trained models" (Section 4.5). The interpretation is clear: "training on only 14% of the data is inadequate for our reference model to offer precise pruning scores." In contrast, the 55% checkpoint models "perform in a similar manner to the fully trained models, performing best with the middle subset, worst with the bottom subset, and comparably with the top subset." The practical takeaway is that "fully training the reference model is shown not to be necessary to uphold comparable performance. Halving the reference model training steps proves effective, enabling the utilization of early checkpoints" (Section 4.5).

The EL2N ensemble experiment—averaging scores across 10 reference models at the 55% checkpoint—yields nuanced results. The ensemble "did not outperform the best pruned models trained on only one reference model's EL2N score" (Section 4.5), meaning the additional compute does not buy a higher peak performance. However, "the pattern of performance more similarly mirrors what we see with the larger, fully trained reference models." Specifically, the ensemble produces more consistent behavior: in the middle subset, 50% retention outperforms 70% retention (mirroring the large-model pattern), and when constrained to the bottom subset, performance more clearly monotonically degrades with less data. The paper interprets this as the ensemble helping "hone the pruning signal, identifying subsets 'easy' or 'hard' subsets in more similar ways to larger models" (Section 4.5). The ensemble improves calibration and consistency but not peak performance—a tradeoff that may or may not be worth the 10× computational cost.

Scaling to 1.5B Parameters Preserves the Benefit

Section 4.6 (Figure 7) validates the strongest pruning variant—perplexity computed using the 52B reference model, keeping the middle subset—at the 1.5B parameter scale. The figure compares perplexity-based pruning against random pruning for both 124M and 1.5B trained models across all pruning percentages (10% through 100%).

The key finding is that "perplexity-based pruning achieves better results than random pruning across all pruning percentages" for the 1.5B model (Section 4.6). The 1.5B model with perplexity-based pruning achieves approximately a 1.5% improvement in test perplexity over the no-pruning baseline of the same size. Notably, random pruning at the 1.5B scale "performs considerably well, even reaching levels below the no-pruning run" (i.e., slightly better than the baseline), which the paper does not fully explain but may relate to the single-epoch training budget (7.6B tokens for 1.5B vs. 33B tokens for 124M). Despite random pruning's competitiveness, perplexity-based pruning consistently outperforms it, and the improvement "follows a consistent pattern for both the 124M and 1.5B models" (Section 4.6).

The importance of this result is that it addresses the natural concern that pruning benefits might be a small-model artifact. The 1.5B experiments demonstrate that the effect persists at larger scale, though the caveat that the 1.5B models are trained on dramatically fewer tokens (7.6B vs. 33B) and for only one epoch of the unpruned data limits the strength of the scaling claim. The paper acknowledges this constraint implicitly—the different training configurations are described in Section 3.1 without a direct apples-to-apples scaling comparison.

Downstream GLUE Performance Confirms Pruning Does Not Harm Transfer

Section 4.7 (Table 2) evaluates a subset of pruned models on six GLUE classification tasks to assess whether pretraining data pruning—which optimizes for language modeling perplexity—has unintended negative consequences for downstream task performance.

The headline result is that "pruning the pretraining dataset consistently improves performance across all tasks" (Section 4.7). However, no single pruning strategy dominates: "while no single pruning strategy (combining both pruning metric and percentage of remaining data) stands out as superior across all tasks, the absence of a universally dominant approach is consistent with earlier findings in the literature" (Section 4.7, citing Gao, 2021).

Specific task-level highlights from Table 2 include:

  • SST-2: The EL2N middle 50% variant achieves the highest accuracy at 79.17% (±0.007), improving over the no-pruning baseline of 78.15% (±0.002). Memorization bottom 30% achieves 78.52%.
  • MRPC: EL2N middle 70% achieves the best result at 66.46% (±0.018) vs. baseline 64.32% (±0.021).
  • QQP: EL2N middle 30% achieves 77.47% (±0.001) vs. baseline 76.55% (±0.001).
  • QNLI: EL2N middle 30% achieves the highest at 68.63% (±0.005) vs. baseline 65.40% (±0.006).
  • RTE: EL2N middle 10% achieves 51.95% (±0.021) vs. baseline 49.69% (±0.024).
  • WNLI: EL2N middle 30% achieves 55.31% (±0.067) vs. baseline 51.56% (±0.040).

Several nuances are worth highlighting. First, EL2N variants appear disproportionately in the best results—EL2N achieves the highest accuracy on 5 of 6 tasks, despite being outperformed by perplexity on the perplexity-based evaluation metric. The paper does not reconcile this tension, though it may reflect different properties of the metrics: EL2N might select for data that is particularly beneficial for transfer learning, even if it doesn't optimize perplexity as effectively.

Second, random pruning shows improvements on several tasks. For MRPC, random 30% achieves 66.04%—better than baseline and competitive with the best pruning variants. For RTE, random 30% achieves 51.33%. For WNLI, random 30% achieves 50.31%. This suggests that "even random pruning shows improvements in certain tasks, underscoring the significance of downsampling when handling noisy data during the pretraining stage to mitigate potential learning degradation" (Section 4.7). In other words, simply reducing the size of a noisy pretraining corpus—even without intelligent selection—can improve downstream transfer, perhaps because the model sees fewer misleading patterns.

Third, the variance across 5 fine-tuning runs (reported as standard deviations in Table 2) is generally small for the larger, more stable tasks (SST-2, QQP, QNLI have standard deviations of 0.001–0.007) but large for the smaller tasks (RTE at ~0.020, WNLI at ~0.040–0.067). This is expected given the smaller dataset sizes for those tasks, but it means that the apparent improvements on RTE and WNLI should be interpreted with caution—the confidence intervals overlap substantially with the baseline.

Ablation Studies and Robustness Checks

The paper's ablation structure is not organized as a separate section but is woven throughout the results. Here, the key non-trivial ablations are extracted and consolidated:

  • Subset selection criteria (bottom vs. middle vs. top): This is the central ablation and is tested for every combination of metric and retention percentage. Result: the middle subset consistently outperforms both extremes for perplexity and EL2N; the bottom subset (least memorized) outperforms for memorization. The bottom (easiest) subset is universally the worst performer for perplexity and EL2N, with the performance gap widening as reference model quality improves (Figures 2, 3a, 3b). This ablation is what makes the paper's contribution possible—without testing all three, the field would continue assuming "low perplexity = good data."

  • Reference model size (124M, 6B, 13B, 52B for perplexity): Result: Monotonically increasing benefit from larger reference models when using the middle or top subsets, with the 52B model achieving a 2.2% improvement over the 124M model (Figure 2). The bottom subset shows the opposite pattern—larger models make the bottom subset perform even worse, suggesting they are better at identifying truly useless "easy" examples. This is tested only for perplexity; EL2N and memorization are computed only from 124M reference models.

  • Reference model training data (CommonCrawl vs. Wikipedia at 124M): Result: The Wikipedia-trained reference model produces better pruning signals than the CommonCrawl-trained model of the same size, with a 0.69% improvement in the best variant. This holds for both middle and top selection criteria (Figure 5). This ablation isolates the effect of reference data quality from reference model scale and demonstrates that cleaner training data for the reference model translates to better data selection for the trained model.

  • Reference model training duration (14% vs. 55% vs. full training for perplexity and EL2N): Result: The 14% checkpoint is insufficient for either metric, producing near-flat pruning performance across all configurations. The 55% checkpoint performs comparably to fully trained models, suggesting that roughly half of training is sufficient to produce a useful pruning signal (Figure 6). This is tested for 124M parameter models only. The practical implication—that early checkpoints can be used, reducing the cost of reference model training—is claimed but not validated at larger reference model scales.

  • Single vs. ensemble EL2N scoring (1 vs. 10 reference models at 55% checkpoint): Result: The ensemble does not improve peak performance over a single reference model's EL2N score, but produces more consistent behavior across subset selection criteria—the ensemble's middle subset shows the "50% outperforms 70%" pattern characteristic of larger models, and its bottom subset degrades more monotonically with less data (Section 4.5, not shown in a standalone figure). This is a negative result for the hypothesis that ensembling would be clearly beneficial and a positive result for the robustness of single-model scoring.

  • Pruning percentage (10%, 30%, 50%, 70%): Result: Performance is not monotonic with retention percentage for the best configurations. For the 52B reference model and 13B reference model using perplexity with middle subset, 30% and 50% retention both outperform 70% retention (Figure 2). This demonstrates that including more data can be actively harmful when that data contains the "easiest" or "hardest" examples. The optimal retention percentage depends on the reference model quality—better reference models enable more aggressive pruning.

  • Trained model scale (124M vs. 1.5B): Result: The benefit of perplexity-based pruning over random pruning persists at 1.5B scale, with approximately a 1.5% improvement over the no-pruning baseline (Figure 7). The pattern of improvement is consistent across scales. However, the 1.5B models are trained on substantially fewer tokens (7.6B vs. 33B) and for only one epoch, making this a comparison of models trained with different data budgets as well as different parameter counts. The confounding of scale and training duration limits the strength of the scaling claim.

  • Downstream task evaluation (GLUE, 6 tasks): Result: Pruned models generally outperform the no-pruning baseline across all six GLUE tasks, confirming that pretraining data pruning does not harm—and may modestly improve—downstream transfer (Table 2). This is a critical robustness check because the pruning objective is perplexity, not downstream accuracy; if pruning improved perplexity at the cost of downstream performance, it would represent a form of overfitting to the language modeling objective.

Critical Assessment

The paper makes three central claims, each of which requires careful scrutiny against the experimental evidence:

Claim 1: Perplexity-based pruning outperforms more complex metrics and the no-pruning baseline, achieving improved performance with substantially less data.

The evidence for this claim comes primarily from Figure 4, which shows perplexity (52B reference, middle subset) outperforming EL2N, memorization, random pruning, and the no-pruning baseline across most retention percentages. The specific numbers—0.97% improvement at 50% retention, 0.80% at 30% retention—are visually apparent in the figure and quoted in the text. However, two aspects of this evidence deserve scrutiny.

First, the perplexity advantage in Figure 4 is for the best perplexity variant (52B reference model, middle subset) compared against the best EL2N and memorization variants. This is a fair comparison—selecting the best variant of each method—but it means the claim is conditional on having access to a 52B parameter reference model. When using a 124M reference model, perplexity still outperforms EL2N and memorization, but the margin is smaller and the no-pruning baseline is not surpassed (inferable from Figure 2's 124M panel, where the middle subset curve does not cross the baseline line with the same clarity). So the claim "perplexity-based pruning beats the baseline" is true for large reference models but may not hold for small ones—a qualification the paper does not emphasize.

Second, the paper does not provide statistical measures of uncertainty for the perplexity results. The 0.97% and 0.80% improvements are single-run point estimates. Given that the test set is 266M tokens (large by typical standards), the variance in perplexity estimates should be small, but without confidence intervals or multiple training runs, it is impossible to assess whether the difference between, say, 30% retention (0.80% improvement) and 50% retention (0.97% improvement) is meaningful or noise. The GLUE results provide standard deviations (Table 2) and show that for some tasks, the variance across 5 fine-tuning runs is substantial (WNLI standard deviation ~0.040–0.067), which underscores the need for similar variance estimates in the pretraining results.

Claim 2: The "easiest" examples degrade performance regardless of metric, and the middle subset is optimal.

This is the strongest-validated claim in the paper. The pattern appears consistently across all three metrics (Figures 2, 3a, 3b), across multiple reference model sizes (Figure 2), and across all pruning percentages. The bottom (easiest) subset is universally the worst for perplexity and EL2N; for memorization, where "easy" means high memorization, the pattern inverts in exactly the predicted way—the least memorized examples are best. The cross-metric consistency of this finding makes it robust: it is not an artifact of how perplexity behaves, but a structural property of how model-based quality scores relate to training utility.

The main weakness is that the "middle subset" conclusion is partially dependent on the percentile boundaries chosen. The paper tests four specific retention percentages (10%, 30%, 50%, 70%) and three selection criteria (bottom, middle, top), where "middle 30%" means keeping the 35th–65th percentiles. At 70% retention, the middle subset spans from roughly the 15th to the 85th percentiles—a very wide band that includes substantial portions of both the "easy" and "hard" tails. The paper's own observation that "as the middle subset grows, it begins to overlap with the easiest examples, degrading performance" acknowledges this. A finer-grained sweep—testing narrow bands at different positions in the distribution rather than just three coarse categories—would provide stronger evidence about the optimal region and whether the ideal band is symmetric around the median.

Claim 3: The pruning benefits scale to larger models (1.5B parameters).

The evidence here is Figure 7, which shows perplexity-based pruning outperforming random pruning at both 124M and 1.5B scales. The pattern is consistent and the improvement at 1.5B (~1.5% over baseline) is directionally larger than at 124M (~1% over baseline). However, the scaling experiment has a serious confound that limits its interpretation: the 124M models are trained for 8000 steps at batch size 2048 (33B tokens, ~4.4 epochs), while the 1.5B models are trained for 14,568 steps at batch size 512 (7.6B tokens, ~1 epoch). These are fundamentally different training regimes—the 1.5B models are undertrained relative to their capacity (one epoch on 7.6B tokens for a 1.5B model is far below what scaling laws would prescribe), while the 124M models are trained closer to convergence (4.4 epochs). The observed improvement at 1.5B might reflect the interaction of pruning with undertraining—when a model sees data only once, removing noisy examples might matter more than when it sees data multiple times and can average out noise. Alternatively, the improvement at 1.5B might overstate the true scaling benefit because the baseline (no pruning at 1 epoch) is weaker than it would be with more training. Without training the 1.5B models for comparable numbers of epochs, the scaling claim remains suggestive but not definitive.

Additionally, the 1.5B experiments use the same 52B reference model for perplexity scoring, which means the improvement is attributable to better data selection, not to better reference model scaling. This is a reasonable experimental choice but means we learn nothing about how the optimal reference-model-to-trained-model size ratio changes with scale.

Missing experiments that would strengthen the paper:

  • Multiple training runs per configuration. Single-run point estimates for perplexity limit the ability to distinguish signal from noise, especially when the claimed improvements are modest (0.8–1.5%). Training 3–5 models per key configuration would provide variance estimates and allow statistical claims.

  • Normalization for total tokens seen. The paper compares models trained for the same number of steps on different amounts of data, meaning models on pruned data see more epochs. An alternative comparison—training all models for the same number of total tokens by extending training for models on pruned data—would separate the effect of data quality from the effect of more repetitions.

  • Testing on a different pretraining corpus or domain. All experiments use CommonCrawl. If the "middle subset is best" finding generalizes to other corpora (e.g., The Pile, C4, domain-specific text), the paper's claims would be much stronger. If it does not—if, for example, Wikipedia's optimal subset is the top (hardest) examples—then the finding is corpus-specific and less actionable.

  • Direct comparison to rule-based pruning baselines. The paper frames rule-based heuristics as the dominant paradigm but never trains a model pruned by those heuristics for direct comparison. A head-to-head of "best rule-based filter" vs. "perplexity middle 50%" would quantify the advantage of metric-based pruning over the status quo, which is the practical question practitioners care about.

  • Qualitative analysis of what the middle subset contains. Appendix B provides examples from bottom, middle, and top subsets, which is helpful, but there is no systematic characterization—no topic modeling, no linguistic feature analysis, no measurement of diversity or redundancy. Understanding what distinguishes middle-perplexity text from low-perplexity text beyond "it looks more substantive" would strengthen the mechanistic interpretation of the results.

  • Adaptive or curriculum-based pruning. The paper acknowledges that dynamic pruning is beyond its scope, but a simple experiment—pruning to 50% of the middle subset, then training, then re-pruning at an intermediate checkpoint—would begin to test whether the optimal data changes as the model learns.

Where the claims hold conditionally:

  • Perplexity outperforms more complex metrics when using a sufficiently large and well-trained reference model. With a 124M reference model, the advantage over EL2N is present but much smaller, and the no-pruning baseline is not surpassed (Figure 2).

  • The "easiest examples degrade performance" finding holds across all tested metrics and reference model configurations, making it the most robust result in the paper. The magnitude of degradation increases with reference model quality—larger models make the bottom subset even worse—suggesting the finding would only strengthen with further scaling.

  • The scaling to 1.5B parameters shows that the benefit persists under the specific training regime tested (single epoch, 7.6B tokens), but the magnitude and even the existence of the benefit under a more standard training regime (multiple epochs, more total tokens) remain unverified.

  • The GLUE downstream improvements show that pruning does not harm transfer, but the best pruning method for downstream accuracy (EL2N, on 5 of 6 tasks) is not the best method for perplexity (perplexity). This suggests a potential tradeoff that the paper does not explore: different pruning metrics may optimize for different downstream properties, and the choice of metric should depend on the intended use case.

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Not Accounted For

The assumption or constraint. The entire pruning framework depends on scoring every training sequence with a reference model before training begins. The paper states this explicitly: "for each instance $z_i$ in $D$, we compute the perplexity metric" and "we use a separate model to compute perplexity from the model trained on the pruned data" (Section 2, Section 3.3). The computational cost of this scoring step—running a full forward pass of a reference model over the entire 7.6B-token training corpus—represents a substantial, non-trivial overhead that occurs before any training benefit is realized. In the most performant configuration, this reference model is a 52B parameter model, making the scoring cost itself significant relative to the training cost of a 124M or even 1.5B parameter student model.

The consequence. The headline numbers—0.97% perplexity improvement using 50% of the data, 1.5% improvement at 1.5B scale—account only for the data savings during the training phase itself. They do not amortize the cost of running the 52B reference model over the full dataset to compute the scores that make the pruning decision possible. For a practitioner deciding whether to adopt this method, the relevant metric is not "training cost saved" but "total cost (scoring + training) relative to the baseline." If scoring 7.6B tokens with a 52B model costs roughly half as much as training a 124M model on those tokens (a plausible estimate given that training requires both forward and backward passes with optimizer state), then the net savings are substantially less than the headline 50% data reduction suggests. For the 1.5B model experiments, where training costs 7.6B tokens, the scoring overhead with a 52B model could easily approach or exceed the training cost itself, potentially eliminating any net efficiency gain.

The paper partially acknowledges this concern when discussing the separate reference model requirement, noting that "in practice, we expect many practitioners to use off the shelf models for computing perplexity and may not need to carry the cost of pretraining a reference model from random initialization" (Section 4.5). But the distinction between pretraining a reference model and running inference with an existing reference model matters: even with a pre-existing model, scoring 7.6B tokens requires a substantial amount of compute—roughly equivalent to training the same reference model on that many tokens for a single forward pass, which for a 52B model is non-trivial.

What evidence exists in the paper. The paper does not quantify the compute cost of reference model scoring relative to training cost. No FLOPs counts, GPU-hour estimates, or wall-clock time measurements are provided for the scoring step. The scaling experiments in Section 4.6 (Figure 7) use "the same 52B reference model for perplexity scoring," meaning the overhead is constant across trained model sizes, but the paper never discusses whether this constant overhead dominates the total cost at small trained model scales. The paper's statement that "for simplicity" they "do not account for this cost" (paraphrasing the spirit of the Section 3.3 discussion) means this limitation is explicitly present but unmeasured.

Mitigation status. The paper partially addresses this through its finding that early reference model checkpoints (55% of full training) are sufficient for effective pruning (Section 4.5, Figure 6). This means the reference model itself can be cheaper to produce—it doesn't need to be fully trained. But the paper does not test whether a smaller reference model (e.g., 6B parameters instead of 52B) run on the full dataset achieves comparable pruning quality to a larger model run on a subsample of the data, which would be the natural cost-benefit exploration. The suggestion that "off the shelf models" could be used (Section 4.5) shifts the cost to the infrastructure provider but doesn't eliminate it. No explicit future work on reducing or amortizing the scoring cost is proposed.

Hard or Informative Examples That Are Still "Noisy" Cannot Be Recovered by This Method

The assumption or constraint. The paper's pruning framework operates on the score distribution of the entire training corpus and selects a fixed percentile band of "middle" examples for retention. This implicitly assumes that the quality of a training example is well-captured by its position in the score distribution and that the optimal pruning strategy is a single contiguous band of that distribution. There is no mechanism for distinguishing between a sequence that is "hard" because it contains genuinely novel and useful information (e.g., a technical explanation with rare terminology) and a sequence that is "hard" because it is malformed or incoherent (e.g., broken markup, garbled encoding, or incomplete sentences). Both types of sequences will fall in the high-score tail of the distribution and be discarded when the top subset is pruned, yet one is high-value training data and the other is genuinely detrimental.

The consequence. The paper's framework cannot selectively retain "useful hard" examples while discarding "noisy hard" examples because the pruning metric—whether perplexity, EL2N, or memorization—produces a unidimensional score that conflates these two categories. The consequence is that the pruning strategy necessarily discards some genuinely valuable training data alongside the noise whenever it removes the upper tail of the distribution. The examples in Appendix B (Tables 3-7) illustrate this: the "top 10%" high-perplexity examples include both clearly degraded text ("and a nice book as a nice price. Postage is via Royal Mail 1st Class...") and text that appears to be well-formed but domain-specific technical content ("several cement manufacturers still prefer ball mills for cement production when they want to design new grinding plants..."). The pruning framework treats both identically (both are discarded in the bottom-retention or middle-retention configurations), but they likely have very different training utility.

This limitation is structural, not merely a matter of finding a better threshold. A unidimensional score—no matter how well-calibrated—cannot separate two distinct failure modes (genuine noise vs. genuinely hard informative content) that occupy the same region of the score distribution. Addressing this would require either a multidimensional quality metric or a fundamentally different selection mechanism that does not rely solely on a single score's percentile.

What evidence exists in the paper. The paper does not directly measure this conflation. The ablation showing that the middle subset outperforms both extremes (Figures 2, 3a, 3b) is consistent with the hypothesis that the top tail contains a mixture of useful and harmful examples, with the harmful fraction dominating at the most extreme scores but the useful fraction becoming significant closer to the median—hence the middle band's superiority. But this is an interpretation, not a measurement. The paper provides no experiment that separates "hard but high-quality" from "hard because noisy" data and compares their individual training utility. The qualitative examples in Appendix B are suggestive but unsystematic.

Mitigation status. Not addressed. The paper does not discuss the conflation of different types of "hard" examples, nor does it propose any mechanism for distinguishing them. The framework remains a unidimensional thresholding approach. Future work on multidimensional quality scoring—perhaps combining perplexity with measures of syntactic well-formedness, semantic coherence, or topic relevance—could address this gap, but the paper neither explores nor suggests this direction.

The Optimal "Middle" Band Depends on the Reference Model and Is Not Portable Across Settings

The assumption or constraint. The entire pruning pipeline depends on computing scores from a specific reference model and then selecting a specific percentile band (the "middle") of the resulting score distribution. The paper demonstrates that the optimal retention percentage and even the qualitative pattern of which band is best depend on the reference model's properties: larger reference models (13B, 52B) show that "retaining only 50% and even 30% of the dataset outperforms retaining 70%" when using the middle subset, while smaller reference models (124M, 6B) do not show this inversion (Section 4.3, Figure 2). Similarly, reference models trained on Wikipedia produce different score distributions than those trained on CommonCrawl, shifting which examples fall into the "middle" band (Section 4.4, Figure 5). Early checkpoint reference models produce different score distributions than fully trained models (Section 4.5, Figure 6).

The consequence. The paper does not provide a pruning recipe that a practitioner can apply without first conducting their own ablation study. The findings are conditional on a specific configuration: "for a 52B reference model trained on web-scale data, keeping the middle 30–50% of the perplexity distribution on CommonCrawl yields the best results." But if a practitioner has a different base model (7B parameters, trained on a different data mixture, with a different tokenizer), a different training corpus (e.g., The Pile, RefinedWeb, domain-specific text), or a different training budget for the student model, the optimal percentile band and even the optimal selection criterion (middle vs. top vs. bottom) may shift. There is no transferable principle like "keep examples at the 40th–60th percentile" that holds across settings—the optimal band is an empirical property of the specific model-data pair.

This matters because the core promise of the paper is that data pruning is a general technique for improving pretraining efficiency. If the optimal pruning strategy must be re-discovered for every new reference model, training corpus, and student model configuration, the approach loses much of its practical value. The cost of running the necessary ablation studies—training dozens of models with different pruning configurations to identify the best band—could easily outweigh the savings from the pruning itself, especially at the scales where pruning is most likely to be deployed (where training even a single model is expensive).

What evidence exists in the paper. The paper's own results demonstrate this sensitivity. Figure 2 shows that the optimal retention percentage changes with reference model size: for the 124M model, 70% retention is best; for 13B and 52B models, 30% and 50% retention outperform 70%. Figure 5 shows that the magnitude of improvement depends on reference model training data (Wikipedia-trained vs. CommonCrawl-trained). Figure 6 shows that the pattern changes with reference model checkpoint: the 14% checkpoint shows essentially no discriminatory power at all, while the 55% checkpoint recovers much of the fully-trained model's pattern. The paper does not present these variations as a limitation but rather as evidence that "reference model quality matters." However, they equally demonstrate that the pruning strategy is not portable—changing the reference model changes the optimal strategy.

Mitigation status. The paper does not address this limitation directly. There is no experiment testing whether the optimal percentile band for one reference model transfers to another reference model (e.g., can the optimal band derived from the 52B model be applied to scores from a 6B model and still outperform random pruning?). There is no discussion of how a practitioner should choose the pruning configuration without running their own ablation. The extensive experimental sweep in the paper is presented as evidence for the method's effectiveness, but it also demonstrates the method's sensitivity—and the paper does not provide guidance on navigating this sensitivity in practice. This leaves a gap between the paper's empirical findings and their deployability.

The Evaluation Is Limited to a Single Corpus, Single Data Distribution, and Single Model Family

The assumption or constraint. All experiments—both reference model scoring and student model training—use the CommonCrawl May 2022 snapshot as the data source. The test set is drawn from the same CommonCrawl snapshot with "identical prefiltering as the training data" (Section 3.4). The student models are all GPT-style autoregressive decoder-only Transformers within a single model family (the paper's in-house architecture, leveraging Cohere's infrastructure). While the paper uses multiple reference model sizes (124M through 52B) and two student model sizes (124M, 1.5B), all models share the same architectural paradigm, tokenizer (Byte Pair Encoding with 51,200 vocabulary), and context length (2048 tokens). The paper tests only one type of task (language modeling perplexity on in-distribution text, plus six GLUE classification tasks for a subset of configurations).

The consequence. Three separate but related generalization gaps exist:

  1. Corpus generalization. CommonCrawl is a specific type of web-scraped text with particular noise characteristics (boilerplate, legal disclaimers, product listings, broken markup). The finding that "middle perplexity" is optimal may not generalize to corpora with different noise profiles. A carefully curated corpus like Wikipedia might have a different optimal band (perhaps the top/hardest examples are most valuable when there is little genuine noise). A code-heavy corpus might have entirely different relationships between perplexity and training utility. A multilingual corpus might require per-language scoring.

  2. Distribution generalization. The test set is drawn from the same CommonCrawl snapshot as the training data. The pruning strategy is optimized to improve perplexity on this specific distribution, which measures how well the model predicts tokens from CommonCrawl-like text. If the goal of pretraining is to produce a model that generalizes to many downstream distributions (code, scientific text, dialogue, etc.), then optimizing for CommonCrawl perplexity may not align with optimizing for downstream utility. The paper acknowledges this indirectly by evaluating on GLUE (Section 4.7), but GLUE is a limited set of relatively homogeneous classification tasks and the best pruning variant for GLUE (often EL2N) is not the best variant for perplexity (perplexity-based), suggesting a tradeoff that is not explored.

  3. Architecture and scale generalization. All models are within one architectural family. The finding that pruning benefits persist at 1.5B parameters (Section 4.6) is suggestive but limited by the confounding of scale and training duration (see Section 5's Critical Assessment). No experiments test pruning on encoder-decoder architectures, mixture-of-experts models, or models with different tokenization strategies.

What evidence exists in the paper. The paper provides no cross-corpus evaluation—no training on one corpus and testing on another, no pruning of a different corpus (e.g., The Pile) to test whether the "middle subset" finding replicates. The only data-related ablation is the Wikipedia-trained reference model experiment (Section 4.4, Figure 5), which changes the reference model's training data but keeps the training and evaluation data fixed. The authors state they "perform extensive experiments evaluating models ranging from 124M to 1.5B parameters across different pretrained corpora" (Section 1, Contribution 1), but "different pretrained corpora" refers to the Wikipedia vs. CommonCrawl reference model data, not to different corpora used for pruning or training the student models.

The absence of a direct comparison to a model trained on a known-clean corpus (e.g., pruning CommonCrawl and comparing to training on Wikipedia alone at the same data volume) is a missed opportunity. Such a comparison would help separate the benefit of pruning from the benefit of simply having higher-quality source data.

Mitigation status. The paper does not discuss these generalization gaps as limitations. The framing in Section 1 positions the work as a foundation for "unexplored strategies in automatically curating high quality corpora," implying that generalization is future work. The practical recommendation that practitioners can use "off the shelf models for computing perplexity" (Section 4.5) implicitly assumes that the findings transfer across model families, but this assumption is untested.

No Comparison Against the Dominant Paradigm of Rule-Based Pruning

The assumption or constraint. The paper frames rule-based heuristics as the status quo against which metric-based pruning should be compared, stating that "these hand-curated filters can eliminate certain noisy examples, [but] they are not a substitute for a measure of 'quality' for individual training examples" (Section 1). The methodology section notes that the CommonCrawl data used in the experiments is "prefiltered using a combination of automatic and hand-crafted filters" (Section 3.2), meaning all experiments—including the no-pruning baseline—operate on data that has already undergone the standard rule-based filtering pipeline. The paper then applies its metric-based pruning on top of this prefiltered data and compares the results to a no-pruning baseline that also uses the prefiltered data.

The consequence. The paper never directly compares its metric-based pruning approach against the alternative of simply applying more aggressive rule-based filtering. The relevant practical question for a practitioner is not "should I use rule-based filtering or perplexity-based pruning?" but rather "given that I am already doing rule-based filtering, should I invest additional effort in metric-based pruning, or should I simply tighten my existing heuristic thresholds?" The paper's experimental design cannot answer this question because it never trains a model on data filtered with stricter rule-based heuristics (e.g., more aggressive deduplication, higher language-detection confidence thresholds, more restrictive blocklists) and compares that model to one trained on perplexity-pruned data at the same data volume.

This matters because rule-based filtering is computationally cheaper than perplexity scoring (heuristics like "remove documents with >50% non-alphabetic characters" require no model inference) and is already integrated into standard data processing pipelines. If tightening existing heuristics achieves similar improvements to perplexity-based pruning at a fraction of the computational cost, then the practical value of the paper's contribution is substantially reduced. The paper's argument that metric-based pruning captures something beyond what heuristics can capture—"a measure of 'quality' for individual training examples"—is plausible but empirically unverified.

What evidence exists in the paper. The paper provides no ablation or comparison involving different rule-based filtering strengths. The prefiltering step is described as applying "a combination of automatic and hand-crafted filters... similar to deduplication steps seen in Taylor et al. (2022); Kocetkov et al. (2022)" (Section 3.2), but the specific filters, their thresholds, and their individual effects on dataset size and model performance are not reported. There is no experiment that varies the strength of these filters and measures whether further rule-based filtering approaches the performance of metric-based pruning. The random pruning baseline (Section 2.1.4) tests whether removing data indiscriminately affects performance, but this is not equivalent to comparing against smarter heuristic-based removal.

Mitigation status. The paper does not acknowledge this as a gap. The related work section (Section 5.1) discusses rule-based filtering approaches and their limitations, but the experimental section does not engage with them as baselines. The framing positions metric-based pruning as a complement to rule-based filtering ("we aim to further improve data quality beyond common rule-based filters," Section 3.2), which is a reasonable approach, but the absence of a head-to-head comparison means the marginal value of metric-based pruning over further heuristic tuning remains unquantified. Future work that compares the best perplexity-based pruning configuration against the best achievable rule-based filtering at the same data retention percentage would directly address this gap.

The Trained Models Are Never Evaluated on Generative or Knowledge-Intensive Tasks

The assumption or constraint. The paper's primary evaluation metric is perplexity on a held-out CommonCrawl test set—a measure of how well the trained model predicts the next token in text drawn from the same distribution as the training data. The only downstream evaluation is fine-tuning on six GLUE classification tasks (SST-2, MRPC, QQP, QNLI, RTE, WNLI). GLUE tasks are sentence-level or sentence-pair classification problems that primarily test a model's ability to produce useful representations for discriminative tasks. They do not test the model's generative capabilities, factual knowledge, reasoning abilities, or few-shot learning performance—the capabilities that are typically the primary motivation for pretraining large language models and that are most likely to be affected by data pruning decisions.

The consequence. The paper cannot speak to whether perplexity-based pruning preserves or degrades the capabilities that practitioners most care about. A model trained on the "middle 50%" of the perplexity distribution might achieve better language modeling perplexity—meaning it is better at predicting the next token of CommonCrawl-like text—but perform worse on tasks requiring factual recall, because the pruning may have disproportionately removed text containing factual information. The examples in Appendix B, Tables 3-5, are suggestive: low-perplexity text includes legal boilerplate and standardized disclaimers (unlikely to contain much factual knowledge), middle-perplexity text includes local descriptions and narrative content (moderate factual density), and high-perplexity text includes technical specifications and product listings (potentially high factual density). If the high-perplexity tail contains a disproportionate share of entity-rich, fact-dense text, pruning it could degrade the model's knowledge base even as it improves perplexity.

This limitation is particularly consequential given the paper's framing. The introduction positions the work as relevant to "the development of large language models in recent years" (Section 1) and situates it within the context of LLM pretraining at scale. Yet the evaluation suite—perplexity on in-distribution web text plus six classification tasks—falls far short of what is standard for evaluating LLMs. There is no measurement of zero-shot or few-shot performance on benchmarks like MMLU, HellaSwag, TriviaQA, or any other knowledge or reasoning benchmark.

What evidence exists in the paper. The GLUE results in Table 2 provide the only downstream signal, and they show that pruning does not systematically harm classification performance—in fact, pruned models often outperform the baseline. But GLUE tasks are not knowledge-intensive; they primarily require understanding of sentence-level semantics and relationships, which may be preserved even if factual knowledge is degraded. The paper provides no experiments on generation quality, factuality, or knowledge recall. There is no discussion of whether the perplexity improvements translate to improvements on the kinds of tasks that motivate LLM development.

Mitigation status. Not addressed. The paper does not acknowledge the gap between its evaluation suite and the capabilities that make LLMs useful. The title and framing emphasize "pretraining LLMs at scale," which implies relevance to the full range of LLM capabilities, but the evaluation is restricted to metrics that were standard for smaller language models before the era of large-scale LLMs and instruction tuning. Including even a small set of knowledge-intensive or generative benchmarks would substantially strengthen the claim that pruning improves—or at least does not harm—the capabilities that matter for deployment.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new architecture, training algorithm, or model. It proposes a diagnostic methodology—the three-way (bottom/middle/top) subset selection framework—and uses it to discover that the relationship between data quality scores and training utility is fundamentally non-monotonic. The implications ripple outward from this single finding.

A reframing of data quality from absolute to relational. The dominant assumption in data curation—both in rule-based heuristics and in prior metric-based work like Muennighoff et al. (2023) and Laurençon et al. (2023)—has been that quality is an intrinsic property of text: clean, well-formed, Wikipedia-like prose is "good"; noisy, boilerplate, or strange text is "bad." This paper dismantles that assumption by demonstrating that the text a capable reference model finds least surprising (lowest perplexity) is the least useful for training, while text it finds moderately surprising (middle perplexity) is the most useful. Quality is not in the text; quality is in the gap between what the text contains and what the model already knows. This insight, if it holds beyond CommonCrawl, should change how the field thinks about corpus construction: the goal is not to find "clean" data but to find learnable data—text that contains information the model can acquire but hasn't yet.

A reconciliation of contradictory intuitions. The paper resolves a latent tension in the literature that prior work had not even clearly articulated. On one side, the scaling laws tradition (Kaplan et al., 2020) and engineering practice treat more data as strictly better. On the other, practitioners have long observed that some data seems "low quality" and that careful filtering helps—but the filtering methods (deduplication, blocklists, length thresholds) were ad hoc and their benefits inconsistent (Black et al., 2022; Biderman et al., 2023b finding no benefit). This paper provides a unified explanation: more data is better only when the additional data falls into the informative middle region of the quality distribution. Adding data from the low-perplexity tail (formulaic boilerplate) or the high-perplexity tail (malformed text) degrades performance—not because the data is "bad" in some absolute sense, but because it displaces more informative examples from the training budget or teaches the model patterns that don't generalize. The inconsistent results from rule-based filtering make sense: rule-based filters are too crude to isolate the informative middle, so their effect depends on whether, in a particular corpus and at a particular threshold, they happen to remove more tail examples than middle examples.

A shift in the cost-benefit calculus of "more data." The paper's most economically significant implication is that, for models within the capability range tested (up to 1.5B parameters), additional data collection and processing may be strictly worse than investing in better data selection. The finding that a model trained on 50% of the data (selected by perplexity) outperforms a model trained on 100% of the data means the remaining 50% was actively harmful, not merely useless. For organizations planning their data strategy, this suggests a reallocation: reduce crawling and storage budgets, and invest instead in reference model inference to score and prune existing corpora. The 4×4\times efficiency improvement in the abstract (30% of data matching full-dataset performance) is a concrete multiplier that should inform resource allocation.

Which research directions become more attractive. The paper strongly suggests that verifier quality is the bottleneck for data pruning—just as the concurrently analyzed test-time compute paper found verifier over-optimization as the bottleneck for inference scaling. The monotonic improvement from larger reference models (Figure 2: 52B > 13B > 6B > 124M) and from cleaner reference training data (Figure 5: Wikipedia > CommonCrawl) means that every advance in model capability automatically improves data pruning—a bootstrap dynamic that makes continued investment in better models doubly valuable. Research on training better reference models (scale, cleaner data, better architectures) now has an additional justification: those models will produce better pruning signals, enabling more efficient training of the next generation.

Which research directions become less attractive. The paper casts doubt on the value of complex, computationally intensive quality metrics that aim to capture learning dynamics beyond what a well-calibrated probability model provides. EL2N requires storing and processing full vocabulary distributions; the ensemble variant requires training 10 reference models; memorization requires greedy generation and exact-match comparison. None of these outperformed perplexity—and two of them (EL2N, memorization) barely outperformed random pruning. This is a strong negative result for a research program focused on developing ever-more-sophisticated quality scores. The implication is that the field should redirect effort from designing new metrics to understanding why perplexity works so well and improving the models that produce perplexity scores.

The paper also reduces enthusiasm for rule-based filtering as a first-class research direction. The finding that the "easiest" examples—which would pass any reasonable rule-based quality filter (they are well-formed, grammatical English)—are the least useful for training means that no combination of hand-crafted heuristics can replicate what perplexity-based scoring achieves. Rule-based filters operate on surface properties; model-based scoring operates on information content relative to what the model knows. The paper's contribution is not a refinement of rule-based filtering but a demonstration of its inherent ceiling.

A new diagnostic tool. Beyond its substantive findings, the paper's three-way (bottom/middle/top) selection framework provides the field with a general diagnostic for probing the relationship between any scoring metric and training utility. Before this work, the standard approach was to threshold a quality metric and compare the pruned model against a baseline—a design that conflates the effect of the threshold with the effect of the metric. By independently testing three regions of the score distribution, the paper separates these effects and reveals the shape of the utility curve. This framework can be applied to any new quality metric, any new corpus, and any new model family to characterize whether and where pruning is beneficial. It is a methodological contribution that should become standard practice in data pruning research.

Follow-Up Research This Work Enables

Are "useful hard" examples and "noisy hard" examples separable by a second quality dimension? The paper's framework treats the high-perplexity tail as a single mass to be pruned, but Appendix B examples suggest it contains both malformed text (broken product listings) and well-formed but domain-specific technical content (cement manufacturing specifications). These two types of sequences likely have very different training utility, but perplexity alone conflates them. A natural follow-up would train a second reference model—perhaps one fine-tuned on a curated corpus—and use its perplexity scores as a second dimension: sequences with high perplexity under the base model but low perplexity under the curated model might represent "hard but high-quality" text, while sequences with high perplexity under both might represent genuine noise. The experiment would test whether retaining this "high base perplexity / low curated perplexity" quadrant improves performance beyond the middle-band strategy, directly addressing the conflation limitation in Section 6.

Does the "middle subset" finding generalize across corpora with different noise profiles? The paper's central result—middle perplexity is optimal—is demonstrated on CommonCrawl only. A high-priority replication would apply the same three-way selection framework to corpora with different noise characteristics: Wikipedia (very low noise, where the "top" subset might be optimal because there is little malformed text to conflate with genuinely hard examples), The Pile (multi-domain, where the optimal band might differ by subdomain), and a deliberately degraded corpus (where known quantities of synthetic noise are injected, testing whether the optimal band shifts predictably). The experiment would use the same 52B reference model across corpora to isolate the effect of corpus properties from reference model properties. A negative result—the middle subset is not optimal on Wikipedia—would not invalidate the paper but would establish important boundary conditions on when perplexity-based pruning helps.

What is the optimal reference-model-to-student-model size ratio, and does it change with scale? The paper demonstrates that larger reference models produce better pruning signals (52B > 13B > 6B > 124M, Figure 2) but tests this across less than an order of magnitude in reference model scale and only for 124M student models. A systematic scaling study would vary the reference model size (e.g., 1B, 3B, 7B, 13B, 30B, 70B parameters) and the student model size (e.g., 350M, 1.5B, 3B, 7B) independently, measuring whether the pruning benefit from a given reference model size saturates at some ratio of reference-to-student parameters. If there are diminishing returns—a 13B reference model produces effectively the same pruning quality as a 52B reference model for a 1.5B student—this would have immediate practical implications for deployment: practitioners could use smaller, cheaper reference models without loss.

Can the scoring overhead be amortized or eliminated by training a difficulty predictor directly from text? The paper's single largest practical limitation is the computational cost of scoring the entire training corpus with a reference model. A natural extension, flagged in Section 3.2's discussion of difficulty estimation cost, would be to train a lightweight classifier—perhaps a small model distilled from the 52B reference model's perplexity scores—that takes raw text as input and predicts which percentile band it falls into, without requiring a full autoregressive forward pass. The experiment would compare the pruning quality achieved by the distilled classifier against the full reference model scoring at the same data retention percentage. If a distilled classifier can achieve, say, 90% of the reference model's pruning benefit at 1% of the computational cost, the approach becomes immediately deployable at scale. The paper's finding that the 55% checkpoint is sufficient for scoring (Section 4.5) already hints that the pruning signal is learnable from limited computation.

Does adaptive or curriculum-based pruning outperform static pruning, and at what computational cost? The paper explicitly limits itself to static pruning—scores computed once before training—noting that dynamic pruning is "computationally infeasible at pretraining scale" (Section 2). But a middle ground exists: re-score the dataset at one or two intermediate checkpoints during student model training and re-prune based on the updated scores. The hypothesis would be that as the student model learns, its own perplexity scores become a better guide to what data is still useful—early in training, the middle perplexity band is optimal (as the paper shows), but later in training, perhaps the harder examples become more valuable as the model saturates on the easy ones. The experiment would compare static pruning (scores from a fixed reference model, applied once) against re-pruning at, say, 25%, 50%, and 75% of total training steps using the student model's own perplexity at those checkpoints. The cost of re-scoring is amortized over fewer sequences (the already-pruned dataset) and could reuse the student model itself, eliminating the need for a separate reference model.

Does perplexity-based pruning preserve or degrade factual knowledge and reasoning capabilities? The paper's evaluation is limited to language modeling perplexity and six GLUE classification tasks—neither of which tests the knowledge-intensive capabilities that motivate large-scale pretraining. A critical stress-test would train models on the same perplexity-pruned subsets (middle 50% and full dataset vs. no-pruning baseline) at a scale where knowledge benchmarks are meaningful (e.g., 3B parameters) and evaluate on standard knowledge and reasoning tasks: MMLU (multi-domain factual knowledge), TriviaQA (closed-book QA), HellaSwag (commonsense reasoning), and perhaps a few-shot generative evaluation. The paper's Appendix B examples suggest that the high-perplexity tail includes entity-rich technical specifications and product listings that might contain factual knowledge, raising the concern that perplexity-based pruning could improve language modeling fluency at the cost of world knowledge. A negative result—perplexity-pruned models underperform on knowledge tasks—would not invalidate the paper's contribution but would establish that different pruning strategies are needed for different downstream objectives, which is information practitioners need.

Practical Applications and Downstream Use Cases

Cost-efficient pretraining at scale for mid-sized organizations. An organization training a 7B parameter model from scratch on web-scraped data—a plausible scenario for a startup or research lab with a multi-million-dollar compute budget—can apply the paper's methodology today: use an off-the-shelf larger model (e.g., an open-source 13B or 70B model) to score their training corpus by perplexity, retain the middle 50% of the distribution, and train on half the data. The paper's results at 124M parameters (0.97% perplexity improvement at 50% retention, Figure 4) and at 1.5B parameters (~1.5% improvement, Figure 7) suggest the benefit is robust to model scale, though the exact improvement at 7B is untested. The compute savings are substantial: training on half the data reduces total training FLOPs by approximately 50% (modulo the cost of perplexity scoring, which is a single forward pass per sequence compared to forward-plus-backward passes during training). At the scale where pretraining costs millions of dollars, a 50% reduction is economically transformative—it can mean the difference between a project being feasible and not.

Data cleaning pipelines for LLM training data providers. Organizations that produce and distribute large pretraining corpora (e.g., CommonCrawl-derived datasets, domain-specific web crawls) currently rely on rule-based filtering to produce "clean" versions of their data. This paper provides a path to adding a model-based quality score as an additional metadata field on each document or sequence, enabling downstream users to apply their own selection thresholds without running reference model inference themselves. The data provider would score the corpus once using a large reference model (e.g., an open 70B parameter model) and distribute the scores alongside the text. Downstream users could then select their own percentile band based on their specific model size and training budget. This is analogous to how image datasets are sometimes distributed with precomputed CLIP embeddings—the scoring is a one-time cost borne by the provider, and the benefit accrues to all users. The paper's finding that larger reference models produce monotonically better signals (Figure 2) means the provider should use the largest feasible model, and the cost is amortized over many users.

Self-improving data curation loops for iterative model development. A research lab that releases a new base model every 6–12 months can use each generation's model as the reference model for pruning the training data of the next generation. The paper's bootstrap finding—larger reference models produce better pruning signals (Section 4.3), and cleaner reference training data produces better pruning signals (Section 4.4)—implies a virtuous cycle: generation-N model scores and prunes the data for generation-(N+1), which trains a better model, which scores and prunes the data for generation-(N+2), and so on. Each generation benefits both from improved model architecture and from improved data selection, with the latter bootstrapped from the former. The only additional cost per generation is the inference pass to score the corpus, which is small relative to the training cost. This turns data curation from a manual, labor-intensive process into an automatic, self-amplifying one, and the paper provides the empirical evidence that the cycle should produce compounding benefits.

Lightweight difficulty estimation for adaptive training curricula. While the paper's primary result is about data pruning (removing examples entirely), the same perplexity scores could be used to order training data without removing any. A curriculum that starts with middle-perplexity examples, then gradually introduces higher-perplexity examples, and finally exposes the model to the full distribution (including the easy boilerplate) might achieve faster convergence or better final performance than uniform random sampling. The paper's finding that early-checkpoint scoring is not discriminative (Section 4.5) but that mid-training checkpoint scoring works well suggests that the "right" curriculum might be discoverable from a partially trained reference model. The practical implementation would be straightforward: score the dataset once using a reference model, bin by perplexity percentile, and train with a non-uniform sampling schedule that favors the middle band early in training and broadens to include the tails later. Even if this does not improve final performance, it could improve training speed—reaching a given perplexity with fewer total steps—which translates directly to cost savings.

When to Prefer This Method

The paper's explicit positioning against alternatives is limited—it compares against rule-based heuristics (conceptually, in Section 5.1) and against alternative quality metrics (EL2N, memorization; experimentally, in Section 4.2), but does not provide a decision framework for practitioners choosing among data curation strategies. The following is therefore inferred from the paper's experimental results and qualifying statements, not presented as a direct recommendation from the authors.

  • Prefer perplexity-based static pruning with a large reference model when: (1) you are training a model on a noisy web-scraped corpus where rule-based filtering is already applied but performance remains suboptimal; (2) you have access to a substantially larger off-the-shelf pretrained model to use as the reference scorer (the paper shows monotonic benefit with reference model size up to 52B, Figure 2, and the bootstrap implication is that "bigger is better" continues); (3) you can tolerate the one-time computational cost of scoring the full training corpus with the reference model (single forward pass per sequence); (4) your primary evaluation metric aligns with language modeling perplexity on in-distribution text, or your downstream tasks are representation-based (the GLUE results in Table 2 suggest pruning does not harm classification transfer).

  • Consider alternatives to perplexity-based pruning when: (1) your reference model is small or poorly calibrated relative to the training data distribution (the 124M reference model in Figure 2 shows much weaker pruning benefits—the method's effectiveness is strongly dependent on reference model quality); (2) your corpus has very different noise characteristics than CommonCrawl (the paper provides no cross-corpus validation, and the "middle subset is optimal" finding may not transfer to corpora where the tails contain different types of content); (3) your primary objective is knowledge-intensive downstream performance (factual recall, reasoning) rather than fluency or representation quality (the paper provides no evidence either way on this critical point); (4) the computational cost of reference model scoring approaches or exceeds the training cost savings from pruning (the paper does not quantify this tradeoff, but for small student models or extremely large corpora, it is a real concern).

  • Avoid EL2N and memorization as pruning metrics, given current evidence. The paper shows that EL2N and memorization "achieve similar performances to random pruning despite being far more computationally expensive" (Section 4.2). Investing in these metrics over perplexity is not supported by the presented data. This is a strong negative recommendation, not a neutral "more research is needed"—the paper's experiments are sufficiently thorough that EL2N and memorization, as implemented here, are dead ends for data pruning unless future work demonstrates a specific regime where they outperform perplexity (which the paper did not find across 10–70% retention, bottom/middle/top selection, or early/late checkpoint variants).