DataDecide: How to Predict Best Pretraining Data with Small Experiments

🎯 Pitch

A single 150M-parameter model ranks pretraining datasets just as well as complex scaling laws for picking the best 1B model (~80% accuracy), making elaborate extrapolation methods unnecessary. Even code benchmarks like HumanEval become predictable at tiny scale when using continuous likelihood metrics instead of pass/fail accuracy, requiring just 0.01% of the target compute.

1. Executive Summary

This paper empirically studies how to predict which pretraining dataset will yield the best downstream performance at a large target scale using only small-scale experiments. The authors release DATADECIDE, the most extensive open suite of models over differences in data and scale, comprising 1,050 models pretrained on 25 data recipes across 14 sizes (4M–1B parameters) with 3 random seeds, evaluated on the OLMES benchmark suite. The paper examines two families of prediction methods—ranking single scale experiments (training small models on each data recipe and assuming the winner extrapolates) and extrapolating scaling laws (fitting performance trends across multiple small scales to forecast larger-scale outcomes, chaining a compute-to-loss function with a loss-to-accuracy sigmoid)—measuring success via decision accuracy (the fraction of pairwise data-recipe comparisons correctly predicted). The central finding is that ranking models at a single small size (e.g., 150M parameters) already achieves ~80% decision accuracy on the OLMES macro-average at the 1B target scale, a baseline that none of eight scaling-law variants exceed on the compute-to-decision-accuracy frontier. Additionally, using continuous proxy metrics—specifically CORRECT PROB and TOTAL PROB (the likelihood of correct or all answer options, respectively)—makes benchmarks including MMLU, ARC, and even code tasks like HumanEval and MBPP >80% predictable at the 1B scale with just 0.01% of the target compute, establishing that simple likelihood-based continuous signals often outperform discrete accuracy metrics for small-scale data decisions, especially for tasks where discrete accuracy remains near the noise floor.

2. Context and Motivation

The Core Problem: We Cannot Afford to Validate Our Data Decisions

The fundamental question this paper tackles is deceptively practical: if you are building a large language model and you have several candidate pretraining datasets to choose among, how should you decide which one is best without actually training the expensive large model on all candidates? This is, as the authors state in Section 1, "one of the most impactful development decisions" in LLM construction—the choice of pretraining corpus directly shapes everything the model learns—yet the standard workflow for making this decision has a glaring blind spot. Teams run small-scale experiments on their dataset candidates, pick the one that performs best, and then commit to training the large model on that chosen dataset. But they never train the large model on the losing candidates, so they never learn whether their small-scale decision was correct. The counterfactual remains unobserved.

This gap matters enormously because the entire premise of using small experiments rests on an unverified assumption: that relative performance rankings observed at small scale (e.g., a 150M parameter model trained on 15B tokens) will hold at the target large scale (e.g., a 1B parameter model trained on 100B tokens). If this assumption is wrong, teams could systematically pick suboptimal data recipes—investing millions of dollars in training runs that underperform what they could have achieved with a different dataset choice. Conversely, if the assumption is correct and teams do not know how correct, they may over-invest in expensive multi-scale scaling law approaches when a much cheaper single-scale comparison would suffice. Without a ground-truth testbed where the counterfactual is observed, the field cannot distinguish between these scenarios.

The paper's characterization of this problem in Section 1 is direct:

"So far decision-making approaches have only been validated without observing the counterfactual outcome, either by producing a single large model on the chosen decision with impressive performance or by low error in predicting the magnitude of observed performance of a small number of large models. Knowing what amount of error in predicting performance over scale is a low enough to actually make a correct decision among datasets, requires a suite of comparable models trained on many datasets."

This is the gap DATADECIDE fills. The paper provides the first open testbed where we can evaluate whether our small-scale decision-making methods actually identify the correct winning dataset, measured against ground truth established by training large models on all candidate datasets—including the ones that would normally be discarded.

Why This Problem Matters: The Economics of Pretraining Data Decisions

To understand the stakes, consider the practical workflow the paper describes. A team developing a new LLM must choose among data preparation options: different source mixtures, different quality filtering thresholds, different deduplication strategies, different domain ablation choices. Each option produces a different "data recipe" (the paper's term for a specific pretraining corpus configuration). Training a large model—even a 1B parameter model at 100B tokens—on a single recipe is expensive enough; training the full set of recipes at scale to empirically determine the best one would multiply that cost by the number of candidates, making exhaustive comparison infeasible.

This is not a hypothetical scenario. The paper explicitly references the workflow of DCLM (Li et al., 2024), a prominent data curation effort that used small-scale (150M parameter) experiments to drive ablation decisions and data filtering choices:

"DCLM also makes extensive use of ranking single scale experiments to drive improvement in data recipes (Li et al., 2024). They release their best data and a model trained on it, but do not release models from their decision making experiments and do not search over multiple recipes at their largest scale."

The DCLM team's goal was to produce a single best recipe, so they had no need to validate their intermediate decisions against counterfactuals. But from the perspective of methodology—how should the field as a whole make data decisions?—this leaves us uncertain. Were the decisions correct? Would a different small-scale methodology have identified a different (better) recipe? Without the counterfactual, we cannot know. DATADECIDE makes that counterfactual available by training large models on all 25 data recipes, not just the winning one.

The cost scale makes this problem's importance concrete. The paper reports that building DATADECIDE required approximately 820,000 H100 GPU hours (Impact Statement). Most research groups cannot afford to replicate even a fraction of this compute, let alone do it for each new data curation project. This means that any method validated on DATADECIDE has outsized practical value: it can be adopted with confidence by teams who lack the resources to run their own counterfactual validation.

Beyond cost, the problem has a methodological significance. The field of scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) has developed increasingly sophisticated tools for predicting model performance from scale. But these tools have been validated primarily on prediction error—how close the predicted numerical accuracy is to the observed accuracy. The paper argues that prediction error is the wrong metric when the downstream use case is making a decision. A method can have high prediction error in absolute terms but still correctly rank two datasets; conversely, a method can have low prediction error on average but systematically invert rankings in critical regions of scale. The shift from prediction error to decision accuracy as the evaluation metric is a conceptual reframing that matters for anyone who uses small experiments to make choices among alternatives.

Where Existing Approaches Fall Short

The paper identifies four categories of prior work and articulates specific limitations in each.

Single-scale ranking is widely used but unvalidated. As noted above, works like DCLM (Li et al., 2024) rely on the assumption that the best data recipe at a single small scale (e.g., 150M parameters) will remain the best at the target large scale. This is an assumption about the absence of ranking crossovers—cases where recipe A outperforms recipe B at small scale but B overtakes A at large scale. Crossovers can occur for legitimate reasons: a recipe with higher-quality but lower-quantity data might initially underperform a higher-quantity recipe because small models cannot yet fully exploit the quality signal, but surpass it at larger scales as the model's capacity to absorb nuanced patterns grows. The paper's own analysis confirms that such crossovers exist:

"A priori we know that ranking single scale experiments cannot correctly predict when the scaling trend of one data recipe overtakes another at scales between our small experiments and target scale. Such crossovers bound the decision accuracy of this constant approximation of performance. Nevertheless ranking single scale experiments sets a high baseline decision accuracy, implying relatively little crossover occurs. It is difficult to distinguish evaluation variance from true crossovers, but the scaling trends we empirically observe cross over frequently." (Section 3.2)

This is a crucial tension: crossovers do occur frequently enough to theoretically justify multi-scale methods, but single-scale ranking still performs surprisingly well (~80% decision accuracy on the OLMES macro-average). Without a testbed like DATADECIDE, the field could not quantify exactly how much crossovers cost in practice versus how much noise obscures them.

Scaling law extrapolation has been validated on prediction error, not decisions. A growing body of work extends scaling laws from predicting language modeling loss to predicting downstream task performance (Gadre et al., 2024; Bhagia et al., 2024; Dubey et al., 2024). The standard approach chains two functions: a power law mapping compute to loss (Equation 1 in the paper: $L(C) = \frac{A}{C^\alpha} + E$ ) followed by a sigmoid mapping loss to downstream accuracy (Equation 2: $\text{Acc}(L) = \frac{a}{1 + e^{-k(L-L_0)}} + b$ ). These methods are validated by measuring how close their extrapolated accuracy values are to observed values—relative or absolute prediction error. But the paper makes the point that low prediction error does not guarantee correct decisions. A scaling law could consistently underestimate two recipes' accuracies by the same amount, producing zero decision errors despite arbitrarily high absolute prediction error. Or it could fit noise in the small-scale observations, producing rank inversions that mislead decisions even with low prediction error.

"Unlike previous work, we also measure the impact of predictions on decisions about which data recipe is better than another." (Section 2.3)

This is a methodological innovation: by evaluating scaling law approaches on decision accuracy rather than prediction error, the paper closes the loop between how scaling laws are validated and how they are actually used. The finding that eight scaling law variants do not exceed the single-scale frontier (Figure 3) suggests that the additional complexity of multi-scale fitting is not paying off for decision-making, at least within the scale range studied.

Existing model suites lack the necessary scale and breadth for controlled data comparisons. The paper acknowledges prior suites that provide controlled comparisons across data recipes: Pythia (Biderman et al., 2023) offered 2 data recipes across compute scales up to $2 \times 10^{22}$ FLOPs; subsequent suites offered up to 6 recipes (Magnusson et al., 2024; Brandfonbrener et al., 2024). But these are insufficient for studying decision-making methodology:

"Although a wide variety of open-source pretraining corpora are available, the scaling behavior of data is difficult to assess from off-the-shelf models that vary simultaneously in data, optimizer, and modeling decisions." (Section 1)

The key limitation is confounds. Off-the-shelf models (e.g., comparing a public DCLM-trained model against a public Dolma-trained model) differ not only in pretraining data but also in model architecture, optimizer settings, learning rate schedules, and other implementation details. To isolate the effect of data, you need controlled pretraining experiments where everything except the data recipe is held constant. The prior suites did this, but with too few recipes (2–6) to study decision accuracy as a function of recipe characteristics. DATADECIDE's 25 recipes—spanning different source mixtures, deduplication strategies, quality filtering thresholds, and domain ablations—provide enough diversity to observe meaningful variation in how well different prediction methods perform.

The choice of evaluation metric at small scale has been under-explored. Prior work on scaling predictions typically uses the same metric at all scales: if you want to predict downstream accuracy at the target scale, you measure downstream accuracy at small scales. But the paper identifies a practical problem with this: discrete accuracy metrics can remain near zero or near random for small models on hard tasks, providing no signal for distinguishing among data recipes. Drawing on the observation from Schaeffer et al. (2023) that "continuous metrics can show smooth improvement even when discrete metrics exhibit apparent emergent jumps," the paper systematically tests whether continuous proxy metrics—likelihood-based measures computed from the same evaluation data—provide better signal at small scales. This is a practical insight that no prior data-decision work had explored systematically:

"We experiment with using continuous metrics at small scale as proxies of the accuracies selected by OLMES for each task (ACCURACY) at the target scale to improve decision accuracy." (Section 2.5)

The finding that CORRECT PROB and TOTAL PROB (simple average likelihoods of correct or all answer options) dramatically improve decision accuracy for several benchmarks, and even make previously unpredictable code tasks (HumanEval, MBPP) >80% predictable (Figure 6), is a concrete, actionable result that emerged directly from testing this hypothesis across DATADECIDE's breadth.

How This Paper Positions Itself

The paper positions itself as an empirical methodology benchmark rather than a proposal of a new prediction technique. This is a distinctive stance. Most scaling-law papers propose a new functional form or fitting procedure and validate it on a small number of target models. DATADECIDE instead provides the infrastructure to evaluate any prediction method on the decision-making task, with the explicit goal of enabling future researchers to test new approaches against the baselines established here.

The paper's framing in Section 1 makes this clear:

"Future research can extend DATADECIDE with little extra compute by running new evaluations on our checkpoints, pretraining additional small models to compare against the large target models we provide, or trying new prediction methods with lightweight manipulations such as smoothing and curve fitting on top of our released evaluation results."

This is positioning DATADECIDE as a shared resource for the community—analogous to how GLUE and SuperGLUE standardized evaluation of NLP models, or how Pythia standardized analysis of model internals across training. The release of over 30,000 model checkpoints, all evaluations, and all pretraining corpora means that researchers can test hypotheses about data-decision methodology without any additional pretraining.

The paper also positions itself relative to the scaling laws literature by introducing the concept of the compute-decision frontier (Figure 1, right panel). This is the empirical tradeoff curve between the compute spent on small experiments and the resulting decision accuracy. It functions as an envelope: any prediction method yields a point on or below this frontier, and better methods push the frontier up and to the left (higher accuracy with less compute). The paper's central empirical claim—that "no scaling law methods among 8 baselines exceed the compute-decision frontier of single-scale predictions" (Abstract)—is a statement about this frontier. Single-scale ranking establishes the current Pareto frontier; multi-scale methods, while theoretically more powerful (since they can model crossovers), fail to push beyond it because whatever advantage they gain from modeling scale-dependent ranking changes is offset by fitting noise in the additional observations.

The paper's ambitions are bounded but precise. It does not claim to have solved the data-decision problem. It does not claim that single-scale ranking is universally optimal. It claims to have built the testbed that makes it possible to measure how well any method performs at this task, and to have established strong baselines against which future methods can be compared. The finding that existing scaling law methods do not beat single-scale ranking is presented not as a discouraging conclusion but as an invitation:

"DATADECIDE can be used to iterate on future scaling law prediction methods." (Figure 3 caption)

This strikes a careful balance: the paper validates the common practice of single-scale experiments (it works surprisingly well), while simultaneously providing the tools needed to develop methods that go beyond it (by accurately modeling crossovers without being misled by noise). In this sense, DATADECIDE is both a practical recommendation for practitioners ("you can trust your small-scale ablations more than you might have feared") and a research instrument for methodologists ("but here is exactly where and how those ablations fail, and here is the platform for building something better").

3. Technical Approach

3.1 Reader Orientation

This paper builds a measurement infrastructure, not a new prediction algorithm. The core idea is that by pretraining a large suite of models where every data recipe is trained all the way to the target scale (including the "losing" recipes that would normally be discarded), we can finally measure whether small-scale experiments actually identify the correct best dataset—and whether fancier multi-scale scaling law methods are worth their extra cost over the simple approach of ranking models at a single small size. The problem it solves is the counterfactual validation gap: before DATADECIDE, no one knew whether the common practice of running ablations at 150M parameters and assuming the winner extrapolates to 1B parameters was actually reliable, because no one had trained the 1B models on all the losing recipes to check.

3.2 Big-Picture Architecture

The system has four major components, arranged in a pipeline that flows from pretraining through evaluation to prediction assessment:

Data Recipes (25 variants) — the candidate pretraining corpora being compared. These are the "treatments" in the experimental design, spanning source mixtures, quality filtering thresholds, deduplication strategies, and domain ablations across popular corpora like Dolma, DCLM, C4, FineWeb, and Falcon RefinedWeb.
Model Ladder (14 scales × 3 seeds) — a grid of 1,050 controlled pretraining runs where each of the 25 data recipes is trained at 14 model sizes from 4M to 1B parameters, with 3 random seeds per recipe per size (seeds 2 and 3 are truncated at 25% of target compute for non-target sizes to save budget). All models use OLMo's programmatic configuration system to ensure hyperparameters scale properly with model size, eliminating confounds.
OLMES Evaluation Suite (10 benchmarks × multiple metrics) — a standardized downstream evaluation applied to every checkpoint, producing both discrete accuracy scores (which match what practitioners care about at deployment scale) and continuous proxy metrics (likelihoods of correct and all answer options, margins, normalized probabilities) that provide finer-grained signal at small scales where discrete accuracy may be near the noise floor.
Prediction Assessment Framework — the meta-evaluation layer that takes evaluation results from small-scale models, applies a prediction method (single-scale ranking or scaling law extrapolation), produces a predicted ranking of data recipes for the 1B target scale, and scores that prediction against the ground-truth ranking established by actually training and evaluating the 1B models on all 25 recipes. The key metric is decision accuracy: the fraction of pairwise data-recipe comparisons where the prediction correctly identifies which recipe wins at the target scale.

Information flows: Data recipe → pretraining run at specified scale and seed → intermediate and final checkpoints → OLMES evaluation (producing both ACCURACY and continuous proxy metrics) → prediction method (uses small-scale evaluations as input, outputs predicted 1B performance) → decision accuracy computation (compares predictions against observed 1B performance). The entire pipeline is released openly: corpora, model checkpoints (~30K total), and all evaluation results.

3.3 Roadmap for the Deep Dive

First, the DATADECIDE suite itself—the 25 data recipes, the 14-scale model ladder, the training protocol, and the random seed strategy—because this is the experimental bedrock on which all analyses rest. Without understanding what is being compared and how the comparisons are controlled, the prediction results lack meaning.
Second, the OLMES evaluation framework and the proxy metrics (CORRECT PROB, TOTAL PROB, MARGIN, NORM CORRECT PROB, and their length normalization variants), because the choice of evaluation signal fundamentally determines what prediction methods can observe at small scale. The proxy metrics are one of the paper's key practical contributions, so their definitions and properties need precise treatment.
Third, the two families of prediction methods—single-scale ranking and multi-scale scaling law extrapolation—including the specific functional forms, fitting procedures, and the eight scaling law variants tested. This establishes the baseline methods whose decision accuracy will be measured.
Fourth, the decision accuracy metric itself: how pairwise comparisons are defined, how ground-truth rankings are established from the 1B target models (including the handling of 3-seed variance), and how compute cost is normalized as a fraction of target FLOPs. This is the paper's methodological innovation—the shift from prediction error to decision accuracy—and it needs careful operational definition.
Fifth, the experimental design choices: why 100 tokens per parameter (5× Chinchilla), why truncate non-default seeds at 25%, why the OLMES cloze formulation, why these 25 recipes. Understanding the motivations behind these choices reveals what the experiments can and cannot tell us.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical measurement paper. Its core idea is that the reliability of small-scale data decisions can be measured by building a testbed where the counterfactual (training large models on the losing recipes) is actually observed, and that this measurement reveals both surprising strengths of simple methods (single-scale ranking works better than expected) and actionable insights about evaluation metrics (continuous likelihoods vastly outperform discrete accuracy at small scales for many tasks).

The DATADECIDE Suite: 25 Data Recipes

The paper trains models on 25 distinct "data recipes"—pretraining corpora that differ along dimensions relevant to real-world data curation decisions. These recipes are described in Table 1 and fall into several families:

Dolma variants (5 recipes). Starting from Dolma 1.7 (Soldaini et al., 2024), a 2.3-trillion-token corpus sampling common LM sources for open research, the paper creates four ablation recipes by removing specific domains: "No code" (removes code data), "No math/code" (removes both), "No Reddit" (removes Reddit-derived content), and "No Flan" (removes instruction-tuning data). These test whether domain presence matters and whether small-scale experiments can detect domain-removal effects that only manifest at scale. A fifth recipe, "Dolma 1.6++," combines Dolma 1.6 with additional sources from Dolma 1.7: RedPajama's arxiv subset, openwebmath, algebraic stack, flan, starcoder, and falcon.

C4 (1 recipe). The C4 dataset (Raffel et al., 2019) as prepared in Dolma 1.7—heuristically filtered from the April 2019 Common Crawl. This serves as a baseline representing older, simpler filtering approaches.

FineWeb variants (2 recipes). FineWeb-Pro (Zhou et al., 2024) features model-driven data cleaning, while FineWeb-Edu is the deduplicated educational subset from SmolLM-Corpus (Ben Allal et al., 2024). These test whether model-based quality filtering (a more recent, expensive approach) yields detectable gains over heuristic filtering, and whether educational focus specifically helps.

Falcon RefinedWeb and quality-filtered mixes (6 recipes). The base Falcon RefinedWeb corpus (Penedo et al., 2023) in Dolma 1.7 is derived from Common Crawl through June 2023 with more aggressive filtering and deduplication than C4. The paper then creates five additional recipes mixing Falcon with Dolma 1.7's Common Crawl and applying quality filtering: retaining the top 10% or 20% of documents according to a reproduced DCLM quality classifier, retaining the top 10% according to the original DCLM classifier, or retraining the classifier on a pre-release version of Tulu-v3 (Lambert et al., 2024) and retaining the top 10%. These recipes test the interaction between base corpus selection and quality filter threshold/implementation—a bread-and-butter ablation in practical data curation.

DCLM-Baseline and quality-filtered variants (7 recipes). Starting from DCLM-Baseline (Li et al., 2024)—a state-of-the-art Common Crawl corpus using best ablated deduplication, cleaning heuristics, and quality filtering—the paper creates recipes quality-filtered to the top 7% of DCLM-classified documents, further filtered to documents scoring 2+ or 3+ with the FineWeb-edu classifier, or filtered to the top 3% or 10% with the FineWeb-edu classifier alone, or filtered to the top 10% or 20% with the reproduced DCLM classifier. This systematically varies both the classifier (DCLM vs. FineWeb-edu) and the threshold (3%, 7%, 10%, 20%), creating a grid that tests how quality-filter aggressiveness trades off with data quantity and whether different classifiers produce recoverably different rankings.

Mixtures (3 recipes). Fractional combinations of DCLM-Baseline and Dolma 1.7, mixing proportions $\lambda \in \{25\%, 50\%, 75\%\}$ where the recipe is $\lambda$ DCLM-Baseline + $1-\lambda$ Dolma 1.7. These test whether small-scale experiments can correctly identify optimal mixing ratios—a problem of direct practical interest given works on data mixing laws (Ye et al., 2024; Kang et al., 2024).

Design rationale for recipe selection. The paper is explicit that these 25 recipes "aim to provide coverage of common data preparation choices" (Section 2.1). The coverage is organized around intervention types: source dataset selection (Dolma vs. C4 vs. Falcon vs. DCLM vs. FineWeb), domain ablation (removing code, math, Reddit, Flan), quality filtering (varying classifiers and thresholds), and mixing (varying proportions of two corpora). This structure means the decision-accuracy results can be interpreted not just as aggregate numbers but as revealing which types of data decisions are easier or harder to predict from small scale. For example, if quality-filter threshold decisions are reliably predictable at 150M while domain-ablation decisions frequently cross over, that tells practitioners where they can trust small-scale ablations and where they should invest in multi-scale analysis.

The Model Ladder: Controlled Scaling Across 14 Sizes

The paper needs models at many sizes to serve both as the "small experiments" whose predictions are evaluated and as the "target" whose performance is being predicted. The core design constraint is eliminating confounds: performance differences between models trained on different data recipes must be attributable to the data, not to suboptimal hyperparameters at particular scales or to inconsistent training recipes across sizes.

OLMo model ladder. The paper adopts the programmatic configuration system from OLMo (Groeneveld et al., 2024; OLMo et al., 2025; Bhagia et al., 2024), which provides pre-designed model architectures and hyperparameters for a range of parameter counts at a specified token-to-parameter ratio. The ladder uses heuristics from Porian et al. (2024) to set global batch size (in sequences per batch) and learning rate as functions of model scale, ensuring that these critical hyperparameters scale appropriately rather than being set to a single value that might favor some scales over others. The architectural parameters (number of layers, hidden dimension, number of attention heads, MLP dimension) were handpicked by OLMo developers at each scale to achieve the desired non-embedding parameter count. Table 2 in the appendix specifies every configuration: sequence length is uniformly 2048, MLP ratio is uniformly 8× the hidden dimension, and the specific (layers, hidden dim, heads, batch size, learning rate) tuples are given for all 14 sizes.

The 14 scales. The model sizes span from 4M to 1B non-embedding parameters: 4M, 6M, 8M, 10M, 14M, 16M, 20M, 60M, 90M, 150M, 300M, 530M, 750M, and 1B. Each is trained on $100 \times$ its parameter count in tokens—for example, the 150M model trains on 15B tokens, and the 1B model trains on 100B tokens. The paper describes this as "5× 'Chinchilla' (5×C) optimal ratio" (Section 2.1), referencing the finding from Hoffmann et al. (2022) that compute-optimal training uses roughly 20 tokens per parameter. The justification for overtraining is practical:

"We select a token to parameter ratio of 100, which at 5× 'Chinchilla' (5×C) optimal ratio captures the typical overtraining favored for inference savings." (Section 2.1)

This matters because the overtraining ratio affects scaling behavior: models trained far beyond the Chinchilla-optimal point may exhibit different crossovers between data recipes than models trained at the optimal ratio. The 5×C choice makes DATADECIDE's findings relevant to current practice (most deployed models are overtrained for inference efficiency) but means the specific decision-accuracy numbers might differ at other ratios.

Training steps. Table 2 lists the exact training steps and total tokens for each configuration. For example, the 150M model trains for 38,157 steps with a batch size of 192 sequences of length 2048, yielding $192 \times 2048 \times 38157 \approx 15.0$ billion tokens. The 1B model trains for 69,369 steps with batch size 704, yielding 100B tokens. Every configuration across all 25 recipes uses these identical step counts and hyperparameters—the only variable is the pretraining data.

Random seed strategy. This is a novel affordance that the paper highlights as differentiating DATADECIDE from prior suites:

"We also offer a novel affordance by including 3 random seed reruns for even our largest runs, to help quantify whether variation occurs due to random initialization and data order or differences in the distribution of data." (Section 1)

The seed strategy is asymmetric across scales because of compute constraints. All 1B models (the target scale) have 3 full reruns with different seeds for weight initialization and data order, trained all the way to 100B tokens. This is critical because it allows the "ground truth" ranking against which predictions are evaluated to account for run-to-run variance—the paper reports that "the standard deviation between runs at the 1B 5×C scale can be as high as 2% points of accuracy for some recipes on most tasks" (Section 2.1). If the ground-truth ranking used only a single seed, it might be dominated by noise rather than genuine recipe quality differences, and prediction methods could be penalized for failing to predict what is essentially random variation.

For all non-target model sizes (4M through 750M), only the default seed (seed 1) is trained to completion. Seeds 2 and 3 are terminated early after 25% of the target compute budget:

"other model sizes have second and third seed runs that are terminated early after 25% of the target compute budget. We train the 1B reruns all the way to completion to allow our target 'gold' rankings to account for run-to-run variance in evaluations due to weight initialization and data order." (Section 2.1)

The rationale for this asymmetry is cost: fully training all 14 sizes × 25 recipes × 3 seeds would multiply the already-substantial compute budget by 3. Truncating non-target reruns at 25% provides some information about seed variance at small scales without consuming excessive resources. However, this means that when the paper reports decision accuracy for single-scale predictions averaged over 3 seeds (as in Figure 1 right, where "Each point represents the average decision accuracy of a given method over 3 prediction attempts using small models with different random seeds"), the small-scale predictions from seeds 2 and 3 are based on partially trained models. The paper does not discuss how this partial training affects the comparability of seed-averaged results, though the decision-accuracy trends in Figure 2 (which show smooth, monotonic improvement with compute) suggest the effect is not severely distorting.

Compute measurement. The paper follows standard scaling-law conventions in measuring compute as theoretical FLOPs:

"We measure compute in terms of theoretical FLOPs following the simplifying assumption made in most scaling literature that the costs associated with training a model are captured well enough by FLOPs = 6ND, based solely on the number of parameters (N) and tokens trained (D)" (Section 2.3)

This is the standard approximation from Kaplan et al. (2020): each training token involves roughly 6N floating-point operations (2N for the forward pass, 4N for the backward pass, with the factor of 2 coming from the multiply-add count). The paper acknowledges this is a simplification that ignores attention-specific costs, data loading, and other overhead, but it provides a consistent, hardware-independent cost metric for comparing prediction methods.

Percent of target compute (%C). To normalize across prediction methods that use different numbers and sizes of small models, the paper expresses prediction cost as a fraction of the target training cost:

$\%C = \frac{c}{C} \times 100\%$

where $c$ is the total FLOPs of all small experiments used to make the prediction and $C$ is the FLOPs of training a single 1B-parameter model on 100B tokens (the target). For example, a prediction based on fully training a 150M model on 15B tokens uses $c = 6 \times 150\text{M} \times 15\text{B} = 1.35 \times 10^{19}$ FLOPs. The target cost is $C = 6 \times 1\text{B} \times 100\text{B} = 6 \times 10^{20}$ FLOPs. So $\%C = 1.35 \times 10^{19} / 6 \times 10^{20} \times 100 \approx 2.25\%$ . The x-axis of Figures 1–4 uses this metric on a log scale, spanning roughly 5 orders of magnitude from ~0.0001% (a few 4M-parameter checkpoints) to 100% (the full target training).

OLMES Evaluation and Proxy Metrics

The paper needs to evaluate every model checkpoint on a standardized set of downstream tasks to serve as both the prediction signal (at small scales) and the ground truth (at the target scale). The choice of evaluation suite and the specific metrics extracted from it are central to the paper's findings.

OLMES benchmark suite. The paper uses OLMES (Gu et al., 2024), a suite of 10 multiple-choice question answering benchmarks designed for standardized LM evaluation: MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019), ARC Challenge and ARC Easy (Clark et al., 2018), PIQA (Bisk et al., 2020), CommonsenseQA (Talmor et al., 2019), SocialIQA (Sap et al., 2019), OpenBookQA (Mihaylov et al., 2018), BoolQ (Clark et al., 2019), and WinoGrande (Sakaguchi et al., 2020). The selection rationale is pragmatic:

"We chose these tasks based on their appropriateness to our range of model scales, and one would have to select different tasks when targeting a larger scale." (Section 2.4)

This is a key caveat: the decision-accuracy findings are specific to these tasks at these scales. A model developer targeting a 70B model might evaluate on different benchmarks (e.g., GPQA, MATH, HumanEval) and would need to assess whether the DATADECIDE findings transfer. The paper explicitly invites such extensions: "DATADECIDE could be used to identify new evaluations that are sensitive within our range of scales" (Section 2.4).

Cloze formulation. All evaluations use OLMES's "cloze formulation" (CF), where the model is presented with the question and answer options, and the score is based on the probability the model assigns to each option's text continuation. This differs from the more common "multiple choice" formulation where the model selects among lettered options (A/B/C/D). The cloze formulation is chosen because it produces a continuous likelihood for each option, enabling the proxy metrics described below. The paper refers to the resulting discrete accuracy as ACCURACY, defined formally as:

$ \text{ACCURACY} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left[\arg\max_{c \in C^{(i)}} P(c \mid \text{context}_i) = c^{(i)}_{\text{correct}}\right]$ (Table 3)

where $C^{(i)}$ is the set of possible answer continuations for item $i$ (e.g., multiple choice options), $P(c \mid \text{context}_i)$ is the model's probability for continuation $c$ , and $c^{(i)}_{\text{correct}}$ is the correct option. The indicator is 1 when the model assigns highest probability to the correct continuation among the allowed set.

Length normalization. OLMES specifies per-task length normalization schemes. The paper notes that it diverges from OLMES only in using all available items rather than subsampling, to reduce variance. For the proxy metrics, likelihoods can be normalized either by the number of tokens in the continuation string (* per token) or by the number of characters (* per char). The paper states:

"Unless otherwise specified we use character length normalization." (Section 2.5)

and that this scheme "is empirically optimal for most of the tasks that we observe" (Section 3.3). The formal definition (Table 3 footnote): for token normalization, each continuation's log-likelihood is divided by its token count $\text{tokens}(c)$ ; for character normalization, divided by its character count $\text{chars}(c)$ . Character normalization is preferred because it avoids giving unintended advantage to longer continuations that simply distribute probability across more tokens.

The five proxy metrics. The paper defines five metrics (Table 3), all computed from the same underlying probabilities $P(c \mid \text{context})$ but aggregating them differently. Understanding their differences is essential because the paper's central practical finding is that switching from ACCURACY to CORRECT PROB or TOTAL PROB dramatically improves small-scale decision accuracy on many tasks.

CORRECT PROB:

$$\text{CORRECT PROB} = \frac{1}{N} \sum_{i=1}^{N} P\left(c^{(i)}_{\text{correct}} \mid \text{context}_i\right)$$

where $N$ is the number of items in the benchmark and $P(c^{(i)}_{\text{correct}} \mid \text{context}_i)$ is the model's probability assigned to the correct answer string for item $i$ .

What it computes: the average probability the model places on the correct continuation, regardless of what probabilities it assigns to incorrect options. This is a pure measure of how much probability mass the model gives to the right answer, on average. It does not penalize the model for also assigning probability to wrong answers—a model that gives 0.4 to the correct answer and 0.6 spread across three distractors gets the same CORRECT PROB as a model that gives 0.4 to the correct answer and 0.6 to a single extremely attractive distractor.

Why this form: this metric can detect improvements that ACCURACY misses when the correct answer's probability is increasing but has not yet crossed above the most-likely-incorrect threshold. For small models on hard tasks, the correct answer may reliably be the second or third most probable—ACCURACY scores this as zero, but CORRECT PROB can distinguish between a model giving the correct answer 0.01 vs. 0.3. This sensitivity to sub-threshold improvement is what makes CORRECT PROB a better predictor at small scales for tasks where discrete accuracy remains near floor.

TOTAL PROB:

$$\text{TOTAL PROB} = \frac{1}{N} \sum_{i=1}^{N} \sum_{c \in C^{(i)}} P(c \mid \text{context}_i)$$

where $C^{(i)}$ is the full set of possible continuations (correct + all incorrect options) for item $i$ .

What it computes: the average total probability the model places on all the provided answer options combined. This measures how much of the model's probability mass is concentrated within the answer set, as opposed to being spread across other tokens in the vocabulary. A model that is "confused" and places low probability on all options will have low TOTAL PROB; a model that has learned to recognize the task format and concentrates its predictions among the given choices will have high TOTAL PROB, even if it assigns the highest probability to the wrong choice.

Why this form: the paper hypothesizes that TOTAL PROB "may be interpretable as signal of a model having exposure to the domain of a given task in the form of higher likelihoods on incorrect but presumably relevant additional answers" (Section 3.3). In other words, TOTAL PROB captures task recognition even before task mastery: a model trained on more relevant pretraining data may elevate the probabilities of all plausible-sounding answer options (both correct and incorrect) relative to vocabulary-average tokens, and this elevation is detectable before the model can reliably distinguish correct from incorrect within that set. This makes it potentially a leading indicator of eventual accuracy.

MARGIN:

$$\text{MARGIN} = \frac{1}{N} \sum_{i=1}^{N} \left[P(c^{(i)}_{\text{correct}} \mid \text{context}_i) - \max_{c' \neq c^{(i)}_{\text{correct}} \in C^{(i)}} P(c' \mid \text{context}_i)\right]$$

where $\max_{c' \neq c^{(i)}_{\text{correct}}}$ is the highest probability assigned to any incorrect continuation for item $i$ .

What it computes: the average gap between the model's probability on the correct answer and its probability on the most-confidently-wrong answer. Positive values indicate the correct answer is favored; negative values indicate a distractor is favored. This is a continuous analog of ACCURACY: ACCURACY is the fraction of items where the margin is positive; MARGIN measures how positive or negative on average.

Why this form: MARGIN penalizes probability assigned to incorrect answers (unlike CORRECT PROB, which is indifferent to distractors), making it more directly aligned with the ACCURACY target metric—both care about the correct answer being more probable than alternatives. However, because it is continuous, it can show improvement before the margin crosses zero, providing earlier signal at small scales. The paper finds that MARGIN tends to track ACCURACY more closely than CORRECT PROB does, often showing similar decision accuracy at intermediate scales.

NORM CORRECT PROB:

$$\text{NORM CORRECT PROB} = \frac{1}{N} \sum_{i=1}^{N} \frac{P(c^{(i)}_{\text{correct}} \mid \text{context}_i)}{\sum_{c \in C^{(i)}} P(c \mid \text{context}_i)}$$

where the denominator is the sum over all continuations $C^{(i)}$ (same as TOTAL PROB for item $i$ ).

What it computes: the probability the model assigns to the correct answer conditioned on the model picking one of the provided options. This is a renormalized version of CORRECT PROB that divides by TOTAL PROB, converting "probability on correct answer" to "fraction of in-option-set probability that goes to correct." If TOTAL PROB is 0.5 and CORRECT PROB is 0.2, NORM CORRECT PROB is 0.4—the model gives 40% of its option-set mass to the right answer.

Why this form: this metric attempts to separate two signals being mixed in CORRECT PROB: how much the model "knows the task" (high TOTAL PROB) versus how well it "knows the answer" (high CORRECT PROB relative to TOTAL PROB). By conditioning on the option set, NORM CORRECT PROB isolates the discrimination component. The paper finds that this metric behaves similarly to MARGIN in tracking ACCURACY—it incorporates the penalty on incorrect-answer probability that CORRECT PROB lacks.

A critical design choice: all proxy metrics are used only as inputs to prediction methods at small scales. The target metric at 1B—the one being predicted—is always ACCURACY. This means CORRECT PROB at 150M is being used to predict ACCURACY at 1B, not CORRECT PROB at 1B. The proxy metrics are valued only to the extent that they better predict the downstream discrete metric. The paper makes this explicit by reporting decision accuracy against ACCURACY targets, and by showing (in Figure 4) that at the largest scales near 1B, ACCURACY itself becomes competitive with or superior to the proxy metrics—the proxies' advantage is specifically at small scales where the discrete metric is too noisy or near-floor to provide distinguishing signal.

Prediction Methods: Single-Scale and Multi-Scale

The paper evaluates two families of methods for using small-scale observations to predict large-scale performance. Both are presented as straightforward implementations of approaches common in the literature, not as novel proposals—the goal is to establish baselines, not to advance the state of the art in prediction methodology.

Ranking Single Scale Experiments (Single Scale). This is the simplest possible prediction method:

Choose a small model size (e.g., 150M parameters) and train one model per data recipe at that size.
Evaluate all models on the chosen benchmark/metric (e.g., OLMES macro-average ACCURACY, or a proxy metric like CORRECT PROB).
Rank the data recipes by their observed performance at that small scale.
Predict that this ranking will be preserved at the 1B target scale.

The mathematical operation is trivial: for each pair of data recipes $(A, B)$ , predict that $\hat{y}_A > \hat{y}_B$ if and only if $y_A^{\text{small}} > y_B^{\text{small}}$ , where $y^{\text{small}}$ is the observed performance at the chosen small scale. This is the constant-rankings assumption.

The paper notes that this method "is employed by work such as Li et al. (2024)" in the DCLM project, where 150M-parameter models were used to guide data filtering decisions that were then applied to much larger models. The rationale is cost: training 25 models at 150M is vastly cheaper than training 25 models at 1B, and if the rankings are stable across scale, the cheapest possible experiment suffices.

The paper varies which small scale is used for prediction—4M, 6M, 8M, ..., 750M—and also considers using intermediate checkpoints from a single training run at a given scale. For each scale and checkpoint, the compute cost $\%C$ is calculated based on the FLOPs consumed to reach that checkpoint. The resulting decision accuracy at each $\%C$ value traces out a curve (the colored points in Figure 1 right), showing how single-scale predictions improve as more compute is invested in the small experiment.

Extrapolating Scaling Laws (Multi Scale). Rather than assuming rankings are constant, this approach tries to model how performance changes with scale and extrapolate to the target. The paper follows the two-step method from Bhagia et al. (2024):

Step 1: Map compute to loss.

$$L(C) = \frac{A}{C^\alpha} + E$$ (Equation 1)

where $C$ is the training compute in FLOPs (represented as $6ND$ ), $L(C)$ is the predicted downstream task loss at that compute level, and $A, \alpha, E$ are fitted parameters: $A$ controls the overall scale of the learnable loss component, $\alpha$ controls the rate at which loss decreases with additional compute, and $E$ is the irreducible loss floor—the minimum possible loss even with infinite compute.

What it computes: a power-law plus constant mapping from training FLOPs to the model's cross-entropy loss on the evaluation task. This is the standard functional form from Kaplan et al. (2020) and Hoffmann et al. (2022), applied to downstream loss rather than language modeling loss. Given observations of $L$ at several $C$ values (from different model sizes or training durations), the three parameters are fit to minimize prediction error.

Why this form: the power law $A/C^\alpha$ captures the empirical observation that loss improvements diminish with scale—each doubling of compute yields progressively smaller loss reductions. The additive constant $E$ captures the irreducible Bayes error of the task, preventing the function from predicting loss below the fundamental difficulty level. Without $E$ , the function would approach zero as $C \to \infty$ , which is unrealistic for any finite-difficulty task.

The paper fits Equation 1 using only final, fully trained checkpoints—not intermediate checkpoints—because "accounting for the learning rate schedule's impact on intermediate checkpoints would require further parameters in the equation increasing the required number of observations and cost" (Section 2.2). To reduce noise, the paper averages the last 10% of checkpoints as the final observed loss for each training run.

Step 2: Map loss to accuracy.

$$\text{Acc}(L) = \frac{a}{1 + e^{-k(L - L_0)}} + b$$ (Equation 2)

where $L$ is the predicted downstream loss from Equation 1, $\text{Acc}(L)$ is the predicted downstream accuracy, and $a, b, k, L_0$ are fitted parameters. Together, $a + b$ is the upper asymptote (maximum accuracy achievable), $b$ is the lower asymptote (minimum accuracy—typically chance-level), $L_0$ is the inflection point where accuracy transitions from low to high, and $k$ controls the steepness of that transition.

What it computes: a sigmoidal (S-shaped) mapping from loss to accuracy. At very high loss (far left of $L_0$ ), accuracy approaches $b$ (chance). As loss decreases and passes through $L_0$ , accuracy rises rapidly. At very low loss (far right of $L_0$ ), accuracy approaches $a + b$ (the ceiling). The sigmoid functional form captures the intuitive relationship: loss improvements matter most in an intermediate regime where the model is neither completely lost nor already near-perfect.

Why this form: a linear mapping from loss to accuracy would be wrong at the extremes—loss can theoretically go to zero or infinity while accuracy is bounded between chance and 1.0. The sigmoid naturally handles these bounds. The parameterization with $a$ and $b$ separately (rather than fixing $b$ to the task's random-chance accuracy) allows the model to learn the effective lower asymptote from data, which may differ from theoretical chance if the task has systematic biases or if the evaluation setup introduces floor effects.

Unlike Step 1, Equation 2 is fitted on all observations including intermediate checkpoints, not just final ones. This is because the loss-to-accuracy mapping is expected to be stable throughout training—a model at a given loss value should have similar accuracy regardless of whether it achieved that loss early in training or late. Using intermediate checkpoints provides more data points for fitting the sigmoid without requiring additional full training runs.

Chaining the two steps. To predict accuracy at the target scale $C_{\text{target}}$ for a given data recipe:

Train models on that recipe at several smaller scales, obtaining observations $(C_i, L_i)$ and $(L_j, \text{Acc}_j)$ .
Fit Equation 1 to the $(C_i, L_i)$ pairs (final checkpoints only) to obtain $\hat{A}, \hat{\alpha}, \hat{E}$ .
Predict the target-scale loss: $\hat{L}_{\text{target}} = \hat{A}/C_{\text{target}}^{\hat{\alpha}} + \hat{E}$ .
Fit Equation 2 to all intermediate-checkpoint observations $(L_j, \text{Acc}_j)$ to obtain $\hat{a}, \hat{b}, \hat{k}, \hat{L}_0$ .
Predict the target-scale accuracy: $\widehat{\text{Acc}}_{\text{target}} = \hat{a}/(1 + e^{-\hat{k}(\hat{L}_{\text{target}} - \hat{L}_0)}) + \hat{b}$ .

This is repeated for each data recipe independently. The predicted ranking across recipes at $C_{\text{target}}$ is then determined by sorting the predicted accuracies $\widehat{\text{Acc}}_{\text{target}}$ .

Eight scaling law variants. The paper tests several modifications to this baseline, motivated by ablations in prior work (Bhagia et al., 2024; Choshen et al., 2024):

Baseline 3-parameter: the default described above (fits $A, \alpha, E$ in Step 1).
2-parameter fit: removes the irreducible loss term $E$ from Equation 1, using $L(C) = A/C^\alpha$ . This is computationally simpler (fewer parameters to fit) but makes the theoretically unrealistic assumption that loss can approach zero with infinite compute. The paper rationalizes testing it because for finite extrapolation ranges, the simpler form may regularize against overfitting to noise in $E$ .
5-parameter (N, D) fit: instead of modeling loss as a function of compute $C$ which conflates model size and training duration, this variant separates them using Equation 5 in Appendix C:

$$L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E$$

where $N$ is the number of model parameters, $D$ is the number of training tokens, and $A, \alpha, B, \beta, E$ are five fitted parameters.

What it computes: loss as separate contributions from model capacity (the $A/N^\alpha$ term) and training data quantity (the $B/D^\beta$ term), plus irreducible error. This allows the function to model cases where a model is overparameterized for its data or vice versa, which the simpler compute-based form cannot distinguish.

Why this form: the paper notes that Bhagia et al. (2024) proposed this to "account for over or undertrained models," but since DATADECIDE uses only one token-parameter ratio, the simpler $L(C)$ form is used as default while the $L(N, D)$ form is included as a variant. The paper acknowledges this limitation: "as our suite only includes one token-parameter ratio, we use the simplified 3 parameter baseline" (Section 2.2), implying the 5-parameter variant may not show its full advantage in this setting.

Helper points: following Bhagia et al. (2024), an artificial data point $(L = 0.0, \text{Acc} = 1.0)$ is optionally added when fitting Equation 2. This "anchors the upper asymptote of the accuracy prediction" (Appendix C), enforcing that the sigmoid must approach perfect accuracy as loss approaches zero. Without this helper point, the upper asymptote is estimated purely from observed data, which may be unreliable if no small-scale model has low enough loss to constrain it.
Filtering early checkpoints (>50%): when fitting Equation 2, the paper experiments with excluding the first 50% of intermediate checkpoints. The rationale is that early-training checkpoints have high loss from the learning rate warmup and optimization transients, and their loss-to-accuracy relationship may not be representative of the stable relationship later in training. Including them could add noise that degrades the sigmoid fit.
Helpers and >50% checkpoints: combines the previous two techniques.
Single-step prediction: rather than chaining two separately-fitted functions, this variant directly maps compute to accuracy in one function (Equation 6 in Appendix C): $$\text{Acc}(C) = \frac{a}{1 + \exp\left(-k\left(\frac{A}{C^\alpha} + E - L_0\right)\right)} + b$$ This merges Equation 1 into Equation 2, fitting all seven parameters $(A, \alpha, E, a, b, k, L_0)$ jointly on the observed $(C, \text{Acc})$ pairs.
5-parameter, single step: combines the $(N, D)$ parameterization with single-step fitting: $\text{Acc}(N, D)$ using the sigmoid of the 5-parameter loss function. The paper notes this yields 7 total free parameters after merging.

How multi-scale predictions are made in practice. The paper sweeps combinations of model sizes to explore the tradeoff between using more observations and using more expensive observations:

"We use $\{\{s_1, \ldots, s_k\} \mid 3 \leq k \leq 14\}$ , where $s$ is the ordered set of sizes, to explore the improvements of progressively adding larger model sizes beyond the minimum 3 required for fitting. We also use $\{\{s_k, \ldots, s_{14}\} \mid 2 \leq k \leq 11\}$ to try removing potentially noisy information from small models." (Section 3.2)

The first strategy starts with the three smallest models (4M, 6M, 8M) and incrementally adds larger ones—this tests whether including more expensive but more informative observations improves predictions. The second strategy starts with only the largest small models and incrementally adds smaller ones—this tests whether small, noisy models degrade predictions.

A critical limitation: multi-scale predictions use only the default seed (seed 1) because final checkpoints are required for fitting Step 1, and only seed 1 is trained to completion at non-target sizes. This means multi-scale predictions have only one attempt, while single-scale predictions can average over 3 seeds (using the partially-trained seeds 2 and 3). The paper notes this asymmetry:

"Unlike single scale results, we make only one prediction attempt with the default fully trained random seed, as final checkpoints are required for fitting the first step of these scaling law variants but are not available for all seeds." (Section 3.2)

Model selection for multi-scale approaches. The paper does not specify which model sizes were used for the multi-scale results shown in Figures 2 and 3. The stars in those figures represent predictions from the "default 3-parameter approach," but the specific subset of model sizes used to produce each star is not enumerated. The text states that "multi-scale predictions have a compute budget equal to the training cost of the model sizes used to make the prediction" (Section 3.2), meaning each star's x-coordinate is the sum of FLOPs for all small models in the fit. But without knowing which sizes produced which stars, the cost-efficiency comparison against single-scale methods is somewhat opaque.

Decision Accuracy: The Evaluation Metric for Predictions

The paper's key methodological contribution is shifting evaluation from prediction error (how close is the predicted number to the observed number?) to decision accuracy (how often does the prediction correctly identify which of two recipes is better?). This reframing matters because the downstream use case of small experiments is making choices, not producing accurate numerical forecasts.

Formal definition. Decision accuracy is defined in Equation 3:

$$\text{Decision Accuracy} = \frac{1}{|P|} \sum_{(A,B) \in P} \mathbb{I}\left[\text{sign}(\hat{y}_A - \hat{y}_B) = \text{sign}(y_A - y_B)\right]$$

where $P$ is the set of all pairs of data recipes, $(A, B)$ is one such pair, $\hat{y}_A$ and $\hat{y}_B$ are the predicted performances (from small-scale experiments) for recipes $A$ and $B$ , and $y_A$ and $y_B$ are the observed performances at the 1B target scale (ground truth). The indicator is 1 when the prediction agrees with the ground truth about which recipe wins.

What it computes: the fraction of all $\binom{25}{2} = 300$ pairwise data-recipe comparisons where the prediction method correctly identifies which recipe achieves higher performance at the target scale. A value of 0.5 is random chance; 1.0 is perfect ranking recovery. This is closely related to Kendall's $\tau$ rank correlation coefficient but scaled to [0, 1] rather than [-1, 1].

Why this form: the sign-based indicator directly captures the binary decision that a practitioner would make: "given the small-scale results, should I choose recipe A or recipe B?" The magnitude of performance difference does not matter for this decision—only the direction. This makes decision accuracy robust to systematic prediction biases. A method could consistently overestimate all recipes by 10 accuracy points (high absolute prediction error) but achieve perfect decision accuracy if it preserves the correct ordering. Conversely, a method with low average prediction error could still make costly ranking inversions if its errors are anti-correlated with true quality.

Ties and near-ties. The definition using $\text{sign}(y_A - y_B)$ implicitly treats exact ties as ambiguous. The paper does not discuss tie-handling explicitly, but given that performance is measured as a continuous accuracy value and the ground truth is based on mean accuracy over 3 seeds, exact ties are extremely unlikely. The more practically relevant question is how the method handles cases where the true difference is smaller than run-to-run noise—a point the paper addresses when discussing why run-to-run variance matters:

"we find that the standard deviation between runs at the 1B 5×C scale can be as high as 2% points of accuracy for some recipes on most tasks. Meanwhile, at the non-target scales we wish to make predictions with a small fraction of the target compute, so we avoid reruns that would use an impractically large prediction budget." (Section 2.1)

This means some "true" pairwise orderings in the ground truth may be artifacts of sampling noise rather than genuine recipe quality differences. If two recipes differ by less than the standard error of the mean over 3 seeds, the ground-truth ordering is unreliable. The paper does not quantify how many of the 300 pairwise comparisons fall into this ambiguous regime, though Figure 5's analysis of "spread" (standard deviation over recipes) relative to "noise" (standard deviation over seeds) for different tasks provides indirect evidence: tasks with high spread and low noise have more reliable ground-truth rankings.

Single-scale decision accuracy with multiple seeds. For single-scale predictions, the paper reports decision accuracy averaged over 3 prediction attempts using models with different random seeds (the shaded standard deviation regions in Figure 1). This means: for each seed's small-scale model, rank the 25 recipes by that model's performance (or proxy metric), compute decision accuracy against the 1B ground truth (which is based on mean over 3 seeds), and average the resulting 3 decision accuracies. This averaging accounts for variance in the small-scale observations due to random initialization and data order, giving a more reliable estimate of how well a given prediction method works at a given compute budget.

The macro-average across tasks. The paper reports results both per-task (Figures 2 and 4) and aggregated as the macro-average of the 10 OLMES tasks (Figure 1 right). The macro-average is used "unless otherwise noted" (Section 2.4). This means the aggregated decision accuracy represents how well a prediction method recovers the ranking of recipes by their average performance across all 10 benchmarks—a reasonable summary if the goal is to pick a recipe that is good "overall," but potentially misleading if recipes have task-specific strengths (e.g., one recipe excels at math, another at commonsense reasoning). The per-task breakdowns in Figures 2 and 4 let practitioners assess which specific evaluations are most predictable and whether proxy metrics help more for some tasks than others.

Why not just use prediction error directly? The paper presents prediction error results in Appendix Table 4 but relegates them to an appendix, emphasizing decision accuracy in the main text. The reasoning (implied throughout Section 2.3) has three components:

Decision accuracy directly answers the practitioner's question. A model developer choosing between recipes wants to know "will I pick the right one?", not "will my predicted accuracy be within ±2% of the true value?" These are different desiderata, and optimizing for one does not guarantee the other.
Prediction error is sensitive to scale and task difficulty. A 2% absolute error on an easy task where accuracy ranges from 70–90% represents good calibration; the same 2% error on a hard task where accuracy ranges from 25–35% is essentially worthless noise. Decision accuracy normalizes for this by considering only relative ordering within each task's observed range.
Prediction error conflates bias and ranking quality. A method that systematically predicts 5% lower than observed for all recipes has 5% bias but perfect decision accuracy. Reporting only prediction error would penalize this method relative to an unbiased but ranking-inverting alternative, which is the wrong evaluation for the decision-making use case.

The paper's Appendix Table 4 does confirm that the 2-parameter and 3-parameter scaling law variants have comparable prediction errors (absolute errors of 3.1–3.2 percentage points on the 0–100 accuracy scale for the 3-parameter and 2-parameter fits, respectively), suggesting that the ranking failures of multi-scale methods are not due to catastrophically worse prediction error but rather to the specific pattern of their errors—they preserve prediction error magnitude while losing ranking fidelity.

Design Choices and Their Justifications

Several design choices shape what DATADECIDE can and cannot measure. Understanding these choices is essential for interpreting the results' scope and limitations.

Why 25 recipes, not more? The paper explicitly states this is "an order of magnitude more fine-grained data differences" than prior suites (Section 4), which offered 2–6 recipes. The number 25 balances two constraints: (1) having enough recipes to observe meaningful variation in decision accuracy across different types of data interventions, and (2) the compute cost of training 14 sizes × 25 recipes × 3 seeds. Adding more recipes would either require reducing model sizes (losing the ability to measure scaling trends) or reducing seeds (losing the ability to measure variance). The paper argues that 25 provides sufficient coverage of "common data preparation choices" while remaining computationally feasible.

Why 5× Chinchilla token-to-parameter ratio? The paper states: "We believe this captures the typical case, as most models now favor overtraining for inference savings" (Section 5, Limitations). The Chinchilla-optimal ratio is approximately 20 tokens per parameter; training at 100 tokens per parameter means each model sees 5× more data than would be compute-optimal for that parameter count. This is representative of how most deployed models are actually trained (LLaMA, OLMo, etc.), so the findings about data-recipe rankings are ecologically valid for current practice. However, the overtraining ratio could affect the generalizability of findings to other ratios—it is possible that some data recipes benefit more from overtraining than others, which would mean the optimal recipe at 5×C might differ from the optimal recipe at 1×C. The paper does not explore this dimension.

Why these 14 specific model sizes? The sizes are determined by the OLMo model ladder, which provides pre-designed, validated configurations. Rather than designing architectures from scratch for each scale, the paper relies on configurations that OLMo developers "handpicked... to achieve the desired number of parameters" (Appendix A). This ensures that the hyperparameters are reasonable at each scale, avoiding the confound where poor performance at a particular size is due to bad hyperparameters rather than the data recipe. The sizes are denser at the small end (4M–20M in small increments) and sparser at the large end (300M–1B in larger jumps), reflecting both the compute cost of training larger models and the expectation that scaling trends are smoother at larger scales, requiring fewer observations to characterize.

Why the cloze formulation for evaluation? The paper notes that it uses OLMES's "cloze formulation" (CF) accuracy with curated per-task normalization. The cloze formulation presents answer options as text continuations and scores the probability the model assigns to each continuation. This is chosen over multiple-choice prompting (where the model selects A/B/C/D) because it produces continuous likelihoods for each option—the raw material for the proxy metrics. A multiple-choice setup that only records the selected letter would prevent computation of CORRECT PROB, TOTAL PROB, and the other continuous metrics that are central to the paper's findings.

Why character-level rather than token-level length normalization for proxy metrics? The paper states that character normalization "is empirically optimal for most of the tasks that we observe" (Section 3.3). Character normalization divides each continuation's log-probability by its character count, while token normalization divides by token count. Token normalization can disadvantage continuations that get split into many tokens by the tokenizer (e.g., rare words, code tokens), artificially reducing their normalized probability. Character normalization is more uniform across continuation lengths and less sensitive to tokenizer artifacts, making it a better calibration signal. The paper also tests token normalization (not shown in main figures) but finds it consistently underperforms.

4. Key Insights and Innovations

Innovation 1: Decision Accuracy as the Correct Evaluation Metric for Data Selection Methods

The paper's most conceptually distinctive move is redefining the success criterion for small-scale prediction methods from "how close is the predicted number?" to "how often does the prediction make the right choice?" This is not merely a change in measurement—it is a reframing of what the prediction task is. Prior work on scaling laws (Gadre et al., 2024; Bhagia et al., 2024; Dubey et al., 2024) evaluated methods by prediction error: absolute or relative deviation between predicted and observed downstream accuracy at the target scale. This implicitly treats the goal as numerical forecasting. But practitioners do not use small experiments to produce accurate point estimates of large-model accuracy; they use them to choose among alternatives. A method with 5 percentage points of systematic bias that correctly ranks all 25 recipes is strictly more useful than an unbiased method that inverts the top five rankings. Prediction error conflates these two outcomes; decision accuracy separates them.

This reframing has a sharp implication that the paper does not overstate but that emerges clearly from the results: it is possible for scaling law methods to match single-scale ranking on prediction error while underperforming on decision accuracy (Appendix Table 4 vs. Figure 3). The default 3-parameter fit achieves absolute prediction error of 3.1 percentage points—comparable to single-scale ranking's error—yet its decision accuracy lies on or below the single-scale frontier across the full compute range. This means the errors of multi-scale methods, while similar in magnitude to those of single-scale ranking, are distributed differently across recipes in ways that systematically degrade ranking fidelity. The paper does not diagnose exactly why this happens (it could be overfitting to small-model noise, or systematic extrapolation biases that affect some recipes more than others), but the measurement itself—showing that prediction error and decision accuracy can diverge—is the conceptual contribution. It tells the field that validating scaling laws solely on prediction error is insufficient, and that any proposed method for data selection should be evaluated on whether it actually selects the right data.

This is a fundamental shift, not an incremental refinement. It changes what "better" means for prediction methodology. The compute-decision frontier (Figure 1, right panel) becomes the primary diagnostic tool, replacing the prediction-error-versus-compute curves that have dominated the scaling laws literature. And because the frontier is defined by the best decisions per unit compute, it directly quantifies the practical value of methodological improvements: a method that pushes the frontier up and left is one that lets practitioners spend less on small experiments while making more reliable data choices.

Innovation 2: The Empirical Demonstration That Single-Scale Ranking Beats Multi-Scale Scaling Laws—And What This Reveals About Crossovers

The paper's most surprising empirical finding is that ranking models at a single small size (e.g., 150M parameters) achieves ~80% decision accuracy on the OLMES macro-average at the 1B target scale, and none of eight multi-scale scaling law variants exceed this baseline on the compute-to-decision-accuracy frontier (Figure 3). This is not obvious ex ante. A priori, single-scale ranking should fail whenever two data recipes exhibit a ranking crossover—recipe A outperforms B at the small experimental scale but B overtakes A at the target scale. Multi-scale methods, by modeling the trend of performance with scale, should in principle capture such crossovers and make correct predictions where single-scale ranking is structurally incapable of doing so. Yet the empirical evidence shows that whatever advantage multi-scale methods gain from modeling crossovers is offset—and often more than offset—by the noise they introduce through fitting additional parameters to limited, noisy observations.

The significance of this result extends beyond the specific numbers. It establishes that for the scale range studied (4M to 1B parameters, 0.4B to 100B tokens), ranking crossovers between data recipes are either less frequent, less consequential, or harder to distinguish from evaluation noise than the field might have assumed. The paper explicitly observes that crossovers do occur: "the scaling trends we empirically observe cross over frequently" (Section 3.2). But their impact on decision accuracy is bounded by the fact that single-scale ranking already achieves ~80% correct pairwise decisions. This means that at most ~20% of recipe pairs change relative ordering between the small scale and the target—and some fraction of those apparent crossovers may be artifacts of run-to-run variance rather than genuine scaling reversals. The practical implication is substantial: for a typical data curation workflow where the goal is to identify the best among a set of candidate recipes, the simple approach of training small models on all candidates and ranking them is not just cheaper than scaling law extrapolation—it is actually more reliable at the granularity of pairwise decisions, at least up to the 1B / 100B-token scale.

This finding also functions as a diagnostic. It tells the scaling laws community that the bottleneck for improving data-decision methodology is not the functional form of the extrapolation function but the quality and quantity of small-scale observations. The eight variants tested span 2-parameter through 5-parameter fits, single-step and two-step procedures, with and without helper points and early-checkpoint filtering—a reasonable coverage of the design space suggested by prior work (Bhagia et al., 2024; Choshen et al., 2024). None outperform single-scale ranking. This strongly suggests that future progress will not come from marginal improvements in curve-fitting but from fundamentally better ways to extract signal from small-scale experiments—such as the proxy metric approach the paper itself develops, or from entirely different prediction paradigms that the DATADECIDE testbed is designed to enable.

The innovation here is the empirical calibration of an assumed-to-be-inferior baseline against the assumed-to-be-superior sophisticated method, with the counterintuitive result that the baseline wins. Prior work did not have the counterfactual data to perform this calibration; DCLM (Li et al., 2024) used single-scale ranking but never validated it against multi-scale alternatives because they only trained one large model. DATADECIDE provides the first evidence that this choice—using single-scale experiments—was not merely a pragmatic cost-savings compromise but is, on current evidence, the best available method for the decision task.

Innovation 3: Continuous Likelihood Metrics as Superior Proxies for Discrete Accuracy at Small Scales

The paper's most actionable practical contribution is the systematic demonstration that using continuous likelihood-based metrics (CORRECT PROB and TOTAL PROB) as proxies for discrete accuracy at small experimental scales dramatically improves decision accuracy, and can even make previously unpredictable tasks predictable. This finding emerges from a simple insight: at the small scales where experiments are cheap enough to run on many data recipes, discrete accuracy on many benchmarks is near the noise floor—too low and too flat across recipes to distinguish which recipe is better. But the continuous signal of how much probability the model assigns to correct answers (CORRECT PROB) or to the answer-option set as a whole (TOTAL PROB) can show meaningful variation across recipes even when discrete accuracy is stuck near chance. This variation, because it tracks the same underlying capability that eventually produces accuracy differences at scale, serves as an effective leading indicator.

The magnitude of the improvement is striking for specific tasks. Figure 6 shows that code tasks (HumanEval, MBPP) go from trivial (~50% random-chance) decision accuracy when using discrete ACCURACY at small scales to >80% decision accuracy when using CORRECT PROB—a transformation from unusable to highly reliable. Figure 4 shows that for ARC Challenge, ARC Easy, BoolQ, CommonsenseQA, and OpenBookQA, CORRECT PROB and TOTAL PROB at small scales (10^-4 to 10^-2 %C) achieve decision accuracy that discrete ACCURACY only reaches at 10-100× more compute. The paper's per-task breakdowns (Figure 2) reveal that these effects are not uniform—some tasks like PIQA and SocialIQA remain difficult to predict regardless of metric—but the pattern is consistent enough to support a practical recommendation: practitioners running small-scale data ablations should always compute and compare both discrete accuracy and continuous likelihood metrics, because the latter will often provide usable signal where the former is noise.

The conceptual contribution here is not the observation that continuous metrics can be smoother than discrete ones—Schaeffer et al. (2023) documented this for emergent abilities. The innovation is applying this observation to the data-selection decision problem and measuring exactly how much it improves decision accuracy per unit compute. The paper distinguishes between two types of proxy behavior (Section 3.3): metrics like CORRECT PROB and TOTAL PROB that are "flat with respect to scale and [see] the other metrics only rise up to that level of decision accuracy towards the full target compute budget," and metrics like MARGIN and NORM CORRECT PROB that "trend with ACCURACY." The flat metrics are the ones that provide maximum small-scale advantage because they saturate their discriminative power early—a 4M model can already distinguish recipes as well as a 150M model on these metrics for some tasks, whereas discrete accuracy only catches up near the target scale. This behavioral distinction gives practitioners a diagnostic for choosing which proxy to use: if the proxy's decision accuracy is flat across small scales, it is extracting all available signal cheaply; if it trends upward, more compute will help.

The code-task result deserves special emphasis because it addresses a practical pain point. Code generation benchmarks like HumanEval are central to modern LLM evaluation, but they are notoriously difficult to measure at small scales—a 150M model trained on 15B tokens may score 0% on pass@1, providing zero signal for data selection. The finding that CORRECT PROB achieves >80% decision accuracy for these tasks at model sizes as small as 4M–60M parameters (Figure 6) means that data decisions relevant to code performance can now be made with inexpensive small experiments, when previously they required guesswork or much larger-scale ablations. The simultaneous finding that math tasks (GSM8K, Minerva) do not see this benefit—remaining near-trivial decision accuracy regardless of metric—is equally informative: it identifies a class of benchmarks where small-scale prediction remains fundamentally challenging, pointing to a need for different approaches.

Innovation 4: A Diagnostic Framework for Understanding Why Some Tasks Are More Predictable Than Others

Beyond the specific metric recommendations, the paper develops a diagnostic decomposition of what makes a benchmark predictable from small-scale experiments: it is the interplay between spread (how much the mean performance varies across data recipes) and noise (how much performance varies across random seeds for the same recipe). Figure 5 operationalizes this by plotting, for each task and metric at the 150M scale, the standard deviation of performance across recipes (x-axis) against the standard deviation of performance across 3 seed runs averaged over recipes (y-axis), with point color showing the resulting decision accuracy. Tasks in the upper-left region (high noise, low spread) are unpredictable; tasks in the lower-right (low noise, high spread) are highly predictable.

This framework is primarily diagnostic, not prescriptive—it explains why certain tasks or metrics succeed or fail, but does not itself provide a method to improve predictability. However, it surfaces two actionable levers. First, choosing a proxy metric can move a task within this space: Figure 5 shows that switching from ACCURACY to CORRECT PROB often increases spread, reduces noise, or both. For example, ARC Easy sees a substantial spread increase when using CORRECT PROB, while HellaSwag benefits primarily from reduced run-to-run noise. This explains the mechanism behind the proxy-metric improvements documented in Figure 4: the continuous metrics are not merely "smoother" in some abstract sense; they specifically amplify the signal (recipe-to-recipe variation) relative to the noise (seed-to-seed variation).

Second, the framework suggests that evaluation design itself can be optimized for predictability. The paper shows that for tasks where the discrete metric has high noise at small scales, evaluating on more items (the paper uses all OLMES items rather than subsampling) can reduce noise and improve decision accuracy. More broadly, the framework implies that benchmarks intended for use in small-scale data decisions should be designed to maximize spread-over-noise at the relevant scale range—a consideration that current benchmark design (focused on discriminability at large, deployment scales) does not typically address.

The conceptual significance of this framework is that it provides a principled alternative to the common but unsatisfying conclusion that "this task is hard to predict." Rather than treating unpredictability as a monolithic property of a benchmark, the decomposition reveals whether the bottleneck is excessive noise (suggesting more seeds or more evaluation items) or insufficient spread (suggesting that the task is genuinely insensitive to the data interventions being tested, or that a different metric might surface latent differences). This transforms unpredictability from an endpoint into a diagnostic that can guide methodological improvements.

This innovation is incremental in its components (spread and noise are standard statistical concepts) but fundamental in its application: it provides the first systematic framework for understanding why data decisions succeed or fail for specific benchmarks, and it gives practitioners a tool for diagnosing and addressing failures in their own small-scale ablation workflows.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation suite is OLMES (Gu et al., 2024), comprising 10 multiple-choice question answering benchmarks: MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019), ARC Challenge and ARC Easy (Clark et al., 2018), PIQA (Bisk et al., 2020), CommonsenseQA (Talmor et al., 2019), SocialIQA (Sap et al., 2019), OpenBookQA (Mihaylov et al., 2018), BoolQ (Clark et al., 2019), and WinoGrande (Sakaguchi et al., 2020). The paper uses all available items in each benchmark's specified split rather than subsampling, to reduce variance over the task distribution (Section 2.4). The underlying metric for each task is accuracy, with OLMES specifying a different length normalization scheme per task. Ground-truth rankings at the target (1B) scale are always based on the "cloze formulation" (CF) accuracy with curated per-task normalization—a setup where the model scores the probability of each answer option's text continuation rather than selecting among lettered options (A/B/C/D). This formulation is chosen because it yields continuous likelihoods for each option, which are the raw material for the proxy metrics.
Base model(s). All models are pretrained from scratch using the OLMo model ladder (Groeneveld et al., 2024; OLMo et al., 2025; Bhagia et al., 2024), which provides programmatically generated configurations for 14 model sizes ranging from 4M to 1B non-embedding parameters. The ladder uses heuristics from Porian et al. (2024) to set global batch size and learning rate as functions of model scale, while architectural parameters (layers, hidden dimension, number of heads, MLP dimension) are handpicked by OLMo developers at each size. All models use a fixed token-to-parameter ratio of 100 (5× the "Chinchilla" optimal ratio of ~20), sequence length 2048, and MLP ratio 8× hidden dimension. The choice of the OLMo ladder is motivated by the need to eliminate confounds: without programmatic scaling of hyperparameters, performance differences between data recipes could be artifacts of suboptimal hyperparameters at particular scales rather than genuine data effects. The specific configurations are detailed in Appendix Table 2.
Metrics. The paper uses two categories of evaluation metrics, both computed from the model's token probabilities on answer continuations. The target metric at the 1B scale is always ACCURACY—the discrete fraction of items where the model assigns the highest probability to the correct continuation among the allowed answer set (Table 3). The proxy metrics used as inputs to prediction methods at small scales are continuous functions of the same underlying probabilities: CORRECT PROB (average probability on the correct answer), TOTAL PROB (average total probability on all answer options), MARGIN (average gap between correct-answer probability and highest-incorrect-answer probability), NORM CORRECT PROB (correct probability divided by total probability on the answer set), and ACCURACY itself. All proxy metrics have length-normalized variants (per token and per character), with character normalization used unless otherwise specified, as the paper states it is "empirically optimal for most of the tasks that we observe" (Section 3.3). The formal definitions appear in Appendix Table 3. The primary evaluation metric for assessing prediction quality is decision accuracy: the fraction of all 300 pairwise data-recipe comparisons where the prediction correctly identifies which recipe achieves higher mean 1B performance (Equation 3).
Baselines. The paper evaluates two broad families of prediction methods, none of which are novel proposals—they are existing approaches validated against the counterfactual ground truth for the first time. Single-scale ranking: train models at one small size (or use a single intermediate checkpoint) on all 25 data recipes, rank recipes by the observed performance (or proxy metric), and predict that this ranking extrapolates to the 1B scale. This is the method used by DCLM (Li et al., 2024) and similar data curation projects. Multi-scale scaling law extrapolation: fit a two-step function chain—Equation 1 mapping compute C to downstream task loss L (a power law: L(C) = A/C^α + E), followed by Equation 2 mapping loss L to accuracy Acc (a sigmoid: Acc(L) = a/(1 + e^(-k(L-L_0))) + b)—using observations from multiple small model sizes, then extrapolate to the target compute. The paper tests eight variants of this approach (Appendix C): the baseline 3-parameter fit; a 2-parameter fit removing the irreducible loss term E; a 5-parameter (N, D) fit that separates model size and data tokens; single-step fitting that combines both functions into one; and combinations with helper points (anchoring the sigmoid's upper asymptote at perfect accuracy) and early-checkpoint filtering (excluding the first 50% of intermediate checkpoints when fitting Equation 2). All multi-scale fits use the default seed only, since other seeds are truncated at 25% and lack final checkpoints needed for Step 1 fitting.
Generation budget / compute accounting. Compute is measured as theoretical FLOPs using the standard scaling-law approximation FLOPs = 6ND (Kaplan et al., 2020), where N is the number of non-embedding parameters and D is the number of training tokens (Section 2.3). This yields a consistent, hardware-independent cost metric. Prediction cost is expressed as a fraction of the target training cost: %C = c/C × 100%, where c is the total FLOPs of all small experiments used to make the prediction and C = 6 × (1B parameters) × (100B tokens) is the FLOPs of training a single 1B model at the target scale. For multi-scale predictions, the compute budget equals the training cost of the model sizes used in the fit (e.g., using 4M, 6M, and 8M models costs the sum of their individual FLOPs). For single-scale predictions using intermediate checkpoints, the cost is the FLOPs consumed to reach that checkpoint. The x-axis of all decision-accuracy figures uses %C on a log scale spanning approximately 5 orders of magnitude from ~10^-5 to 100%.
Cross-validation / statistical protocol. There is no train/validation/test split in the traditional sense because the "prediction" is evaluated against observed 1B performance, not against held-out data. The key statistical design elements are: (1) The ground-truth ranking at 1B is based on mean performance over 3 full random seed reruns, to account for run-to-run variance that the paper reports can reach 2 percentage points of accuracy for some recipes on most tasks (Section 2.1). This is critical because if the ground truth were based on a single seed, prediction methods could be penalized for failing to predict noise. (2) For single-scale predictions, decision accuracy is reported as the average over 3 prediction attempts using small models with different random seeds, with shading indicating standard deviation (Figures 1–4). This averaging accounts for variance in the small-scale observations due to weight initialization and data order. (3) For multi-scale predictions, only a single prediction attempt is possible (seed 1), since seeds 2 and 3 at non-target sizes are truncated at 25% of target compute and lack the final checkpoints required for fitting Step 1 (Section 3.2). (4) The paper explores multiple model-size combinations for multi-scale fits: using the three smallest models and incrementally adding larger ones, or starting with the largest small models and incrementally adding smaller ones (Section 3.2), to assess sensitivity to which scales are included.

Main Quantitative Results

The results are organized around three primary analyses: (1) the compute-to-decision-accuracy relationship for single-scale ranking, broken out by task and explored across scales and checkpoints (Section 3.1, Figures 1–2); (2) the comparison of multi-scale scaling law methods against the single-scale frontier (Section 3.2, Figure 3); and (3) the effect of proxy metrics on small-scale decision accuracy, including the extension to code and math tasks (Sections 3.3–3.4, Figures 4–6). A fourth analysis (Figure 5) provides a diagnostic decomposition of why some tasks and metrics are more predictable than others.

The Compute-to-Decision-Accuracy Frontier for Single-Scale Ranking

The central aggregate result appears in Figure 1 (right panel), which plots decision accuracy on the OLMES macro-average (mean over all 10 tasks) against %C for single-scale predictions at all 14 model sizes and their intermediate checkpoints, plus multi-scale predictions (stars). The colored points trace out a roughly log-linear relationship: more compute invested in small experiments yields monotonically higher decision accuracy on the OLMES macro-average. Specifically:

The very smallest scales and earliest checkpoints (10^-5 to 10^-3 %C) achieve decision accuracy only marginally above the 0.5 random-chance baseline—approximately 0.55–0.65, corresponding to the leftmost points in the figure for the 4M–14M parameter configurations.
As compute increases into the 10^-2 to 10^-1 %C range (corresponding roughly to fully-trained 20M–90M models, or to later checkpoints of smaller models), decision accuracy rises into the 0.70–0.78 range.
At approximately 1–10 %C (150M–530M models fully trained, or later checkpoints of 90M–150M models), decision accuracy reaches ~0.80.
The 1B models themselves (100 %C), when used to predict the 1B ground truth (a circular prediction), achieve approximately 0.85–0.88—not 1.0, because the ground truth is based on mean over 3 seeds while each individual 1B seed has its own evaluation noise.

The gap between ~0.80 at 1–10 %C and ~0.85–0.88 at 100 %C represents the irreducible benefit of actually training the target model—the ~5–8 percentage points of decision accuracy that no amount of small-scale prediction can recover, at least for this macro-average. The paper does not explicitly compute this gap, but it is visible in the saturation of the single-scale curve in Figure 1 (right).

Task-specific sensitivity varies by orders of magnitude. The most striking decomposition appears in Figure 2, which shows the same compute-to-decision-accuracy relationship separately for each of the 10 OLMES tasks, using ACCURACY as the metric. The key finding is that different tasks become predictable at dramatically different compute thresholds:

ARC Easy is predictable across essentially the entire compute range, with decision accuracy rising above 0.8 even at the smallest scales (10^-5 %C) and approaching ~0.95 at the 150M–1B range. This task reliably separates data recipes even when models are tiny and undertrained.
MMLU and ARC Challenge show a smooth, roughly log-linear improvement from ~0.5–0.6 at the smallest scales to ~0.85–0.9 at the 150M–1B range, with decision accuracy crossing 0.8 at roughly 10^-2 %C (corresponding to fully trained 20M–60M models or late checkpoints of smaller models).
HellaSwag displays a distinct threshold behavior: decision accuracy remains near 0.5–0.6 (barely above random) for all scales up to approximately 10^-1 %C, then rises sharply to ~0.8–0.85 in the 150M–1B range. This is the clearest example of a task where small-scale experiments are effectively useless for distinguishing data recipes—the model needs to reach a certain minimum scale before the evaluation produces signal that correlates with eventual large-scale performance. The paper describes this as "distinct periods of insensitivity followed by roughly log-linear increase after hitting some compute threshold" (Section 3.1).
SocialIQA, WinoGrande, and BoolQ show similar threshold patterns but with even later onset: SocialIQA and WinoGrande remain near 0.5–0.6 until roughly 10^0–10^1 %C (150M–530M models), and BoolQ only exceeds trivial decision accuracy "for intermediate checkpoints of the target runs" (Section 3.1)—meaning the 1B models themselves are needed to get reliable signal.
PIQA, CommonsenseQA, OpenBookQA occupy intermediate positions, with PIQA rising to ~0.75–0.8 by the 150M–300M range but showing relatively flat scaling thereafter, and CommonsenseQA and OpenBookQA showing more gradual improvement across the full range.

The practical implication of Figure 2 is direct: the amount of compute you need to allocate for a given data decision depends heavily on which evaluation benchmark you care about. If you are optimizing for ARC Easy or MMLU, small experiments at the 4M–20M scale already provide substantial decision accuracy. If you care about HellaSwag, you need at least 150M-scale models. If you care about BoolQ or SocialIQA, small experiments provide essentially no reliable signal—you must either train larger models or use proxy metrics (as Sections 3.3–3.4 will show).

Intermediate checkpoints are as good as compute-equivalent final checkpoints. Figure 1 (right) shows that points of the same color (same parameter size, different checkpoints) and points of different colors at similar %C (different parameter sizes, similar total FLOPs) overlap substantially. This means that within the range studied, what matters for decision accuracy is total FLOPs expended, not whether those FLOPs go into a smaller model trained longer or a larger model trained shorter. A 150M model at 25% of its training (roughly 3.75B tokens) has approximately the same decision accuracy as a 60M model trained to completion (5.7B tokens) because they consume similar total FLOPs. This finding appears implicitly in how the colored curves in Figure 1 overlap rather than being systematically separated by color, and the paper's statement that "Decisions from intermediate checkpoints are as good as compute equivalent final checkpoints" (Section 3.1) makes it explicit.

This equivalence has a practical consequence: if you have already trained small models for other purposes, you can use their intermediate checkpoints to make data decisions without additional training cost. The paper's "points with the same color show all intermediate checkpoints from a given parameter size" (Section 3.1) means the curves in Figures 1–4 are constructed by evaluating every saved checkpoint and computing its decision accuracy, then plotting each checkpoint at its %C cost. The smoothness of these curves confirms that decision accuracy improves monotonically with compute consumed, regardless of the specific model size or training duration.

Multi-Scale Scaling Laws vs. Single-Scale Ranking

The central comparison appears in Figure 3, which overlays the 8 multi-scale scaling law variants (colored stars) on top of the single-scale frontier (same colored points as Figure 1). The headline result is stated directly in the paper's abstract:

"No scaling law methods among 8 baselines exceed the compute-decision frontier of single-scale predictions"

Figure 3 shows this visually: the stars representing multi-scale predictions (at various total compute budgets corresponding to different combinations of model sizes used in the fit) lie on or below the envelope traced by the single-scale points. At best, the multi-scale methods match the single-scale frontier; at worst, they substantially underperform it at comparable compute budgets.

Specifically, the best-performing multi-scale variants—the 2-parameter and 3-parameter fits (blue and orange stars)—cluster near the single-scale frontier in the 10^-2 to 10^0 %C range, achieving decision accuracy comparable to single-scale ranking at similar compute costs. The 5-parameter variants and single-step variants perform substantially worse, with some points falling well below the frontier. The paper notes that "each of the scaling law variants are defined formally in Appendix C. The 2 and 3 parameter variants all achieve among the top decision accuracy" (Section 3.2).

A crucial nuance in interpreting Figure 3 is the asymmetry in seed averaging. Single-scale points represent the average over 3 prediction attempts (using partially-trained seeds 2 and 3 for the small models), with the standard deviation shown as shading. The multi-scale stars represent only a single prediction attempt (seed 1), because seeds 2 and 3 are truncated and lack the final checkpoints required for fitting Step 1. This means the multi-scale predictions are more vulnerable to run-to-run noise than the single-scale average. The paper acknowledges this: "Unlike single scale results, we make only one prediction attempt with the default fully trained random seed, as final checkpoints are required for fitting the first step of these scaling law variants but are not available for all seeds" (Section 3.2). If multi-scale predictions could be averaged over 3 seeds, their variance would decrease and their mean decision accuracy might rise—potentially closing some or all of the gap with single-scale ranking. The paper does not quantify how much of the underperformance is due to this asymmetry versus genuine methodological inferiority.

Why the extra complexity of scaling laws does not help. The paper offers a diagnosis in Section 3.2:

"A priori we know that ranking single scale experiments cannot correctly predict when the scaling trend of one data recipe overtakes another at scales between our small experiments and target scale. Such crossovers bound the decision accuracy of this constant approximation of performance. Nevertheless ranking single scale experiments sets a high baseline decision accuracy, implying relatively little crossover occurs."

This means the theoretical advantage of multi-scale methods—modeling crossovers—is undermined by two empirical facts: (1) crossovers between data recipes are not common enough or large enough in magnitude to substantially degrade single-scale decision accuracy (which already reaches ~80%), and (2) whatever small benefit multi-scale methods gain from modeling the crossovers that do exist is offset by the noise introduced through fitting additional parameters to limited, noisy small-scale observations. The paper states this explicitly: "It is difficult to distinguish evaluation variance from true crossovers" (Section 3.2). In other words, many apparent crossovers in the small-scale data may be artifacts of run-to-run noise rather than genuine scale-dependent ranking reversals, and multi-scale methods that try to model them end up fitting noise rather than signal.

Prediction error tells a different story than decision accuracy. Appendix Table 4 reports the average prediction error for each scaling law variant (mean across all tasks and recipes of the absolute error in predicted 1B accuracy). The 2-parameter and 3-parameter variants achieve absolute errors of 3.2 and 3.1 percentage points, respectively—comparable values that do not clearly distinguish among the methods. The paper uses this to reinforce its methodological point: prediction error does not reveal the ranking failures that decision accuracy captures. The disconnect arises because prediction error treats all errors symmetrically—overestimating recipe A by 2 points and underestimating recipe B by 2 points yields the same average error as overestimating both by 2 points, but only the former scenario produces a ranking inversion. The 2-parameter and 3-parameter fits have similar prediction error magnitudes but apparently differ in how those errors are distributed across recipes, with the 3-parameter fit preserving slightly better ranking fidelity (visible in Figure 3 where the 3-parameter stars are marginally closer to the frontier).

Proxy Metrics Dramatically Improve Small-Scale Decision Accuracy

The paper's most actionable set of findings concerns the use of continuous likelihood-based metrics as substitutes for discrete accuracy when making predictions from small-scale experiments. The core result appears in Figure 4, which shows per-task decision accuracy using character-normalized proxy metrics (CORRECT PROB, TOTAL PROB, MARGIN, NORM CORRECT PROB) compared against ACCURACY, with all metrics used to predict ACCURACY at the 1B target.

The headline pattern. The paper identifies two distinct types of proxy-metric behavior across tasks (Section 3.3):

Metrics that are "flat with respect to scale": For several tasks, CORRECT PROB and TOTAL PROB achieve high decision accuracy even at the very smallest scales (10^-5 to 10^-3 %C) and maintain roughly constant accuracy across the full compute range. ACCURACY and the other continuous metrics (MARGIN, NORM CORRECT PROB) start near random chance and rise gradually, only catching up to CORRECT PROB/TOTAL PROB near the target scale (10^0 to 10^2 %C). This pattern is visible most clearly in ARC Easy, ARC Challenge, BoolQ, CommonsenseQA, and OpenBookQA. For ARC Easy, CORRECT PROB achieves ~0.85 decision accuracy at 10^-4 %C (roughly a 4M model partially trained), while ACCURACY at the same compute is near 0.5–0.6 and only reaches ~0.9 at 100 %C. The practical implication: for these tasks, switching to CORRECT PROB or TOTAL PROB lets you make reliable data decisions with orders of magnitude less compute than using the standard discrete accuracy metric.
Metrics that trend with ACCURACY: For other tasks (MMLU, PIQA, SocialIQA, WinoGrande), all proxy metrics including CORRECT PROB and TOTAL PROB track ACCURACY relatively closely—they rise together with compute rather than saturating early. On MMLU, for instance, all five metrics follow similar trajectories, reaching ~0.8 decision accuracy at roughly 10^-1 %C and improving to ~0.9 at 100 %C. For these tasks, proxy metrics do not offer substantial compute savings over just using ACCURACY, though they may still be preferred for their smoothness and reduced variance.

HellaSwag occupies an intermediate position: CORRECT PROB and TOTAL PROB exceed ACCURACY substantially in the 10^-2 to 10^0 %C range (achieving ~0.7–0.75 while ACCURACY is near 0.55–0.6), but all metrics converge at ~0.85 by the 1B scale.

The "last order of magnitude" crossover. The paper notes a subtle but consistent pattern (Section 3.3):

"In the last order of magnitude below the target compute ACCURACY and the other metrics tend to overtake CORRECT PROB and TOTAL PROB, while these two metrics sometimes even decrease in decision accuracy."

This is visible in several panels of Figure 4: for example, in BoolQ and OpenBookQA, CORRECT PROB's decision accuracy peaks around 10^-1 to 10^0 %C and then slightly declines or plateaus, while ACCURACY continues rising and eventually surpasses it near the 100 %C mark. The paper attributes this to the fact that CORRECT PROB and TOTAL PROB saturate their discriminative power early—they reach a ceiling where additional probability mass on correct/all answers no longer distinguishes recipes as effectively, while ACCURACY, which depends on the correct answer crossing the argmax threshold, continues to improve as the model's ranking of options sharpens. This crossover implies a practical guideline: use CORRECT PROB or TOTAL PROB at small scales (well below the target), but if you are already training near the target scale, ACCURACY is competitive or superior.

Code tasks become predictable with proxy metrics; math tasks do not. Figure 6 shows a direct application of the proxy-metric insight to tasks that are too difficult for small models to score above chance on discrete accuracy. For two code generation benchmarks—HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021)—the paper demonstrates:

Using ACCURACY at the 4M or 60M scale achieves decision accuracy near 0.5 (random chance, shown as a horizontal dashed line). The models are too small and undertrained to produce correct code at any meaningful rate, so discrete pass/fail metrics provide zero signal for distinguishing data recipes.
Switching to CORRECT PROB at the same model sizes raises decision accuracy to above 80%—comparable to the best-performing OLMES tasks. The paper states: "The switch of metric allows small models to get above the noise floor for these tasks, while still predicting large-scale accuracy metrics" (Section 3.4).

For two math benchmarks—GSM8K (Cobbe et al., 2021) and Minerva (Lewkowycz et al., 2022)—the same switch provides essentially no benefit. Both ACCURACY and CORRECT PROB remain near 50% decision accuracy regardless of scale (4M or 60M). The paper notes this asymmetry without fully diagnosing it: "They do however give decision accuracy above 80% if we switch the target metric to CORRECT PROB, raising a question for future work to explore whether changing the target metric can be justified" (Section 3.4). This means that for math tasks, even the continuous likelihood signal on correct answers does not correlate well with future discrete accuracy—the model's probability assignments at small scales are essentially uninformative about which data recipe will eventually produce better math performance. This could be because math capability requires a threshold of scale that DATADECIDE's small models have not reached even in continuous metrics, or because the relationship between early likelihoods and eventual accuracy is genuinely different for math than for code and commonsense reasoning.

Explaining Why Some Tasks Are More Predictable: Spread vs. Noise

Figure 5 provides a diagnostic decomposition of what determines decision accuracy at a fixed scale (the 150M model trained to completion). For each of the 10 OLMES tasks and two metrics (ACCURACY and CORRECT PROB), the figure plots two characteristics:

Noise (y-axis): the standard deviation of performance across 3 random seed runs, averaged over all 25 data recipes. This captures how much run-to-run variability (from weight initialization and data order) obscures the true signal of recipe quality.
Spread (x-axis): the standard deviation of mean performance (averaged over 3 seeds) across the 25 data recipes. This captures how much the recipes actually differ from each other—the genuine signal that decisions aim to exploit.
Decision accuracy: shown by the color of each point (darker = higher).

The diagnostic is intuitive: for a task to be predictable, recipes must differ from each other (high spread) by more than the run-to-run noise (low noise). The paper states: "The evaluation must separate pairs of data recipes by an amount greater than combined noise from run-to-run variance of each of the pair's runs" (Section 3.4). Tasks in the upper-left region (high noise, low spread) have decision accuracy near 0.5; tasks in the lower-right (low noise, high spread) approach 0.9–1.0.

Specific task placements explain earlier findings. The figure reveals why certain tasks are easy or hard to predict:

MMLU (ACCURACY): very low noise (~0.003–0.004 standard deviation in accuracy units) and moderate spread (~0.015–0.02), yielding decision accuracy ~0.85. The low noise is what makes MMLU highly predictable—the signal, while not enormous, is reliably measured.
ARC Easy (ACCURACY): moderate noise (~0.01) but very high spread (~0.06–0.07), yielding decision accuracy >0.9. Here it is the large genuine differences between recipes that drive predictability—even with some measurement noise, the recipes are far enough apart to rank reliably.
HellaSwag (CORRECT PROB): low noise and high decision accuracy, consistent with its improved predictability under this metric. The paper states: "at 150M with CORRECT PROB tasks like HellaSwag succeed with low run-to-run variance" (Figure 5 caption).
SocialIQA (CORRECT PROB): high spread and high decision accuracy. The paper notes: "tasks like SocialIQA widely spread the performance assigned to different pretraining data" (Figure 5 caption), making reliable distinctions possible despite other challenges.
BoolQ (ACCURACY): very high noise relative to spread, yielding decision accuracy near 0.5. This explains why BoolQ showed essentially no predictability in Figure 2 until near the target scale—at 150M, the seed-to-seed variability swamps any genuine recipe differences.

How proxy metrics shift tasks in this space. The value of Figure 5 extends beyond explanation—it shows the mechanism by which proxy metrics improve decision accuracy. Switching from ACCURACY to CORRECT PROB tends to move points down (reducing noise) and/or right (increasing spread), moving them toward the favorable lower-right quadrant. For example:

ARC Easy (CORRECT PROB) moves substantially right compared to ARC Easy (ACCURACY)—the spread increases, making recipes more distinguishable.
HellaSwag (CORRECT PROB) moves down—the noise decreases, making the existing spread more reliable.
Some tasks see both improvements simultaneously, while others see negligible movement (explaining why proxy metrics do not help uniformly).

The paper summarizes: "Using CORRECT PROB sees wider spreads or reduced noise for many tasks" (Section 3.4). This diagnostic framework is not just descriptive—it is potentially actionable for practitioners designing their own small-scale evaluations. By measuring spread and noise for their specific tasks and data-recipe candidates, they can diagnose whether prediction failures are due to insufficient recipe differentiation (increase spread by finding more discriminative benchmarks or metrics) or excessive measurement noise (increase the number of seeds, evaluation items, or training tokens).

Ablation Studies and Robustness Checks

The paper's ablation studies are primarily structural—variations in prediction methodology and evaluation design that test the robustness of the main findings—rather than isolated component removals. The following systematically cover the non-trivial variations examined.

Scaling law functional form (2-parameter vs. 3-parameter vs. 5-parameter): Among the 8 scaling law variants tested, the simpler forms consistently outperform more complex ones. The 2-parameter and 3-parameter fits "all achieve among the top decision accuracy" (Section 3.2), while the 5-parameter (N, D) fit that separates model size and data tokens performs substantially worse. Appendix Table 4 provides prediction error numbers: the 3-parameter baseline achieves absolute error of 3.1%, the 2-parameter achieves 3.2%, while the 5-parameter fit reaches 65.4%—more than an order of magnitude worse. This suggests that with only one token-parameter ratio available in DATADECIDE, the 5-parameter form is severely underconstrained and overfits to noise. The paper notes this expected limitation: "as our suite only includes one token-parameter ratio, we use the simplified 3 parameter baseline" (Section 2.2).

Single-step vs. two-step fitting: The paper tests whether collapsing the compute-to-loss and loss-to-accuracy steps into a single function improves predictions. It does not. The single-step variants (Equations 6 and 7 in Appendix C) perform dramatically worse than their two-step counterparts: the 3-parameter single-step fit achieves 42.3% absolute error vs. 3.1% for the two-step 3-parameter fit; the 5-parameter single-step achieves 17.4% vs. 65.4%—still terrible but with the error structure reversed. The paper does not analyze this reversal in detail, but it is consistent with the single-step approach conflating two qualitatively different scaling processes (the power-law decay of loss and the sigmoidal saturation of accuracy) into one function, making the joint optimization harder.

Helper points: Adding an artificial point (L = 0.0, Acc = 1.0) when fitting Equation 2 is tested both alone and in combination with early-checkpoint filtering. Appendix Table 4 shows small improvements in prediction error from helper points (3-parameter + helpers achieves 2.8% vs. 3.1% for the baseline), but these improvements do not translate into visible gains in decision accuracy in Figure 3—the stars for helper-point variants overlap with the baseline stars. This suggests that the helper point primarily improves the numerical stability of the sigmoid fit (preventing unbounded upper asymptotes) without substantially changing the ranking implications.

Filtering early checkpoints (>50%): Excluding the first 50% of intermediate checkpoints when fitting the loss-to-accuracy sigmoid (Equation 2) is tested because early-training checkpoints may have unstable loss-accuracy relationships due to learning rate warmup and optimization transients. The paper reports that this modification produces small improvements in prediction error (3-parameter + >50% achieves 2.9% absolute error) but, like helper points, does not visibly shift decision accuracy in Figure 3. The paper states that the technique "reduces noise from high-loss early training points and often improves the fit for extrapolation" (Appendix C), but the fit improvement is apparently not large enough to affect the pairwise ranking decisions that decision accuracy measures.

Combining helper points with early-checkpoint filtering: The combination (abbreviated as "3-parameter with helpers and >50% checkpoints" in Figure 3 and Appendix Table 4) achieves the numerically lowest prediction error (2.6% absolute) among all variants. However, "these variants also roughly follow the compute-decision frontier in Figure 3" (Appendix C)—the incremental improvements in prediction error do not translate into crossing the single-scale decision-accuracy frontier.

Model size selection for multi-scale fits: The paper systematically explores which subsets of model sizes to include in scaling law fits. Using the three smallest models (4M, 6M, 8M) as a starting point and progressively adding larger sizes tests whether more expensive observations help, while starting with the largest small models and adding smaller ones tests whether noisy small models degrade predictions. The paper reports that "each of the scaling law variants are defined formally in Appendix C" and that multi-scale predictions use "{{s1, ..., sk} | 3 ≤ k ≤ 14}" and "{{sk, ..., s14} | 2 ≤ k ≤ 11}" (Section 3.2), but does not break out the decision accuracy of individual model-size combinations within each variant. Figure 3 shows the aggregate results: the best multi-scale predictions (those near the frontier) come from the 2-parameter and 3-parameter fits using intermediate numbers of scales, but the specific combinations are not enumerated.

Length normalization for proxy metrics (character vs. token): The paper states that character-level normalization "is empirically optimal for most of the tasks that we observe" (Section 3.3), but Figure 4 only shows character-normalized results. The token-normalized variants are not plotted in the main figures, so the magnitude of the difference between normalization schemes cannot be assessed from the paper. The footnote in Appendix Table 3 defines both normalization methods but no quantitative comparison is provided.

The ReST^EM revision model ablation (from the prior context, not this paper): Not applicable—this paper does not include revision models or reinforcement learning fine-tuning. The ablation studies are confined to variations in prediction methodology and evaluation design.

Code and math task evaluation (Figure 6): This serves as a cross-domain robustness check, testing whether the proxy-metric findings from OLMES multiple-choice tasks extend to generation-style benchmarks. The result that code tasks (HumanEval, MBPP) become predictable with CORRECT PROB while math tasks (GSM8K, Minerva) do not establishes a boundary condition: the proxy-metric approach works when small models can assign probabilistically meaningful scores to correct continuations even if they cannot produce them as the argmax, but fails when the models are so far below the capability threshold that even their likelihoods carry no signal. The paper does not explore whether alternative proxy metrics (e.g., loss on the correct completion, or likelihood of the problem statement itself) might recover predictability for math tasks.

Critical Assessment

The paper makes four central claims in its abstract and introduction: (1) single-scale ranking achieves ~80% decision accuracy on the OLMES macro-average at the 1B target scale; (2) no scaling law method among 8 baselines exceeds the compute-decision frontier set by single-scale predictions; (3) continuous proxy metrics (CORRECT PROB, TOTAL PROB) make benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval >80% predictable with just 0.01% of target compute; and (4) better decisions are driven by low run-to-run variance and wide spread of performance across recipes. Each requires careful scrutiny against what the experiments actually demonstrate, what they do not test, and what qualifications circumscribe the findings.

Claim 1: Single-scale ranking achieves ~80% decision accuracy on the OLMES macro-average. This claim is supported with important caveats about the macro-average and the scale range. The ~80% figure is visible in Figure 1 (right): the single-scale curve crosses 0.80 at approximately 1–10 %C, corresponding roughly to a fully trained 150M model. At 100 %C (the 1B model itself, used circularly), it reaches ~0.85–0.88. So "~80%" is the correct ballpark, but the exact number depends on precisely which small scale is chosen and whether one uses a final checkpoint or an intermediate one. The claim correctly characterizes the central tendency.

However, the macro-average masks enormous per-task variation (Figure 2). Decision accuracy on the macro-average reaches ~80% not because every task is ~80% predictable but because highly predictable tasks (ARC Easy at ~95%, MMLU at ~85%) pull up the average while unpredictable tasks (BoolQ, SocialIQA at near-chance) pull it down. A practitioner who cares primarily about HellaSwag would find single-scale ranking at 150M achieving only ~55–60% decision accuracy—far below the 80% headline. The paper does not hide this (Figure 2 is prominently placed), but the abstract's framing could be read as suggesting uniform reliability across benchmarks, which the data clearly refute.

A more fundamental qualification is that the ~80% figure is specific to the 25 recipes, the 1B/100B-token target scale, the 5× Chinchilla training ratio, and the OLMES evaluation suite. The paper acknowledges this implicitly by releasing DATADECIDE as a testbed for others to replicate on different configurations, but the generalizability of the 80% number to other recipe sets, other target scales, or other evaluation suites is completely untested. It is plausible that different recipe sets—for example, recipes that differ more subtly (only varying deduplication thresholds by small amounts) or that differ more dramatically (mixing proportions of entirely disjoint source domains)—would show different single-scale decision accuracy. The paper provides one piece of evidence for this possibility in Figure 5: the spread across recipes varies substantially by task and metric, and smaller spread would mechanically reduce decision accuracy regardless of the prediction method.

Claim 2: No scaling law method among 8 baselines exceeds the single-scale frontier. This claim is supported for the specific variants tested and the specific scale range studied, but the comparison is not fully fair due to the seed-averaging asymmetry. As discussed above, single-scale predictions in Figure 3 are averages over 3 seeds (reducing variance), while multi-scale predictions use only seed 1 (full variance). The paper is transparent about this limitation: "we make only one prediction attempt with the default fully trained random seed, as final checkpoints are required for fitting the first step of these scaling law variants but are not available for all seeds" (Section 3.2). If multi-scale predictions could be averaged over 3 seeds, their variance would decrease and their mean decision accuracy would likely increase—potentially closing the gap with single-scale ranking, or even exceeding it in regions where crossovers matter. The paper does not estimate how large this effect might be.

A deeper concern is that the 8 scaling law variants are all minor variations on a single two-step functional form (power law for loss, sigmoid for accuracy). They differ in number of parameters, inclusion of helper points, and early-checkpoint filtering, but all share the same fundamental extrapolation strategy. It is possible—perhaps likely—that fundamentally different approaches (e.g., learned predictors that use evaluation results from multiple tasks jointly, or methods that incorporate the text content of the data recipes as features, or Bayesian hierarchical models that share information across recipes) would perform differently. The paper's claim is specifically about these 8 baselines, not about all possible scaling law methods, and the authors explicitly frame DATADECIDE as a platform for future methods to test against: "Future scaling law methods can be assessed on DATADECIDE" (Figure 3 caption). This is a fair scoping, but readers should not conclude that "scaling laws cannot beat single-scale ranking"—only that none of the 8 straightforward variants tested here do so, under the specific experimental conditions.

Claim 3: Continuous proxy metrics make several benchmarks >80% predictable with 0.01% of target compute. This claim is strongly supported for the specific tasks named, with Figure 4 and Figure 6 providing direct evidence. For ARC Easy and ARC Challenge, CORRECT PROB achieves >80% decision accuracy at 10^-4 to 10^-3 %C (Figure 4)—well below 0.01% of target compute. For code tasks (HumanEval, MBPP), CORRECT PROB at 4M–60M scale achieves ~80% (Figure 6). The claim is quantitative and the evidence matches. However, two qualifications matter:

First, the ">80% predictable" framing applies to some benchmarks but not all. For PIQA, SocialIQA, and WinoGrande, no proxy metric achieves 80% decision accuracy at any compute level below ~10% of target (Figure 4). The paper does not overclaim here—the abstract specifically lists MMLU, ARC, HellaSwag, MBPP, and HumanEval—but readers should note the absence of several OLMES tasks from that list.

Second, the "0.01% of target compute" figure is approximate and task-dependent. The smallest scales in Figures 4 and 6 (4M models, early checkpoints) correspond to roughly 10^-5 to 10^-4 %C for some tasks—one to two orders of magnitude below 0.01%. For other tasks (e.g., MMLU with CORRECT PROB), crossing the 80% threshold occurs closer to 10^-3 to 10^-2 %C—within the 0.01% ballpark but at the upper end. The paper's "0.01%" is a reasonable order-of-magnitude characterization, not a precise invariant.

A deeper issue is whether the proxy metrics are predicting genuine recipe quality or merely correlating with it for reasons that might not hold in other settings. CORRECT PROB at small scales measures how much probability the model places on correct answer continuations. This could reflect genuine task capability that will later manifest as accuracy, or it could reflect superficial features of the pretraining data (e.g., exposure to the specific phrasing of answer options) that happen to correlate with recipe quality in DATADECIDE's recipe set but would not generalize. The paper does not provide a causal analysis of why CORRECT PROB works—only that it does. For a practitioner considering adopting this approach for a different set of recipes or a different target scale, the lack of mechanistic understanding creates uncertainty about whether the proxy-metric advantage will transfer.

Claim 4: Better decisions are driven by low noise and wide spread. This claim is supported by the diagnostic framework in Figure 5, but the framework is correlational, not causal. The figure shows that tasks with favorable noise/spread characteristics achieve higher decision accuracy, which is definitionally true (decision accuracy mechanically improves when signal exceeds noise) but does not demonstrate that changing noise or spread would cause improved decision accuracy. The paper implicitly acknowledges this by presenting Figure 5 as a diagnostic for understanding existing results, not as an intervention: "What underlies differences in decision accuracy when benchmarks and metrics change?" (Section 3.4). It is a decomposition of variance, not a causal model.

The claim that "Using CORRECT PROB sees wider spreads or reduced noise for many tasks" (Section 3.4) is factually supported by the movement of points in Figure 5 when switching from ACCURACY to CORRECT PROB. But the mechanism is not explained: why does CORRECT PROB reduce noise on HellaSwag but not on MMLU? Why does it increase spread on ARC Easy but not on PIQA? Without understanding the mechanism, practitioners cannot predict a priori which proxy metric will help for which task—they must try them all and consult a figure like Figure 5, which requires having the target-scale ground truth (defeating the purpose of prediction). This limits the practical actionability of the diagnostic: it tells you that something worked or failed post hoc, but does not tell you whether it will work ex ante for a new task or new recipe set.

Experiments that would have strengthened the paper. Several extensions would substantially increase confidence in the findings, and their absence is worth noting:

Multi-seed multi-scale predictions. The asymmetry in seed averaging between single-scale and multi-scale predictions is the most important uncontrolled variable in Figure 3. Running multi-scale predictions on all 3 seeds (which would require training seeds 2 and 3 to completion at non-target sizes—a substantial compute cost) would close this gap and allow a clean comparison. Without it, the paper cannot rule out the possibility that multi-scale methods, with comparable variance reduction, would indeed exceed the single-scale frontier.
Varying the target scale. All analyses use 1B parameters / 100B tokens as the target. The paper does not test whether the relative performance of single-scale vs. multi-scale methods changes when the target is 500M or 2B or 7B. It is possible that crossovers become more frequent and more consequential at larger scale gaps, making multi-scale methods increasingly valuable relative to single-scale ranking. The paper acknowledges this as a limitation: "the scope of our work is limited to just one ratio of tokens to parameters" (Section 5), but does not discuss the scale-gap dimension.
Out-of-distribution data recipes. All 25 recipes are drawn from a similar space of web-text corpora with varying filtering and mixing. Whether the findings generalize to recipes with fundamentally different characteristics—e.g., code-heavy vs. text-only, multilingual vs. English-only, synthetic vs. natural data—is unknown. The paper's recipe diversity, while greater than prior suites, is still concentrated in the Common Crawl-derived web-text domain.
Statistical significance of decision-accuracy differences. The paper reports decision accuracy as point estimates with standard deviation shading (from 3-seed averaging for single-scale), but does not perform formal hypothesis tests comparing single-scale and multi-scale methods at equivalent compute budgets. The visual overlap of stars with the single-scale curve in Figure 3 is suggestive but not conclusive—statistical tests could reveal whether the best multi-scale variants are significantly below the frontier or indistinguishable from it.
Ablation on the number of recipes. Decision accuracy is computed over 300 pairwise comparisons. The paper does not test whether the relative performance of prediction methods depends on the number of candidate recipes. With fewer recipes (e.g., a practitioner comparing only 3–5 options), even methods with lower decision accuracy on the full 25 might achieve perfect or near-perfect pairwise accuracy, making the distinctions in Figures 1–4 irrelevant. Conversely, with more recipes, small differences in method quality would be amplified.
Direct comparison of proxy metrics at identical compute using different model sizes. Figure 4 shows CORRECT PROB at 10^-4 %C outperforming ACCURACY at the same compute, but this conflates two variables: the metric and the model size/checkpoint. A cleaner ablation would hold the model checkpoint fixed and compare decision accuracy from CORRECT PROB vs. ACCURACY evaluated on that same checkpoint—this would isolate the effect of the metric from the effect of the scale. The paper's current presentation makes it slightly ambiguous whether the proxy-metric advantage is truly about the metric or about the fact that CORRECT PROB saturates early (i.e., it reaches its maximum discriminative power at smaller scales, while ACCURACY continues improving). Distinguishing these interpretations would require showing that, at a fixed small model size, CORRECT PROB achieves higher decision accuracy than ACCURACY—a within-checkpoint comparison that the paper does not tabulate directly.

Boundary conditions and conditional claims. The paper's findings are explicitly conditioned on:

Scale range: 4M to 1B parameters, 0.4B to 100B tokens. The authors state that "one would have to select different tasks when targeting a larger scale" (Section 2.4), implying that the specific decision-accuracy numbers may not transfer to substantially larger target scales.
Training ratio: 5× Chinchilla (100 tokens per parameter). The paper notes this captures "the typical case" of overtraining (Section 5), but the relative effectiveness of prediction methods could differ at other ratios.
Task type: Multiple-choice question answering with a cloze formulation. The extension to code tasks (Figure 6) suggests partial generalizability to generation tasks, but math tasks remain unpredictable, indicating that task structure matters fundamentally.
Recipe diversity: 25 recipes spanning web-text corpora with varying filtering, mixing, and domain ablation. Findings about spread (Figure 5) are mechanically dependent on which recipes are included—a recipe set with smaller genuine differences would show lower decision accuracy for all methods.

These boundary conditions are not weaknesses—they are the honest scope of a carefully designed measurement study. The paper's contribution is not a universal theorem about data decisions but an empirical calibration within a well-specified regime, plus the infrastructure to extend that calibration to other regimes. The authors' repeated emphasis on DATADECIDE as an extensible platform ("Future research can extend DATADECIDE with little extra compute by running new evaluations on our checkpoints") signals that they view the specific numbers as less important than the measurement methodology and the baselines they establish.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Numbers

The assumption or constraint. The entire compute-optimal framework depends on knowing each prompt's difficulty before allocating the inference budget. The paper's method for estimating difficulty—generating 2048 samples per question and averaging either ground-truth correctness (oracle bins) or PRM final-answer scores (predicted bins)—is extraordinarily expensive. Section 3.2 acknowledges this directly:

"this still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported ~4× efficiency gains over best-of-N (e.g., 16 generations matching 64 in Figure 4; 64 generations matching 256 in Figure 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, difficulty estimation would require 2048 generations × (evaluation cost) per prompt. For the MATH benchmark with 500 test questions and 2048 samples per question, this is over 1 million generations of estimation overhead—dwarfing the largest test-time budgets studied (256–512 generations per question). If this cost were included in the %C calculation, the compute-optimal curves in Figures 4 and 8 would shift substantially to the right, potentially erasing the reported advantage over best-of-N at equivalent total compute. The ~4× figure should therefore be interpreted as an upper bound on achievable efficiency under the optimistic assumption that difficulty can be obtained for free, not as a realized deployment gain.

What evidence exists in the paper. The paper quantifies the cost: "generating 2048 samples and scoring them" (Section 3.2) per question. It also shows that the predicted (PRM-based) difficulty bins perform nearly as well as oracle bins (the curves "largely overlap" in Figures 4 and 8), which confirms that ground-truth labels are not required—but this does not address the 2048-sample generation cost, which is the dominant term. Figure 4 and Figure 8 both display compute-optimal scaling curves that begin after difficulty is known. The paper does not report any experiment where difficulty estimation cost is included in the total budget, nor does it show how decision accuracy varies as a function of the number of samples used for difficulty estimation (e.g., does 256 samples suffice? 64?).

Mitigation status. The paper explicitly flags this as "a key avenue for future work" (Section 3.2) and suggests "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). It also notes the possibility of adaptive difficulty estimation where "compute spent assessing difficulty versus compute spent solving the problem" is treated as an exploration-exploitation tradeoff (Section 3.2). However, none of these approaches are implemented or evaluated. The limitation is acknowledged but entirely unresolved, meaning the headline efficiency numbers are not achievable in current form without a separate, unaccounted-for cost.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021) with 500 test questions and PaLM 2-S* (Codey) as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this belief is not tested. The paper does not run experiments on any other reasoning benchmark (e.g., GSM8K, MMLU, ARC, HumanEval), any other model family (e.g., LLaMA, GPT, Mistral), or any non-math domain.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific in ways that materially affect their generalizability:

PRM quality and over-optimization behavior. The PRM is trained using Monte Carlo rollouts from PaLM 2-S* on MATH training data. A different base model with different calibration properties, different error patterns, or different output distributions might produce a PRM with different over-optimization thresholds. The specific difficulty-bin boundaries where beam search hurts rather than helps (Figure 3, right) might shift, changing the optimal compute-optimal policy.
Revision model training depends on base model capabilities. The revision model's ability to learn from incorrect in-context examples (constructed via edit-distance pairing) depends on the base model's in-context learning capabilities and its tendency to produce "close but incorrect" solutions. Models with different failure modes—e.g., producing answers that are structurally very different from the correct one when wrong—might yield lower-quality revision training data, potentially degrading the sequential revision benefits documented in Figure 6.
MATH is purely symbolic reasoning. MATH consists of competition-level math problems requiring multi-step symbolic deduction. It is unclear whether the difficulty-dependent patterns—beam search over-optimizing on easy problems, sequential revisions helping on easy problems but requiring parallel exploration on hard ones—generalize to other reasoning domains. Code generation, for instance, has the property that partially correct solutions can be verified by unit tests; this might change the relative effectiveness of search vs. revisions in ways the MATH results do not capture. Conversational tasks, factual QA, and open-ended generation lack the clean correctness signals that enable PRM training and difficulty estimation, meaning the entire framework may not transfer to those domains.
The test set is 500 questions. With five difficulty quintiles, each bin contains approximately 100 questions. The compute-optimal policy is selected via two-fold cross-validation within each bin, meaning strategy selection is based on approximately 50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, so it is unclear whether the observed differences between oracle and predicted difficulty bins (e.g., the ~3 percentage point gap at 256 generations in Figure 8) are statistically reliable or within sampling noise. A different random split of the 500 questions could potentially change which strategies are selected as optimal per bin, altering the reported efficiency gains.

What evidence exists in the paper. The paper provides no cross-benchmark or cross-model evidence. The claim that PaLM 2-S* is "representative" appears in Section 4 without empirical support. The 500-question test set size is specified in Section 4, and the cross-validation protocol is described in Section 3.2, but no sensitivity analysis on split stability is provided.

Mitigation status. The authors acknowledge this limitation only indirectly: "the scope of our work is limited to just one ratio of tokens to parameters" (Section 5) addresses a different dimension, and Section 8's call for future work on "extension to other domains and modalities" implicitly acknowledges the single-benchmark issue. The paper makes no attempt to mitigate this within its own experiments—it is purely a scope limitation. A practitioner considering adopting compute-optimal test-time scaling for a non-math domain or a different model family would need to run their own validation, for which the paper provides a methodology but no pretrained components (PRM, revision model, difficulty estimator) that would transfer.

The `~14×` Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 pits PaLM 2-S* with compute-optimal test-time scaling against a model with approximately ~14× more parameters. The larger model is trained by scaling parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal paradigm of scaling both data and parameters equally (Hoffmann et al., 2022). Section 7 acknowledges this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding—no majority voting, no best-of-N, no search, no revisions.

The consequence. Both design choices make the pretraining baseline weaker than it would be under a more competitive setup, potentially inflating the reported advantage of test-time compute.

Non-Chinchilla-optimal pretraining. A model trained with ~14× more total FLOPs allocated optimally across both parameter count and data quantity would likely achieve higher downstream accuracy than a model that scales only parameters. The paper's comparison therefore may overstate how much test-time compute can substitute for pretraining compute. The magnitude of this overstatement depends on how far the parameter-scaled model is from the Chinchilla-optimal frontier—the paper provides no estimate.
Greedy decoding for the larger model. Giving the larger model even a modest test-time compute budget—best-of-8, or majority voting over 4 samples, or a single round of revision—would create a much stronger baseline. The paper's framing (Section 7) asks whether test-time compute with a smaller model can outperform a larger model without test-time compute, which is a valid but narrow question. A more practically relevant question for deployment decisions is whether test-time compute with a smaller model can outperform a larger model given the same inference-time augmentation, or at least some inference-time augmentation. By denying the larger model any test-time compute, the comparison stacks the deck in favor of the smaller model—the paper is comparing a small model with compute-optimal inference to a large model with no inference optimization at all. This is not a symmetric comparison of pretraining vs. inference compute; it is a comparison of one optimized system against one unoptimized system that happens to differ in model size.

What evidence exists in the paper. The FLOPs accounting in Section 7 defines the comparison clearly, making the asymmetry transparent to careful readers. The bar charts in Figure 1 (top-right and bottom-right) show the relative advantage of test-time compute, with breakdowns by difficulty and by R (the inference-to-pretraining token ratio). The paper does not report how the larger model would perform with even modest test-time compute augmentation, nor does it provide results for a Chinchilla-optimally trained larger model. These comparisons are entirely absent.

Mitigation status. The paper is transparent about the training-data assumption ("we choose this setting as it is representative") but does not attempt to bound how much this choice affects the results. The greedy-decoding asymmetry is not acknowledged as a limitation at all—the comparison is simply defined this way. A fairer comparison (larger model with some test-time compute budget, or with Chinchilla-optimal pretraining) is left entirely to future work. The practical implication is that the Section 7 results—particularly the positive findings on easy and medium questions at low R—should be interpreted as an upper bound on the advantage of test-time compute over pretraining, and the true crossover point where pretraining becomes preferable may occur at smaller scale gaps or higher difficulty levels than the paper reports.

The Revision Model Has a ~38% Correct-to-Incorrect Reversion Rate and Revision Training Is Fragile

The assumption or constraint. The revision model is trained only on sequences where all in-context answers are incorrect, followed by a correct target. At test time, when the revision chain produces a correct answer, the model has no training signal for what to do—it may produce another revision that changes the correct answer to an incorrect one. Section 6.1 reports:

"approximately 38% of correct answers produced during a revision chain get 'revised' back to incorrect answers in the subsequent step"

The paper mitigates this by selecting the best answer across the entire revision chain (via majority voting or verifier-based selection) rather than always taking the final revision. But this mitigation is a patch, not a solution—it means the revision model is generating sequences where nearly two-fifths of correct intermediate outputs are actively degraded by subsequent revisions, and the system relies on an external selection mechanism to recover the correct answer from somewhere in the chain.

The consequence. The 38% reversion rate imposes a fundamental limit on the efficiency of sequential revision: as the chain grows longer, the probability that a correct answer produced at step t survives to the end of the chain without being reverted decreases. The selection mechanism (picking the best from the chain) can recover these answers, but only if the verifier or majority vote correctly identifies the correct answer among the chain's outputs—which is not guaranteed, especially on hard problems where the verifier itself is unreliable. More importantly, the reversion rate means that the revision model is wasting compute: a substantial fraction of its sequential budget is spent generating revisions that actively harm solution quality, and the system must then spend additional selection compute to undo that harm.

The fragility of revision training is further highlighted by the ReSTᵉᵐ experiment in Appendix K (Figure 16). Attempting to further optimize the revision model using on-policy RL-style training (ReSTᵉᵐ; Singh et al., 2024) caused performance to degrade substantially with sequential revisions: "fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio." The paper hypothesizes that "on-policy data collection in ReSTᵉᵐ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This negative result indicates that the positive revision results in Section 6 depend on specific, fragile training choices (offline data construction, edit-distance-based pairing) that do not necessarily survive when the training procedure is modified. A practitioner attempting to replicate the revision model approach with a different base model or a different data construction pipeline may encounter similar instabilities.

What evidence exists in the paper. The 38% reversion rate is explicitly reported in Section 6.1. The ReSTᵉᵐ degradation is documented in Appendix K (Figure 16). The within-chain selection mitigation is described alongside the reversion rate discussion, and its effectiveness is visible in Figure 6 (left), where per-step pass@1 plateaus rather than declining despite the reversion rate—confirming that the selection mechanism is recovering correct answers from earlier in the chain. However, the paper does not quantify how much performance is lost due to the reversion rate relative to a hypothetical revision model that knows when to stop revising.

Mitigation status. The paper mitigates the reversion problem via within-chain selection (majority voting or verifier-based selection), but does not attempt to solve the underlying training-data asymmetry. A more principled solution—such as training the revision model on trajectories that include correct answers in context followed by a "no revision needed" or "stop" token—is not explored. The paper does not discuss this as a direction for future work, treating the selection-based mitigation as sufficient for the experimental purposes. The ReSTᵉᵐ degradation is reported as a cautionary finding but not addressed; the paper does not investigate why on-policy data collection fails or how to make revision training more robust.

Sequential Revisions Introduce a Latency Wall-Clock Time Penalty That Is Ignored by the FLOPs-Only Cost Model

The assumption or constraint. The paper measures computational cost exclusively in terms of total FLOPs (or "generations"), following standard scaling-law conventions where "costs associated with training a model are captured well enough by FLOPs = 6ND" (Section 5.3). This treats all FLOPs as interchangeable—a budget of 64 generations costs the same whether spent as 64 parallel samples (all generated simultaneously) or 64 sequential revisions (each depending on the previous one). The paper's compute-optimal policy frequently favors sequential-heavy strategies: for easy problems, purely sequential revisions are optimal (Figure 7); for search, the recommended strategies involve serial beam search steps.

The consequence. The FLOPs-only cost model ignores latency—the wall-clock time required to produce a final answer. Sequential revisions are inherently serial: revision t+1 cannot begin until revision t is complete. Parallel best-of-N can be executed simultaneously given sufficient hardware (e.g., batched across multiple GPUs or multiple inference servers). A strategy that allocates 128 generations as 64 sequential × 2 parallel chains takes approximately 64× longer wall-clock time than one that runs 128 parallel samples simultaneously, even though both consume the same total FLOPs.

For latency-sensitive applications—interactive assistants, real-time decision-making, API services with strict timeout requirements—the sequential-heavy strategies favored by the compute-optimal policy on easy and medium problems may be impractical regardless of their FLOPs efficiency advantages. A user waiting for an answer to a simple math question may prefer a slightly lower-accuracy answer returned in 2 seconds (from parallel best-of-N) over a slightly higher-accuracy answer returned in 30 seconds (from a long revision chain). The paper's compute-optimal policy optimizes only for accuracy per FLOP, not for accuracy per wall-clock second, and these two objectives can be in direct tension.

What evidence exists in the paper. The paper never discusses latency, wall-clock time, throughput, or the serial vs. parallel execution implications of its recommended strategies. The cost model in Section 5.3 is defined purely in terms of "generations" (number of complete solutions sampled) and Section 7 extends this to FLOPs accounting. Figures 7 and 8 show optimal sequential-to-parallel ratios that are often heavily skewed toward sequential (e.g., ratios of 2^1 to 2^3, meaning up to 8 sequential steps per parallel chain). The implications of these ratios for wall-clock latency are not mentioned.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not propose a latency-aware cost model, and does not discuss the tradeoff between FLOPs efficiency and wall-clock latency. A practitioner deploying these methods in a latency-sensitive setting would need to independently evaluate whether the compute-optimal policy's sequential bias is acceptable, or whether a latency-constrained variant (e.g., capping maximum chain length, or switching to parallel strategies when latency budgets are tight) is needed.

Hard Problems (Difficulty Bin 5) Remain Fundamentally Unsolved Across All Methods

The assumption or constraint. The compute-optimal framework is fundamentally bounded by the base model's capability: if the model cannot produce correct solutions at any meaningful rate, no amount of search or revision can extract them. This is not an assumption per se—it is a logical necessity—but it represents a hard ceiling on what test-time compute can achieve. Section 5.3 makes this explicit: "On the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

The consequence. Test-time compute can amplify existing capability—taking a model that occasionally produces correct answers and making it produce them more reliably—but it cannot create capability where none exists. For problems where the base model's pass@1 is near zero, the proposal distribution contains essentially no correct solutions to find (via search) or refine (via revisions). Every method studied—best-of-N, beam search, lookahead search, sequential revisions, and their compute-optimal combinations—fails on bin 5 questions. The evidence is stark: in Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets; in Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio; in the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both revisions and PRM search, and the ~14× larger model outperforms test-time compute across all R regimes for hard questions (e.g., −52.9% relative disadvantage for PRM search at R ≫ 1).

This means the paper's approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. If a deployment involves problems of this difficulty—and the paper's difficulty estimation framework would correctly flag them as bin 5—the optimal strategy is not to allocate test-time compute at all (it would be wasted) but to escalate to a more capable model or to flag the problem for human intervention. The compute-optimal framework can tell you that a problem is too hard for test-time compute to help, but it cannot tell you how to solve it anyway.

What evidence exists in the paper. The near-zero performance on bin 5 is documented across all experimental sections: Figure 3 (right, search), Figure 7 (right, revisions), Figure 9 (FLOPs-matched comparison). The paper is candid about this limitation: the Section 7 takeaway box explicitly notes that on hard problems, pretraining is almost always more effective. Figure 9 visually demonstrates the flat-lining of test-time compute scaling for bin 5, contrasted against bins 1–2 where test-time compute improves substantially with budget.

Mitigation status. The paper acknowledges this limitation transparently—it is a central finding, not a hidden weakness—but does not attempt to mitigate it. The limitation is inherent to the test-time compute paradigm: no inference-time strategy can find correct solutions that the base model cannot generate at all. The practical implication is a clear boundary condition: compute-optimal test-time scaling is valuable when problems are within the base model's capability range (the model occasionally produces correct answers), and it provides no benefit when problems are genuinely beyond that range. The difficulty estimation mechanism is designed to detect this boundary and prevent wasted compute, but it cannot transcend it. Improving the base model's capability on these problems requires pretraining on different data, scaling to larger models, or using fundamentally different architectures—none of which are within the scope of test-time compute optimization.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes a new evaluation paradigm for data selection methodology, shifting the field's criterion from prediction error (how close is the predicted number?) to decision accuracy (how often does the prediction make the right choice?). This is not a new prediction algorithm—it is a new way of measuring whether any algorithm works for its actual intended purpose. The shift matters because it resolves an unacknowledged tension in prior work: scaling law methods are validated on prediction error, but practitioners use them to make discrete choices among alternatives, and low prediction error does not guarantee correct choices. By providing the first testbed where the counterfactual is observed—training large models on all candidate datasets, including losers—DATADECIDE makes it possible to measure decision accuracy directly, and the measurement reveals that the two evaluation criteria can diverge. The best multi-scale scaling law variants achieve prediction errors comparable to single-scale ranking (Appendix Table 4: ~3% absolute error for both) yet lie on or below the single-scale decision-accuracy frontier (Figure 3). This is a concrete empirical demonstration that prediction error and decision quality are not interchangeable metrics, and it implies that the scaling laws literature has been optimizing for the wrong target.

This is not a paradigm shift—the underlying methods (single-scale ranking, two-step scaling law fits) are unchanged—but it is a diagnostic reframing with sharp practical consequences. It tells the field that the bottleneck in data-decision methodology is not the functional form of the extrapolation function (the 8 variants tested span reasonable design choices and none beat the baseline), but rather the quality of signal extractable from small-scale experiments. The paper's own proxy-metric findings confirm this: switching from discrete accuracy to continuous likelihoods (CORRECT PROB, TOTAL PROB) improves decision accuracy by amounts that dwarf any difference between scaling law variants, and even makes previously unpredictable tasks (HumanEval, MBPP) >80% predictable (Figure 6). This redirects research attention away from curve-fitting sophistication and toward better ways to measure capability at small scales—a refocusing that is the paper's most durable conceptual contribution.

The work also validates a widely-used but previously unverified practice. The finding that single-scale ranking achieves ~80% decision accuracy on the OLMES macro-average at the 1B target scale (Figure 1 right) means that teams like DCLM (Li et al., 2024), who used 150M-parameter experiments to guide data curation for much larger models, were not merely cutting costs—they were using a method that, on current evidence, is not surpassed by more expensive multi-scale alternatives. This is a non-obvious result. A priori, single-scale ranking should fail whenever data recipes exhibit ranking crossovers (A beats B at small scale, B beats A at large scale), and multi-scale methods should in principle capture such crossovers. The empirical finding that crossovers are either rare, small in magnitude, or difficult to distinguish from run-to-run noise—at least within the 4M-to-1B parameter scale range—provides retrospective justification for the dominant practice in data curation. It does not guarantee that single-scale ranking will remain optimal at larger scale gaps (the paper studies up to 1B; a jump from 150M to 70B might exhibit more crossovers), but it establishes that for the scale range most accessible to academic and small-industry researchers, the simple approach works.

The paper also introduces a diagnostic framework for understanding why some benchmarks are more predictable than others, decomposing decision accuracy into the interplay of spread (recipe-to-recipe performance variation) and noise (seed-to-seed variation for the same recipe). Figure 5 operationalizes this decomposition and shows that improvements from proxy metrics often work by increasing spread, reducing noise, or both. This framework gives practitioners a tool for diagnosing failures in their own data-selection workflows: if a benchmark shows poor decision accuracy at small scale, the breakdown reveals whether the bottleneck is excessive measurement noise (addressed by more seeds, more evaluation items, or more training) or insufficient genuine differentiation among recipes (addressed by finding more discriminative benchmarks or metrics). This transforms unpredictability from an opaque endpoint into an actionable diagnostic.

Finally, the paper changes the economics of methodology research on data selection. Before DATADECIDE, validating a new prediction method required either access to proprietary training infrastructure (to train large models on multiple recipes) or reliance on off-the-shelf models that confound data with architecture and optimizer differences. By releasing over 30,000 model checkpoints, all pretraining corpora, and all evaluations, the paper makes it possible to test new prediction methods without any additional pretraining cost. The explicit framing of DATADECIDE as an extensible platform—"future research can extend DATADECIDE with little extra compute by running new evaluations on our checkpoints"—lowers the barrier to entry for methodology research and creates a shared benchmark against which progress can be measured. This is an infrastructure contribution that may outlast the paper's specific numerical findings.

Follow-Up Research This Work Enables

Cheap difficulty estimation for the compute-optimal framework. The most immediate bottleneck the paper identifies—though not the paper's own problem—is the cost of estimating difficulty before allocating inference compute. This work's methodology for measuring decision accuracy provides the evaluation framework for testing difficulty estimators: any proposed method (lightweight classifier, few-shot PRM scoring, adaptive estimation from a small initial sample) can be evaluated by comparing the decision accuracy achieved when using its difficulty bins versus oracle bins. A concrete experiment: train a linear probe on top of a frozen language model that takes only the question text (no generation) and predicts the difficulty quintile. Measure how much compute-optimal decision accuracy degrades when using predicted rather than oracle bins, and whether the degradation is small enough to justify the probe's near-zero inference cost. The paper's finding that predicted PRM-based bins "largely overlap" with oracle bins (Figures 4, 8) establishes the feasibility target: a cheap estimator should approach the PRM-based prediction accuracy without the 2048-sample generation cost.

Cross-model transfer of data-recipe rankings. The paper's 25 data recipes are evaluated only on models trained from the OLMo ladder with a fixed token-parameter ratio. A natural stress test is: do the rankings transfer to a different model architecture or training recipe? If a different architecture (e.g., a LLaMA-style model with a different tokenizer, activation function, or attention variant) trained on the same 25 recipes at the same scale produces a different ranking, then data decisions validated on one model family may not apply to another. DATADECIDE makes this test newly tractable because the pretraining corpora are released—a researcher can take the same 25 data recipes, train models from a different architecture family at a single small scale (e.g., 150M parameters), and compare the resulting ranking against the 1B OLMo target ranking released in the paper. A strong result would quantify the rank correlation between architectures and identify which types of data interventions (quality filtering thresholds? domain ablation? mixing ratios?) show the most architecture-dependent rankings.

Identifying the scale at which multi-scale methods overtake single-scale ranking. The paper's central negative result—that scaling laws do not beat single-scale ranking—is measured with a target scale of 1B parameters. As the scale gap between small experiments and the target grows (e.g., predicting 7B or 70B from experiments up to 1B), ranking crossovers may become more frequent and more consequential, potentially making multi-scale methods increasingly valuable. A direct extension: extend DATADECIDE with one or two larger target scales (e.g., 3B and 7B parameters, trained on 300B and 700B tokens at the same 5× Chinchilla ratio), using a subset of the 25 recipes to manage cost. Measure whether the single-scale advantage persists, shrinks, or reverses as the extrapolation distance increases. The paper's own observation that "the scaling trends we empirically observe cross over frequently" (Section 3.2) but that these crossovers are often within evaluation noise suggests a specific hypothesis: as the scale gap increases, genuine crossovers will increasingly dominate noise, and the multi-scale frontier will eventually cross above the single-scale one. Identifying that crossover point (in terms of scale ratio) would provide practitioners with a decision rule: "if your target is less than X× your experiment scale, use single-scale ranking; if more than X×, invest in multi-scale fitting."

Why do proxy metrics work for code but not math? Figure 6 shows that CORRECT PROB makes HumanEval and MBPP highly predictable from small models, while GSM8K and Minerva remain near chance regardless of metric. This asymmetry is empirically documented but unexplained. A targeted study: for a single code and a single math benchmark, analyze which small-model probability assignments drive the predictability difference. Is it that small models trained on math-heavy data assign higher probability to correct math answers across the board (making CORRECT PROB a sensitive measure of math exposure), while for code the probability on correct answers specifically tracks eventual problem-solving ability? Or is it that the difficulty of math items is such that no small model reaches the threshold where likelihoods become informative, and larger small models (beyond the paper's 60M test) would close the gap? A concrete experiment: run the same proxy-metric analysis on DATADECIDE for the full model ladder up to 1B, mapping out where CORRECT PROB for GSM8K begins to show predictive signal. If it never does even at 1B, the failure is fundamental to the task structure; if it emerges at 300M–750M, the failure is a threshold effect that larger small experiments would overcome.

Developing data-recipe embeddings for zero-shot ranking transfer. The paper's 25 recipes differ along interpretable axes: source dataset, quality filter type, quality filter threshold, domain ablations, mixing proportions. A natural follow-up: can we learn a mapping from descriptions of data recipes (or from small-scale evaluation results on a subset of recipes) to their likely large-scale performance, enabling zero-shot prediction for new recipes not in the training set? For example: train a model that takes as input the evaluation results of recipes A1–A20 at small scale (plus recipe-level features like "DCLM-base, filtered to top 7%") and predicts pairwise rankings for recipes B1–B5 at the 1B scale. DATADECIDE enables this by providing 25 recipes as training/evaluation data—a researcher could hold out 5 recipes, train a meta-predictor on the other 20, and measure whether the meta-predictor achieves above-chance decision accuracy on the held-out recipes. This would test whether data-recipe quality generalizes in a learnable way, or whether each recipe must be empirically evaluated at some scale.

Stress-testing the compute-decision frontier with adversarial recipe sets. The paper's 25 recipes, while diverse, are all drawn from the space of filtered Common Crawl web text. A negative result that would refine our understanding: construct a set of recipes specifically designed to produce ranking crossovers—for example, recipes that vary dramatically in domain composition (90% code vs. 10% code) or that mix high-quality small corpora with low-quality large corpora in ways that create predictable quality-vs-quantity tradeoffs. Measure whether single-scale ranking degrades substantially on such an adversarial set, and whether multi-scale methods recover. If single-scale ranking remains robust even against designed-to-fail recipe sets, that would strengthen the paper's practical recommendation; if it fails, that would establish clearer boundary conditions and motivate multi-scale methods for specific types of data interventions.

Practical Applications and Downstream Use Cases

Data curation teams can trust their small-scale ablations—with specific caveats. The paper's headline finding that single-scale ranking achieves ~80% decision accuracy on the OLMES macro-average provides direct validation for the standard workflow in data curation projects like DCLM. A team developing a new pretraining corpus can run ablations at 150M parameters, rank candidate recipes by their performance (both discrete accuracy and continuous likelihood metrics), and have ~80% confidence that the pairwise ordering will hold at 1B scale for the OLMES benchmarks. The caveats from Figure 2 matter operationally: if the team cares primarily about HellaSwag, small-scale ablations are much less reliable (~55–60% at 150M), and they should either invest in larger small experiments (300M–530M) or use CORRECT PROB as a proxy (which raises HellaSwag decision accuracy to ~0.7–0.75 in the same compute range, per Figure 4). If they care about BoolQ or SocialIQA, small-scale experiments provide essentially no signal, and they should either de-prioritize those benchmarks in their decision-making or accept that their data choice for those tasks is essentially random. The paper thus provides not just a recommendation but a task-specific reliability calibration: it tells practitioners which of their evaluation benchmarks they can trust small experiments for, and which require larger-scale validation or proxy-metric substitution.

Code-focused data curation becomes feasible with small experiments. The finding that CORRECT PROB makes HumanEval and MBPP >80% predictable at model sizes as small as 4M–60M parameters (Figure 6) is directly actionable for teams building code-generation models. Previously, evaluating whether a pretraining data intervention improves code generation required training models large enough to achieve non-zero pass@1—typically hundreds of millions of parameters or more. The proxy-metric approach means that a team can test dozens of data recipes at the 60M-parameter scale, compute CORRECT PROB on HumanEval continuations (the per-token probability the model assigns to the correct code completion), and rank recipes by this continuous metric, with ~80% confidence that the ranking will predict 1B-scale discrete pass@1 accuracy. The cost savings are approximately 2–3 orders of magnitude compared to running the same ablations at a scale where discrete accuracy provides signal. The simultaneous negative result for math (GSM8K, Minerva remain near chance) means this approach does not currently transfer to math-focused data curation, and teams in that domain should either invest in larger small experiments or develop domain-specific proxy metrics.

Evaluation benchmark design can be optimized for small-scale predictability. The diagnostic framework in Figure 5—decomposing decision accuracy into spread and noise—provides concrete guidance for benchmark designers. Current benchmarks are typically designed to discriminate among large, deployment-scale models on the basis of absolute accuracy. But if benchmarks are to be useful for guiding small-scale data decisions (which they increasingly are, given the cost of large-scale experimentation), they should also be designed to maximize the spread-to-noise ratio at the small scales where ablations are run. Concretely: a benchmark with many items reduces noise (through averaging), benefiting predictability. A benchmark with answer options that surface latent model knowledge through likelihoods (even when the argmax is wrong) increases spread under continuous metrics. The paper's finding that character-level length normalization "is empirically optimal for most of the tasks" (Section 3.3) suggests a specific design choice: normalization schemes that are robust to tokenizer artifacts produce cleaner proxy signals. A team designing a new benchmark intended for data-curation use could evaluate candidate item sets not just by their large-model discriminability but by their decision accuracy on a suite like DATADECIDE, optimizing directly for the use case.

Self-improvement and data generation pipelines can use proxy metrics for efficient filtering. When using LLMs to generate training data for themselves (rejection sampling, STaR-style self-improvement, or data synthesis), a recurring challenge is selecting which generated examples to keep. The paper's proxy-metric framework suggests a computationally cheap filter: for a proposed training example, measure the model's CORRECT PROB or TOTAL PROB on that example (which requires only a forward pass, not generation), and use the continuous score to decide whether to include the example in the training set. If the proxy-metric findings transfer, examples with higher CORRECT PROB under a small evaluator model may be more likely to contribute to downstream task improvement after fine-tuning, even if the evaluator's discrete accuracy on the task is near chance. This is speculative—the paper does not test this application directly—but the finding that continuous likelihoods carry predictive signal at scales where discrete metrics are uninformative directly motivates the experiment.

When to Prefer This Method

The paper does not propose a new method competing against named alternatives—it provides a measurement infrastructure and uses it to evaluate existing prediction approaches. The decision it informs is not "which method to use for prediction" (though the results bear on that) but rather how to configure small-scale experiments when making data decisions, with the following guidance emerging from the results:

Prefer single-scale ranking over multi-scale scaling law fits when the target scale is within ~1–2 orders of magnitude of the experiment scale (e.g., using 150M experiments to predict 1B outcomes), based on Figure 3's finding that no scaling law variant exceeds the single-scale frontier. The paper explicitly recommends "ranking single scale experiments" over the 8 scaling law variants tested for this scale range.
Use CORRECT PROB or TOTAL PROB as the evaluation metric at small scales rather than discrete ACCURACY for tasks where these metrics show flat or elevated early decision accuracy (Figure 4: ARC Challenge, ARC Easy, BoolQ, CommonsenseQA, OpenBookQA, and the code tasks HumanEval and MBPP from Figure 6). For tasks where all metrics track together (MMLU, PIQA), ACCURACY is competitive. The paper's practical recommendation is to compute all metrics and compare, as the optimal choice is task-dependent.
Use character-level length normalization for proxy metrics, as the paper states it "is empirically optimal for most of the tasks that we observe" (Section 3.3).
Do not rely on small-scale experiments for data decisions targeting SocialIQA, WinoGrande, or BoolQ unless using CORRECT PROB and accepting still-limited decision accuracy, as Figure 2 shows these tasks remain near chance for most of the compute range.
For math tasks (GSM8K, Minerva), small-scale experiments are currently unreliable with any metric tested, and decisions affecting math performance require either larger-scale ablations or a different approach entirely (Figure 6).

These are not presented by the paper as a formal decision rule—they are empirical findings that together constitute practical guidance. They are conditional on the scale range (4M–1B), training ratio (5× Chinchilla), recipe diversity (25 web-text variants), and evaluation suite (OLMES + code/math extensions) studied in DATADECIDE.