ArXiv: 2604.01411
🎯 Pitch
When you account for the cost of repeated sampling at inference, the optimal pretraining recipe flips: you should train much smaller models on vastly more data than Chinchilla recommends. These deliberately overtrained models beat Chinchilla-optimal ones by a wide margin across all tasks, and the advantage remarkably survives fine-tuning.
1. Executive Summary
This paper proposes Train-to-Test (T2) scaling laws that jointly optimize model size, training tokens, and the number of inference samples under fixed end-to-end compute budgets, closing the loop between pretraining and test-time scaling. Using a suite of over 100 checkpoints from Porian et al. (2024) extended into the overtrained regime—spanning 5M to 901M parameters trained on RefinedWeb and evaluated across eight downstream tasks including LAMBADA, ARC-Easy, SciQ, and synthetic reasoning benchmarks—the authors develop two complementary modeling approaches: one that extends Chinchilla scaling with a power-law term in the number of samples to model the negative log pass@k (Approach 1), and one that models pass@k accuracy directly by composing a Chinchilla loss estimate with a Beta distribution over per-question success probabilities (Approach 2). Both approaches converge on a single finding: when inference cost is accounted for via repeated sampling, optimal pretraining decisions shift radically into the overtraining regime, recommending models that are substantially smaller and trained on far more tokens than standard Chinchilla prescriptions—for example, at a pretraining budget of 2.56 × 10¹⁹ FLOPs and an inference budget of 2 × 10⁹ FLOPs, overtrained models as small as 37M parameters outperform Chinchilla-optimal models up to 901M parameters across all eight tasks, with gains persisting after supervised fine-tuning, establishing that test-time scaling makes overtraining compute-optimal only when the per-sample quality improvement from additional training outweighs the per-token cost savings of smaller models under repeated sampling.
2. Context and Motivation
The Core Gap: Pretraining and Test-Time Scaling Are Optimized in Isolation, Yet Fundamentally Coupled
The fundamental problem this paper addresses is that the two most important scaling decisions in the LLM lifecycle—how to train the model and how to sample from it at deployment—are currently made independently of each other, despite being deeply interdependent. This is not a minor oversight; it is a structural gap in how the field approaches scaling, with significant practical consequences.
To understand why this gap matters, consider the following scenario. A team decides to train a model following Chinchilla scaling laws (Hoffmann et al., 2022): they allocate their pretraining compute budget optimally between model size and training tokens, converging on a model with roughly 20 tokens of training data per parameter. At deployment, they plan to use repeated sampling—drawing many independent answers and selecting the best one via majority voting or a verifier—a strategy that has become standard practice for reasoning-heavy applications (Jaech et al., 2024; Guo et al., 2025). But here is the tension: the Chinchilla recipe was derived under the assumption that the model is queried exactly once. It does not account for the fact that smaller models cost less per sample, meaning that under a fixed inference budget, a smaller model can afford more samples than a larger one. If repeated sampling provides compounding returns—each additional sample gives another chance to produce the correct answer—then the optimal pretraining recipe under a joint train-and-test budget might look very different from Chinchilla's prescription. Yet no existing scaling law captures this tradeoff.
The paper crystallizes this gap with a concrete question:
"Should parameter and token counts change if you know how your model will be used at test time?"
The answer, as the paper demonstrates, is a resounding yes. But before this work, the field lacked both the formalism and the empirical evidence to answer it.
Why This Gap Matters: Three Practical Drivers
The paper grounds its motivation in three converging trends that make the pretraining-inference disconnect increasingly costly and theoretically unsatisfactory.
1. The rise of test-time scaling as a deployment paradigm. In the past two years, the dominant deployment strategy for LLMs on reasoning tasks has shifted from single-pass generation to repeated sampling with verification (Snell et al., 2024; Brown et al., 2025; Orlanski et al., 2025). Models like OpenAI's o1 (Jaech et al., 2024) and DeepSeek-R1 (Guo et al., 2025) are designed to be sampled from hundreds or thousands of times per query. This is not a niche use case—it is how frontier reasoning models are deployed. Yet the models being sampled from were trained according to scaling laws that assume single-pass evaluation. The disconnect means that billions of dollars in training compute are being allocated under assumptions that do not match how the models will actually be used.
The paper makes this distinction explicit: models that will be "sampled from hundreds or thousands of times" for frontier research problems "should be trained differently from chat models that instantly answer everyday questions." The training recipe should depend on the inference strategy, but currently it does not.
2. The empirical observation that smaller models with more samples can outperform larger ones. Several recent works have shown a striking phenomenon: when you equalize total inference compute, a smaller model with many repeated samples can match or exceed the performance of a larger model with few samples (Snell et al., 2024; Brown et al., 2025). For instance, Brown et al. (2025) demonstrated that smaller models given more test-time compute could outperform larger models on coding benchmarks. This suggests that inference compute and model size are partially substitutable—a finding with enormous economic implications, since smaller models are cheaper to serve per token.
However, these studies treat pretrained models as fixed givens. They take a model that was already trained at some (N, D) configuration and ask: "how much test-time compute do I need to match a larger model?" The crucial follow-up question—which this paper is the first to systematically ask—is: if you know you will be doing repeated sampling at deployment, should you have trained the smaller model differently in the first place? Perhaps the optimal small model for repeated sampling is not the Chinchilla-optimal small model, but one that has seen far more training data, trading off some per-sample speed for higher per-sample quality in a regime where the multiplicative effect of sampling amplifies small per-sample improvements.
3. The deliberate overtraining trend in practice, lacking a principled framework. The paper points out that major model releases already routinely deviate from Chinchilla's 20 tokens-per-parameter recommendation, but for reasons of single-pass inference efficiency rather than test-time scaling:
- Llama-2-7B (Touvron et al., 2023): trained on 2 trillion tokens, roughly 290× the Chinchilla-optimal ratio for a 7B model.
- Gemma-7B (Team et al., 2024): trained on 6 trillion tokens, roughly 857× the recommendation.
- Gemma 2-9B (Team et al., 2024): trained on 8 trillion tokens, roughly 889×.
- OLMo (Groeneveld et al., 2024): follows a similar overtrained philosophy.
These models are overtrained to reduce per-query inference cost: a smaller, heavily-trained model is cheaper to serve than a larger, Chinchilla-optimal model of equivalent single-pass quality. But this motivation only considers single-pass serving. The paper's insight is that overtraining may be even more valuable—and should be quantitatively prescribed—when repeated sampling is factored in, because the compounding effect of multiple samples amplifies the per-sample quality gains that overtraining provides. The field currently lacks a principled framework for determining how much to overtrain given a joint train-and-test budget. Practitioners are essentially guessing—the 20:1 rule of thumb is widely cited but increasingly ignored, and the decisions about how far to depart from it are made without a systematic optimization framework.
Where Prior Approaches Fall Short
The paper identifies specific limitations in existing work along three axes: pretraining scaling laws, test-time scaling laws, and works that make partial attempts at unifying them.
Pretraining scaling laws ignore inference strategy. Scaling laws from Kaplan et al. (2020) and Hoffmann et al. (2022) provide principled recipes for allocating pretraining compute between model size and training tokens. The Chinchilla formulation models the pretraining loss as:
and finds that compute-optimal training allocates roughly equal scaling to N and D (with the famous ~20 tokens per parameter ratio). However, this optimization is performed under a single constraint: (the approximate FLOPs for training). There is no term for inference cost, no variable for the number of samples drawn at deployment, and no mechanism for the multiplicative interaction between per-sample quality and repeated sampling. The framework implicitly assumes the model is queried exactly once, making it blind to the tradeoffs that arise when inference compute is a significant fraction of the total lifecycle cost.
Recent extensions have broadened the scope of scaling law modeling—incorporating data quality (Goyal et al., 2024), downstream task accuracy (Isik et al., 2024; Bhagia et al., 2024), knowledge vs. reasoning decomposition (Roberts et al., 2025), and multimodal settings (Shukor et al., 2025)—but none address the interaction with test-time sampling strategies. They all share the same implicit assumption: pretraining is optimized for single-pass evaluation.
Test-time scaling laws treat pretrained models as fixed. On the other side, a growing body of work studies how to optimally allocate inference compute, treating the pretrained model as a black box. Snell et al. (2024) showed that test-time compute can be more effective than scaling model parameters under a fixed inference budget. Brown et al. (2025) formalized the scaling behavior of repeated sampling (what they call "large language monkeys"), demonstrating that pass@k follows predictable power-law scaling with k. Schaeffer et al. (2025) developed scaling laws that predict pass@k from pretraining compute. However, these works are fundamentally descriptive rather than prescriptive: they predict what performance will be for a given model, not what model should have been trained for a given budget. As the paper puts it:
"Schaeffer et al. (2026) develop scaling laws that predict pass@k from pretraining compute, but treat this as forecasting rather than an optimization problem—they predict what performance will be for a given model, not what model should be trained for a given budget."
The distinction is crucial. Forecasting tells you "if you train model X, you will get performance Y with k samples." But to make optimal resource allocation decisions, you need the inverse: "given a total budget, what model should I train and how many samples should I draw to maximize performance?" This requires a joint optimization framework that no prior test-time scaling work provides.
Partial attempts at unification miss the key mechanism. Sardana et al. (2023) took an important step toward deployment-aware scaling by extending Chinchilla to account for inference serving volume. Their framework folds the total inference cost (number of queries × cost per query) into the compute budget, recognizing that a model that is served many times should potentially be smaller to reduce per-query cost. However, their analysis is limited to single-pass queries—they consider the aggregate volume of inference but not the multiplicative cost and performance gains from repeated sampling. This misses the crucial nonlinearity: under repeated sampling, the benefit of a smaller model is not just that it is cheaper per sample, but that the cost savings can be reinvested into more samples, and the relationship between number of samples and success probability (pass@k) is highly nonlinear. A model that is 2× cheaper per sample can afford 2× more samples under the same budget, but pass@k typically scales as a power law with diminishing returns—the interaction between these two effects determines the optimal tradeoff, and Sardana et al. (2023) does not capture it.
No joint optimization framework exists. The paper's central claim is that "no existing work jointly optimizes model size, training duration, and the number of inference samples under a single compute budget." This is the gap they fill. The optimization problem is:
where is the loss (or accuracy) as a function of model size, training tokens, and number of samples, and the constraints capture both training FLOPs () and inference FLOPs (). No prior work has formulated or solved this joint problem.
How This Paper Positions Itself
The paper positions itself as closing the loop between pretraining and test-time scaling—the conceptual contribution is not a new scaling law per se but a unified optimization perspective that treats training and inference as coupled decisions. The paper's framing is explicitly prescriptive, not descriptive: they do not merely observe that overtrained models perform well under repeated sampling (though they do observe this), they develop a framework for determining how much to overtrain given a specific inference budget.
The paper also explicitly positions itself against the status quo where pretraining decisions are made using Chinchilla scaling and test-time decisions are made separately using pass@k estimation (Figure 1). The key visual in the paper contrasts three bars: Chinchilla (pretraining-only optimization), test-time scaling applied to Chinchilla models (the status quo), and T2 scaling (joint optimization). T2 wins because it recognizes that the model should be trained for its intended inference regime.
A subtle but important positioning choice: the paper does not argue that Chinchilla is wrong as a pretraining scaling law. Rather, it argues that Chinchilla is incomplete—it correctly optimizes the pretraining loss under a pretraining compute constraint, but this is the wrong objective when inference cost is part of the total budget and when repeated sampling provides nonlinear performance amplification. The paper's ambition is to modernize pretraining scaling laws by incorporating the inference strategy that modern deployments actually use.
The paper also positions itself against the prevailing "overtraining is for inference efficiency" narrative by providing a different—and potentially stronger—justification: overtraining is not just about making single queries cheaper, but about making the compounding effects of repeated sampling more effective. This reframes overtraining from a cost-saving measure to a performance-maximizing strategy under test-time scaling budgets.
Finally, the paper acknowledges its scope limitations: it focuses on parallel repeated sampling (the simplest and most widely deployed form of test-time scaling), operates at relatively small scale (models up to ~1B parameters to enable comprehensive empirical validation), and evaluates primarily on downstream task accuracy rather than open-ended generation. These choices are deliberate—they enable the systematic, controlled experiments needed to establish the joint scaling phenomena before scaling to larger regimes.
3. Technical Approach
3.1 Reader Orientation
The system being built is a joint optimization framework that tells you what model size, how many training tokens, and how many inference samples to use to maximize downstream task performance under a single fixed budget that covers both training and deployment. The core problem it solves is that standard pretraining scaling laws (like Chinchilla) ignore the inference strategy, while test-time scaling laws treat the model as fixed—T2 closes this loop by modeling performance as a function of model size $N$, training tokens $D$, and number of repeated samples $k$, then optimizing all three simultaneously under constraints on training FLOPs ($6ND$) and inference FLOPs ($2Nk$).
3.2 Big-Picture Architecture
The system has four major components connected in a pipeline that flows from raw empirical data to optimal deployment prescriptions:
-
Checkpoint population — a grid of over 100 pretrained language models spanning 5M to 901M parameters trained on 50M to 120B tokens of RefinedWeb data, including both Chinchilla-optimal checkpoints from Porian et al. (2024) and additional overtrained checkpoints trained for this work. Each checkpoint provides empirical measurements of negative log-likelihood (NLL) and per-question success probabilities across eight downstream tasks.
-
Scaling law fitting engine — two complementary parametric models (Approach 1 and Approach 2) that are fit to the checkpoint population. Approach 1 extends the Chinchilla loss model with a power-law term in
$k$to predict the negative log pass@k. Approach 2 models the per-task pass@k accuracy directly by composing a Chinchilla loss estimate with a Beta distribution over per-question success probabilities. Both models are fit via L-BFGS-B optimization with hundreds of random restarts. -
Inference cost correction mechanism — a procedure that substitutes the relationship
$k = C_{\text{inf}} / (2N)$into the fitted models, encoding the constraint that smaller models can afford more repeated samples under the same inference budget. This transforms the models from functions of$(N, D, k)$to functions of$(N, D, C_{\text{inf}})$, enabling joint optimization over pretraining and inference decisions. -
Compute-optimal frontier extractor — for a given pretraining budget
$C_{\text{train}}$and inference budget$C_{\text{inf}}$, this component searches over$(N, D)$pairs satisfying$6ND \leq C_{\text{train}}$to find the configuration that minimizes predicted NLL (Approach 1) or maximizes predicted pass@k accuracy (Approach 2). The output is a set of isoFLOP curves and optimal$(N^*, D^*)$recommendations that can be compared directly to Chinchilla prescriptions.
Information flows as follows: pretrained checkpoints → empirical NLL and accuracy measurements → parametric model fitting (Chinchilla loss fit + Beta regression or additive k-term fit) → inference budget substitution ($k = C_{\text{inf}}/(2N)$) → constrained optimization over $(N, D)$ → optimal pretraining recipes and isoFLOP visualizations.
3.3 Roadmap for the Deep Dive
- First, the joint optimization formulation (Equations 1 and 2), because it defines the mathematical problem that both T2 approaches solve and establishes the constraints that couple pretraining and inference decisions.
- Second, Approach 1 (the NLL-based model), because it is the simpler and more direct extension of Chinchilla scaling—it adds a single power-law term in
$k$to the familiar Chinchilla loss function, making it the natural entry point for understanding how test-time scaling modifies the optimization landscape. - Third, Approach 2 (the accuracy-based model), because it addresses the practical need for pass@k forecasts by composing a loss-to-accuracy mapping with a Beta distribution model—this is more complex but produces more interpretable predictions.
- Fourth, the Beta regression machinery that underpins Approach 2, since it is the novel technical contribution for modeling per-question difficulty distributions rather than just mean accuracy.
- Fifth, the inference cost correction (
$k = C_{\text{inf}}/(2N)$) applied to both approaches, because this substitution is what creates the coupling between pretraining and inference that drives the overtraining result. - Sixth, the fitting procedures and optimization details, since the practical reliability of the T2 recommendations depends on how the models are parameterized and optimized.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a scaling laws paper whose core idea is that pretraining decisions (model size $N$ and training tokens $D$) and inference decisions (number of samples $k$) must be optimized jointly under a single compute budget, and that doing so reveals that standard Chinchilla-optimal models are substantially undertrained for deployment regimes that use repeated sampling.
The Joint Optimization Formulation
The paper defines the train-to-test scaling problem as a constrained optimization over three variables: model size $N$ (number of parameters), training dataset size $D$ (number of tokens), and number of inference samples $k$ (independent draws from the model at deployment). There are two separate budget constraints—one for training and one for inference—and the objective can be formulated in terms of either the negative log-likelihood (NLL) or the downstream task accuracy.
NLL formulation (Equation 1):
where $L(N, D, k)$ is the aggregated negative log-likelihood (or a surrogate derived from it) over the evaluation distribution as a function of model size $N$, training tokens $D$, and number of repeated samples $k$. The constraint $6ND \leq C_{\text{train}}$ approximates the total training FLOPs (following the standard 6ND approximation from Kaplan et al., 2020, where each token processed during training requires roughly 6N floating-point operations for the forward and backward passes). The constraint $2Nk \leq C_{\text{inf}}$ approximates the total inference FLOPs, where 2N is the approximate FLOPs for a single forward pass per token (only the forward pass, no backward pass) and $k$ is the number of independent samples drawn.
What this formulation computes: for a given total training budget $C_{\text{train}}$ and total inference budget $C_{\text{inf}}$, find the model size, training duration, and sampling strategy that achieves the lowest possible loss. The training budget limits how large a model you can train and for how many tokens; the inference budget limits how many samples you can draw, with larger models consuming more of that budget per sample.
Why this form: the separation into two distinct constraints—rather than a single combined budget—reflects the practical reality that training and inference compute are often allocated from different pools (training is a one-time capital expenditure, inference is an ongoing operational cost per query). The joint optimization would be trivialized if they were lumped together, because then you could simply trade off training and inference FLOPs at a fixed exchange rate, which ignores the time-value and operational separation of these resources. The use of the 6ND and 2N approximations follows the standard FLOP counting conventions from the scaling laws literature, enabling direct comparison with prior work like Chinchilla (Hoffmann et al., 2022) and Sardana et al. (2023).
Accuracy formulation (Equation 2):
where $\text{Acc}(N, D, k)$ is the expected pass@k accuracy on a downstream task—the probability that at least one of $k$ independent samples from the model produces the correct answer, averaged over the task distribution.
What this formulation computes: exactly the same constrained optimization as the NLL version, but with the objective flipped to maximize accuracy rather than minimize loss. The constraints are identical.
Why this form: practitioners care about pass@k accuracy as an interpretable metric—"if I give this model k attempts, what is the probability it solves the problem?" The NLL is a smoother optimization surface (being continuous and well-behaved under standard scaling laws) but is less directly meaningful for deployment decisions. By providing both formulations and showing they agree, the paper establishes robustness: the overtraining finding is not an artifact of choosing one particular objective function.
A critical subtlety: $L(N, D, k)$ and $\text{Acc}(N, D, k)$ are not directly measured for arbitrary $k$ from the raw checkpoint evaluations—they must be modeled as parametric functions fit to empirical data. The modeling choices for each are the core technical contributions of Approaches 1 and 2.
Approach 1: T2 as a Parametric Model of the Task Loss
Approach 1 models the negative log-likelihood under repeated sampling as an additive extension of the standard Chinchilla scaling law. The key idea is that the effect of drawing $k$ independent samples can be captured by adding a power-law term in $k$ to the single-pass Chinchilla loss.
Step 1: Define a loss-compatible objective for repeated sampling. The challenge is that the standard NLL is defined for a single model output—it is $\mathbb{E}_{(x,y) \sim \mathcal{D}}[-\log p(y|x)]$, the expected negative log-probability the model assigns to the correct answer. But under repeated sampling with pass@k, the quantity of interest is whether any of $k$ samples is correct, not the per-sample probability. To bridge this gap, the paper defines a loss for repeated sampling using the negative log of the pass@k probability:
where $\mathcal{D}_{\text{task}}$ is the distribution over evaluation examples (questions) in the downstream task, $p_i$ is the per-sample probability that the model correctly answers question $i$ on a single attempt, and $\text{pass@k}_i = 1 - (1 - p_i)^k$ is the probability that at least one of $k$ independent attempts succeeds.
What this computes: for each question $i$, it takes the model's per-sample success probability $p_i$, computes the pass@k probability (which is the complement of the probability that all $k$ attempts fail), takes the negative log, and averages over questions. This transforms the pass@k metric—which is an accuracy, not a loss—into a quantity that behaves like a log-loss and can be modeled with power-law scaling.
Why this form: the negative log pass@k serves as an upper-bound surrogate for the quantity we actually care about. By Jensen's inequality, $\mathbb{E}[-\log X] \geq -\log \mathbb{E}[X]$, so minimizing the expected negative log pass@k also minimizes (an upper bound on) the negative log expected pass@k. The paper notes that prior work has found the negative log pass@k itself exhibits power-law scaling in $k$ under the assumption that the per-question probabilities $p_i$ follow a Beta distribution across the task—an empirical regularity that has been validated by Brown et al. (2025) and Schaeffer et al. (2025). This makes it compatible with the power-law form of Chinchilla scaling.
Step 2: Model the negative log pass@k as a Chinchilla extension. The parametric model is:
where $\hat{L}(N, D) = E + A/N^\alpha + B/D^\beta$ is the standard Chinchilla scaling law for single-pass NLL, $E$ is the irreducible loss floor (the minimum achievable loss on the data distribution even with infinite model size and training data), $A$ and $\alpha$ control how loss decreases with model size, $B$ and $\beta$ control how loss decreases with training tokens, and $G$ and $\gamma$ are new parameters that control how the loss decreases with the number of repeated samples.
What this model computes: given values for $N$ (parameters), $D$ (training tokens), and $k$ (inference samples), it predicts the expected negative log pass@k on the task. The first three terms come from standard Chinchilla scaling and capture how single-pass quality depends on model capacity and training data. The fourth term $G/k^\gamma$ captures how repeated sampling reduces the effective loss—as $k$ increases, the probability that at least one sample is correct increases, so the negative log pass@k decreases (improves), following a power law with exponent $\gamma > 0$ and coefficient $G > 0$.
Why this form: it has several desirable properties that make it a natural extension of Chinchilla scaling. First, when $k = 1$, the $G/k^\gamma$ term becomes $G$, which gets absorbed into the constant $E$: $\hat{L}(N, D, 1) = (E + G) + A/N^\alpha + B/D^\beta$, recovering the standard Chinchilla law with a shifted irreducible loss $E' = E + G$. This means the model is consistent with prior work when evaluated in the single-pass regime. Second, as $N, D, k \to \infty$ simultaneously, all three power-law terms vanish, and the loss approaches the irreducible floor $E$—the model maintains the asymptotic saturation property of standard scaling laws. Third, the additive form assumes that the benefits of scaling $N$, $D$, and $k$ are independent and combine additively in log-loss space, which is the simplest possible interaction and has been empirically supported by prior work showing that pass@k power laws are well-behaved under varying model scales.
What is NOT modeled: this additive form assumes no interaction between $N$/$D$ and $k$—that is, the benefit of additional samples does not depend on model size or training duration. This is a modeling choice made for simplicity; the paper does not test whether interaction terms would improve fit, and the fact that Approach 2 (which does capture some interaction via the Beta distribution parameterization) produces similar results suggests the additive assumption is reasonable in the studied regime.
Step 3: Fit the seven parameters to empirical data. The model has seven free parameters: $\log A$, $\log B$, $\log E$, $\alpha$, $\beta$, $\log G$, and $\gamma$ (the logs are fit to ensure positivity of the coefficients). Fitting is done by minimizing the sum of squared errors between the model's predictions and the empirically measured negative log pass@k values across all checkpoints in the population and across all values of $k$ sampled. The optimization uses the L-BFGS-B algorithm with 500 random restarts, each with up to 5,000 iterations and a convergence tolerance of $10^{-15}$, and the run with the lowest objective value is selected. The empirical negative log pass@k values are computed from the per-question pass@k probabilities, which are in turn estimated from the per-question single-pass accuracies $p_i$ (measured by evaluating each checkpoint on each task) and the pass@k formula $1 - (1 - p_i)^k$.
Approach 2: T2 as a Parametric Model of the Task Accuracy
Approach 2 models the pass@k accuracy directly as a function of $N$, $D$, and $k$. The key challenge is that pass@k is a nonlinear function of the per-question success probabilities, and simply plugging the mean accuracy into the pass@k formula overestimates the true expected pass@k due to Jensen's inequality (the pass@k function $1 - (1 - p)^k$ is concave in $p$). The solution is to model the full distribution of per-question accuracies using a Beta distribution parameterized by the Chinchilla loss.
Step 1: Recognize and address the concavity problem. A naive model would estimate the mean single-pass accuracy $\mathbb{E}[\text{Acc}(N, D)]$ from the Chinchilla loss (via a sigmoid mapping), then compute pass@k as:
where $\sigma_\theta$ is a parameterized sigmoid function mapping loss to accuracy. However, this overestimates the true expected pass@k because of the concavity inequality:
What this inequality means: the left side applies the pass@k transformation to the mean accuracy; the right side is the true expected pass@k, which averages the pass@k transformation over the distribution of per-question accuracies. Because pass@k is a concave function of the per-question accuracy (diminishing returns: going from 0% to 10% per-sample accuracy increases pass@k more than going from 90% to 100%), applying it to the mean ignores the spread of difficulties and overestimates performance. Intuitively, if half your questions have 0% accuracy and half have 100% accuracy, the true pass@k is much lower than what you would compute from the 50% mean accuracy, because the easy questions saturate quickly with few samples while the hard questions never succeed.
Why this matters for scaling laws: the degree of overestimation depends on the variance of per-question accuracies, which itself varies with model size and training tokens. A scaling law that ignores this distributional effect would systematically overestimate the benefits of test-time scaling, potentially recommending models that are too small (because it overestimates how much repeated sampling compensates for low per-sample quality).
Step 2: Model the per-question accuracy distribution using a Beta distribution. The paper assumes that across questions in a task, the per-question single-pass accuracies $\text{Acc}(N, D)$ follow a Beta distribution with parameters that depend on $N$ and $D$:
where $a_{N,D} > 0$ and $b_{N,D} > 0$ are the Beta distribution's shape parameters, which together control the mean and variance of the distribution. The Beta distribution is a natural choice for modeling probabilities because it is supported on $[0, 1]$ and its two parameters provide flexible control over both location (mean) and spread (concentration).
Why the Beta distribution specifically: prior work by Kazdan et al. (2025) found that Beta distributions provide good fits to the per-question accuracy distributions of language models, and the Beta distribution has convenient mathematical properties for computing pass@k analytically (as shown below). The alternative would be to use a nonparametric model of the difficulty distribution, but that would not extrapolate to unseen $(N, D)$ configurations—the whole point of a scaling law is parametric extrapolation.
Step 3: Relate the Beta parameters to the Chinchilla loss via Beta regression. The Beta distribution is parameterized in terms of its mean $\mu \in (0, 1)$ and sample size (concentration) $\nu \in (0, \infty)$, where:
The mean $\mu$ is the expected per-question accuracy, and $\nu$ controls the spread—larger $\nu$ means lower variance (more concentrated around the mean). These are in turn modeled as functions of the Chinchilla loss estimate $\hat{L}(N, D) = E + A/N^\alpha + B/D^\beta$:
where $\theta_0, \theta_1, \theta_2, \theta_3, \theta_4$ are five free parameters to be fit, $\hat{L}(N, D)$ is the Chinchilla loss estimate for a given model configuration, and $\sigma_\theta$ is a parameterized sigmoid function (a rescaled and shifted logistic function) that maps the loss to a probability in $(0, 1)$.
What the mean model $\mu_{N,D}$ computes: it maps the Chinchilla loss to an expected per-question accuracy using a sigmoid link function. The parameter $\theta_0$ controls the horizontal shift (the loss value at which accuracy is at the sigmoid's midpoint), $\theta_1$ controls the steepness (how quickly accuracy transitions from low to high as loss decreases), and $\theta_2$ is a scaling factor that sets the maximum achievable accuracy. This form is motivated by the empirical observation that the relationship between NLL and downstream accuracy is well-approximated by a sigmoid (Grattafiori et al., 2024)—models with very high loss have near-zero accuracy, models with very low loss approach some ceiling accuracy $\theta_2 \leq 1$, and the transition is smooth.
What the concentration model $\nu_{N,D}$ computes: it models how the spread of per-question accuracies changes with model quality. The log-link $\exp(\cdot)$ ensures positivity of $\nu$. The linear predictor $\theta_3 + \theta_4 \cdot \hat{L}(N, D)$ captures whether higher-quality models (lower loss) tend to have more concentrated (less variable) or more dispersed per-question accuracies. If $\theta_4 < 0$, then lower loss (better models) implies larger $\nu$ (less variance)—intuitively, as models improve uniformly, the gap between easy and hard questions might narrow. If $\theta_4 > 0$, the opposite. The data determines the sign.
Why this Beta regression setup: standard approaches to accuracy prediction in scaling laws (e.g., Isik et al., 2024) model only the mean accuracy as a function of loss, then apply a pass@k transformation to this mean. This ignores the distributional effects and overestimates pass@k. By explicitly modeling both the mean and the concentration of the accuracy distribution, Approach 2 captures how the shape of the difficulty distribution evolves with model scale, enabling unbiased extrapolation of pass@k. The use of a linear predictor in the loss (for $\mu$ via the sigmoid's argument and for $\nu$ via the log link) is standard in generalized linear models and ensures that the scaling behavior is smooth and extrapolatable.
Step 4: Compute the expected pass@k analytically from the Beta distribution. Given the Beta model for per-question accuracies, the expected pass@k has a closed form:
where $B(x, y) = \Gamma(x)\Gamma(y) / \Gamma(x + y)$ is the Beta function, and $\Gamma$ is the Gamma function (the continuous extension of the factorial).
What this computes: the expected pass@k accuracy, averaged over the distribution of per-question difficulties. The first line defines it as an expectation. The third line uses a known property of the Beta distribution: the k-th moment of $(1 - \text{Acc})$ when $\text{Acc} \sim \text{Beta}(a, b)$ is $B(a, b + k) / B(a, b)$. The fourth line substitutes the parameterization in terms of $\mu$ and $\nu$. The result is a deterministic function of the five Beta regression parameters and the Chinchilla loss estimate.
Why this closed form matters: it makes the scaling law fast to evaluate at arbitrary $(N, D, k)$ configurations—no Monte Carlo sampling over the Beta distribution is needed. This is critical for the constrained optimization step, which must evaluate the model many times while searching over $(N, D)$ space. The ratio of Beta functions is numerically stable and can be computed using standard implementations of the log-Gamma function.
Step 5: Fit the model in two stages. The paper uses a two-stage fitting procedure:
Stage 1: Fit the standard Chinchilla loss model $\hat{L}(N, D) = E + A/N^\alpha + B/D^\beta$ to the empirical NLL values across all checkpoints. This stage uses inverse-variance weighting across isoFLOP groups (giving more weight to measurements with lower variance) and profiles over a grid of 40 candidate $E$ values spaced between $0.01 \cdot \min(\text{NLL})$ and $0.95 \cdot \min(\text{NLL})$. For each candidate $E$, the remaining four parameters ($\log A$, $\log B$, $\alpha$, $\beta$) are optimized via L-BFGS-B with 50+ random restarts. This two-stage design (profiling $E$ then optimizing the rest) is standard in Chinchilla fitting because $E$ is the asymptotic loss floor and is often poorly constrained by finite-scale data—grid search over $E$ avoids getting stuck in local optima.
Stage 2: Fit the Beta regression parameters $\theta_0, \theta_1, \theta_2, \theta_3, \theta_4$. Using the fitted Chinchilla loss model from Stage 1 as a fixed input, these five parameters are optimized to minimize the sum of squared errors between the predicted pass@k accuracy (from the Beta-derived formula) and the empirical pass@k accuracy, computed from the checkpoint evaluations across all tasks, all checkpoints, and multiple values of $k$. Optimization uses L-BFGS-B initialized from a grid of starting values seeded from a simple sigmoid baseline (the naive model without Beta distributional correction). The two-stage approach ensures that the loss model (which is shared across tasks) is fit first using the cleanest available signal (NLL on the pretraining distribution), and only then is the task-specific accuracy mapping fit on top.
Inference Cost Correction
The mechanism that couples pretraining and test-time decisions is the inference cost correction, which is applied identically to both Approach 1 and Approach 2. The key equation is:
where $C_{\text{inf}}$ is the total inference budget (in FLOPs) and $2N$ is the approximate FLOPs per token for a single forward pass through a model with $N$ parameters.
What this equation encodes: under a fixed inference budget, the number of samples you can afford is inversely proportional to model size. A model with half the parameters costs roughly half as many FLOPs per forward pass, so you can draw twice as many samples for the same total inference cost. This is the fundamental tradeoff that Chinchilla scaling laws ignore: they optimize $N$ and $D$ assuming the model is queried once ($k = 1$), but when $k$ is chosen adaptively based on $N$, smaller models get a "sample boost" that can compensate for their lower per-sample quality.
How it is applied to Approach 1: substitute $k = C_{\text{inf}} / (2N)$ into the additive model:
What this transformed function represents: the predicted negative log pass@k for a model of size $N$ trained on $D$ tokens and evaluated with as many repeated samples as the inference budget allows (which is $C_{\text{inf}}/(2N)$ samples). The last term now contains $N^\gamma$ in the numerator, creating a tension: increasing $N$ reduces the single-pass loss (through the $A/N^\alpha$ and $B/D^\beta$ terms, where larger $N$ helps via the $D$ constraint $D \leq C_{\text{train}}/(6N)$) but increases the loss contribution from the sampling term (because larger $N$ means fewer allowed samples at fixed inference budget). This tension did not exist in the standard Chinchilla formulation, where $N$ and $D$ only appeared as denominators improving the loss.
Why this form drives overtraining: for a fixed pretraining budget $C_{\text{train}}$, increasing $D$ requires decreasing $N$ (since $6ND \leq C_{\text{train}}$). In standard Chinchilla, this tradeoff is balanced around the 20:1 D:N ratio. But with the inference correction, decreasing $N$ has an additional benefit: it increases the allowed samples $k$, which reduces the loss through the $G/k^\gamma$ term. This tilts the optimal balance toward smaller $N$ and larger $D$—that is, toward overtraining. The strength of the tilt depends on $C_{\text{inf}}$: larger inference budgets (relative to pretraining) amplify the benefit of small models, pushing the optimum further into the overtrained regime.
How it is applied to Approach 2: the same substitution $k = C_{\text{inf}}/(2N)$ is made in the pass@k formula:
What this transformed function represents: the expected pass@k accuracy when the number of samples is set to the maximum allowed by the inference budget. The Beta function ratio now depends on $N$ through both the Beta parameters (which depend on the loss, which depends on $N$ via the Chinchilla model) and through the $k = C_{\text{inf}}/(2N)$ term in the second argument of the numerator Beta function. The effect of $N$ on accuracy is now more complex than in Approach 1: it enters through the mean accuracy (via $\mu_{N,D}$), the concentration (via $\nu_{N,D}$), and the number of samples (via the explicit $k$ substitution). This richer structure allows Approach 2 to capture effects that Approach 1's additive form might miss—for example, if the benefit of additional samples depends on the per-question accuracy distribution in a way that varies with model scale.
The inference budget $C_{\text{inf}}$ as a hyperparameter: the paper evaluates multiple values of $C_{\text{inf}}$ across experiments:
- In Figure 2 and the isoFLOP profiles (Figures 3, 6, 7),
$C_{\text{inf}}$is set to various levels spanning orders of magnitude, ranging from approximately$2 \times 10^9$FLOPs (roughly the cost of a single forward pass through a 70B Chinchilla model) to$1.4 \times 10^{11}$FLOPs. The specific value$C_{\text{inf}} = 2 \times 10^9$is used for the detailed isoFLOP visualization in Figure 3. - For the checkpoint comparisons in Tables 1 and 2,
$C_{\text{inf}} = 2 \times 10^9$is used alongside$C_{\text{train}} = 2.56 \times 10^{19}$. - The paper does not explore in detail how the optimal overtrained ratio varies continuously with
$C_{\text{inf}}$, though the per-task isoFLOP profiles in Appendix B (Figures 6 and 7) show optimal frontiers at multiple inference budget levels.
Fitting Procedures and Optimization
The paper uses L-BFGS-B for all parameter optimization, with extensive random restarts to avoid local minima. The specific fitting details are in Appendix F, but the main text provides enough information to understand the procedure:
Approach 1 fitting: seven parameters $(\log A, \log B, \log E, \alpha, \beta, \log G, \gamma)$ are fit by minimizing the sum of squared errors between predicted and empirical negative log pass@k values. The empirical values are computed from per-question single-pass accuracies $p_i$ measured on each checkpoint, which are then converted to pass@k via $1 - (1 - p_i)^k$, then to negative log pass@k via $-\log(\cdot)$, and then averaged over questions. Multiple values of $k$ are sampled for each checkpoint to provide the regression targets. L-BFGS-B is run with 500 random restarts, each with up to 5,000 iterations and a tolerance of $10^{-15}$. The run achieving the lowest sum of squared errors is selected.
Approach 2 fitting: a two-stage procedure. Stage 1 fits the Chinchilla loss model (parameters $E, A, B, \alpha, \beta$) to empirical NLL values across checkpoints, using inverse-variance weighting across isoFLOP groups and profiling over 40 candidate $E$ values. Stage 2 fits the Beta regression parameters ($\theta_0$ through $\theta_4$) by minimizing the sum of squared errors between the predicted pass@k (from the Beta-derived formula using the fitted Chinchilla loss as input) and empirical pass@k values. L-BFGS-B is used in both stages with multiple random restarts.
What is NOT jointly optimized: the two-stage fitting for Approach 2 means that the Chinchilla loss parameters are fit once using NLL data and then held fixed during the Beta regression stage. The Beta regression does not backpropagate errors into the Chinchilla parameters. This design choice simplifies optimization (reducing the parameter count at each stage) but means that the loss model is optimized for NLL prediction, not for pass@k prediction. A fully end-to-end approach would fit all parameters jointly to pass@k accuracy, which might yield slightly different loss model parameters that are better tuned for the downstream task. The paper's finding that both approaches agree suggests this decoupling does not introduce significant bias.
The role of the checkpoint grid: the empirical foundation for all fitting is the grid of 106 checkpoints (85 from Porian et al., 2024, plus 21 overtrained checkpoints trained for this work). These span model sizes from 5M to 901M parameters and training FLOPs from $1.25 \times 10^{16}$ to $2.56 \times 10^{19}$. The overtrained checkpoints—trained at tokens-per-parameter ratios far exceeding the standard Chinchilla 20:1—are essential because Approach 1 and Approach 2 are extrapolating into regimes where no standard Chinchilla scaling suite provides data. Without these checkpoints, the models would be making predictions in a purely extrapolative regime with no validation; by including overtrained checkpoints, the paper can measure extrapolation error (as in Figure 4) and confirm that the parametric forms remain well-behaved far from the Chinchilla optimum. The checkpoint grid is visualized in Figure 8 of the paper, with each cell reporting the tokens-per-parameter ratio for that $(N, D)$ configuration.
Design Choices and Their Justifications
Why two approaches instead of one? The paper explicitly tests two modeling strategies that differ in what quantity they model (NLL vs. accuracy) and in their structural assumptions (additive vs. distributional). The goal is robustness through triangulation: if both approaches—despite their different mathematical forms and fitting procedures—converge on the same optimal $(N, D)$ recommendations, then the finding is not an artifact of a particular parametric choice. The paper emphasizes this in the abstract: "Forecasts from T2 are robust over distinct modeling approaches." The agreement between Approach 1 (which adds a simple power-law term) and Approach 2 (which models the full difficulty distribution) is presented as evidence that the overtraining result is robust.
Why model the NLL at all (Approach 1) when practitioners care about accuracy? NLL is a smoother and more predictable quantity than downstream accuracy—it follows clean power laws across many orders of magnitude of compute (Kaplan et al., 2020; Hoffmann et al., 2022), while accuracy on specific benchmarks can exhibit phase transitions, saturation effects, and high variance. By fitting the scaling law in NLL space and only converting to accuracy via the sigmoid link in a second stage (Approach 2) or by working with the negative log pass@k as a loss-like surrogate (Approach 1), the paper inherits the well-characterized scaling behavior of NLL while still producing practically meaningful accuracy forecasts.
Why a Beta distribution specifically for modeling per-question accuracies? The Beta distribution is the conjugate prior for the Bernoulli distribution and the natural exponential family distribution for probabilities on $[0, 1]$. Its two parameters provide independent control over mean and variance, which is exactly what is needed to capture how model improvements affect both the average accuracy and the spread between easy and hard questions. Alternative distributions on $[0, 1]$ (e.g., truncated normal, logit-normal) would either not have the same analytical convenience for computing pass@k moments or would require numerical integration at each evaluation of the scaling law, making the constrained optimization prohibitively expensive.
Why the L-BFGS-B optimizer with many random restarts? The Chinchilla loss function is known to have a challenging optimization landscape for parameter fitting—the parameters $\alpha$, $\beta$, and $E$ in particular can trade off against each other, creating multiple local minima. L-BFGS-B is a quasi-Newton method that handles bound constraints (needed to enforce positivity of parameters like $\alpha, \beta > 0$) and converges quickly when near a local optimum. The hundreds of random restarts (500 for Approach 1, 50+ for Approach 2 Stage 1) provide coverage of the parameter space, making it likely that at least one restart finds the global optimum. The profiling over $E$ in Approach 2 Stage 1 (40 candidate values) is an additional safeguard: $E$ is the asymptotic loss floor and is particularly hard to estimate from finite-scale data, so grid search reduces the risk that the optimizer gets stuck in a poor local minimum for this critical parameter.
4. Key Insights and Innovations
Innovation 1: Pretraining and Test-Time Decisions Are a Joint Optimization Problem, Not Sequential Ones
The paper's most fundamental conceptual contribution is not a new scaling law but the reframing of pretraining and inference as coupled decisions that must be jointly optimized. This shifts the intellectual landscape from two separate optimization problems—one for training, one for deployment—to a single unified problem where the choice of model size simultaneously determines both training cost and per-sample inference cost, with the latter feeding back into how many samples can be drawn under a fixed inference budget.
What the field assumed before. The dominant paradigm treated pretraining and inference as a sequential pipeline: first, train a model using compute-optimal recipes from Chinchilla (Hoffmann et al., 2022) or follow the Kaplan et al. (2020) scaling laws; second, at deployment, decide how many samples to draw using test-time scaling strategies (Snell et al., 2024; Brown et al., 2025). The two stages were optimized under entirely separate loss functions and budget constraints. Chinchilla minimizes single-pass NLL subject to $6ND \leq C_{\text{train}}$. Test-time scaling maximizes pass@k subject to $k \leq C_{\text{inf}} / (2N)$, with $N$ taken as a fixed given. The structural disconnect means that a Chinchilla-optimal model was never evaluated on whether it is optimal for being sampled from hundreds of times—the optimization criterion didn't include that downstream behavior.
This sequential approach has a concrete consequence that the paper exposes: a Chinchilla-optimal model trained at $C_{\text{train}}$ and then subjected to test-time scaling at $C_{\text{inf}}$ represents a specific point in the joint $(N, D, k)$ space—but it is not the point that would be chosen if $N$, $D$, and $k$ were optimized together from the start. The paper demonstrates this visually in Figure 3: Chinchilla-optimal models, when evaluated under inference-corrected budgets, produce isoFLOP curves that are non-monotonic in $C_{\text{train}}$—adding more pretraining compute can reduce accuracy because the larger model it recommends eats into the inference budget and reduces the allowed $k$. This non-monotonicity is a direct symptom of optimizing $N$ and $D$ without accounting for $k$. T2 scaling, which jointly optimizes all three, produces consistently monotonic improvement.
Why this reframing is fundamental rather than incremental. This is not a small refinement of existing scaling laws with an extra term; it is a different optimization problem entirely. The Chinchilla problem has two variables $(N, D)$ and one constraint. The T2 problem has three variables $(N, D, k)$ and two constraints, with $k$ being both a decision variable and a function of $N$ through the inference budget coupling $k = C_{\text{inf}}/(2N)$. This creates a new tension that doesn't exist in either pretraining-only or inference-only optimization: making the model larger helps single-pass quality but hurts sample count, and the nonlinear pass@k relationship mediates how these two effects combine. No prior framework captures this tension because no prior framework treats $N$, $D$, and $k$ as jointly adjustable.
Sardana et al. (2023) came closest by accounting for inference volume in pretraining decisions, but their framework adds inference cost as an aggregate multiplier on total FLOPs without modeling repeated sampling—it assumes each query uses one sample. The critical mechanism they miss is the $k = C_{\text{inf}}/(2N)$ substitution and its interaction with pass@k's nonlinearity. As a result, their framework can recommend smaller models for cost reasons but cannot capture the fact that smaller models get a compounding benefit from being able to draw more samples—a benefit that might justify even more aggressive overtraining than single-pass inference efficiency alone would suggest.
Significance beyond raw performance. The joint optimization framing provides a new diagnostic lens for understanding why modern deployment practices work. When practitioners overtrain models (Llama-2 at 290× the Chinchilla token-to-parameter ratio, Gemma at 857×), the standard justification is single-pass inference cost: smaller models are cheaper per query. T2 scaling reveals that this justification is incomplete—overtraining is not just about making each query cheaper, but about reinvesting the cost savings into more samples, where the nonlinearity of pass@k amplifies the per-sample quality gains. This transforms overtraining from a cost-cutting measure (to be done as much as the inference budget requires) into a performance-maximizing strategy (to be done at a specific, computable ratio that depends on the joint train-test budget). The field's current practice of overtraining by guesswork—"train as long as you can afford"—can now be replaced with a principled optimization target.
The diagnostic power is visible in the paper's Figures 6 and 7 (Appendix B): the optimal $(N, D)$ frontier shifts continuously with the inference budget $C_{\text{inf}}$, becoming more overtrained as $C_{\text{inf}}$ increases. This is a testable prediction that no prior framework could make: if you know your model will be sampled from heavily (large $C_{\text{inf}}$), you should train a smaller model on more data than if it will be sampled lightly. The field has not previously had a quantitative relationship between inference intensity and optimal pretraining recipe.
Innovation 2: Two Radically Different Modeling Approaches Converge on the Same Answer, Establishing Robustness Through Triangulation
The paper's methodological innovation is the use of two complementary but structurally distinct modeling approaches that predict the same optimal overtraining regime despite modeling fundamentally different quantities (NLL vs. accuracy) and making different parametric assumptions (additive vs. distributional). This is more than a robustness check—it is a triangulation strategy that increases confidence in the qualitative finding (overtraining is optimal) by showing it is not an artifact of any single modeling choice.
What the field did before. Prior scaling laws work typically presents a single parametric form and fits it to data, with robustness evaluated through held-out validation or ablation of functional form choices. When Schaeffer et al. (2025) model pass@k from pretraining compute, they use one approach (their specific power-law parameterization). When Brown et al. (2025) study repeated sampling scaling, they use one parametric form (their Beta-derived scaling law). The field's standard is to propose one model and defend it. The paper departs from this by presenting two models that differ along multiple axes and showing they agree on the key qualitative prediction.
What makes the two approaches genuinely different. Approach 1 models the negative log pass@k as an additive function: $\hat{L}(N, D, k) = E + A/N^\alpha + B/D^\beta + G/k^\gamma$. This assumes the benefits of scaling $N$, $D$, and $k$ are independent and separable in log-loss space. The model has 7 parameters and treats $k$ symmetrically with $N$ and $D$. Approach 2 models the pass@k accuracy directly via a Beta distribution over per-question accuracies, where the Beta parameters are themselves functions of the Chinchilla loss: $\text{Acc}(N,D) \sim \text{Beta}(\mu_{N,D} \nu_{N,D}, (1-\mu_{N,D})\nu_{N,D})$, and pass@k is computed analytically from Beta moments. This model has a different structure: it first maps $(N, D)$ to a loss, then maps the loss to a distribution over per-question accuracies, then computes expected pass@k from that distribution. The model has 5 (Beta regression) + 5 (Chinchilla loss) = 10 parameters, uses a two-stage fitting procedure, and does not assume additive separability between $N$/$D$ and $k$—the effect of $k$ on expected pass@k depends on the Beta parameters, which in turn depend on $N$ and $D$ through the loss.
These are not minor variants. Approach 1 could fail if the additive separability assumption is wrong—if the benefit of additional samples depends on model quality. Approach 2 could fail if the Beta distribution is a poor fit for the per-question accuracy distribution, or if the sigmoid link function mis-specifies the loss-to-accuracy mapping. Their different failure modes make their agreement informative: it suggests the qualitative result is not driven by parametric assumptions about separability or distribution shape.
Why the agreement matters. The paper presents this as what it is: "Despite modeling fundamentally different quantities (NLL vs accuracy), both T2 recommend extreme overtraining" (Section 4.1). The key evidence is in Figure 2, where Approach 1 (blue) and Approach 2 (red) both forecast optimal tokens-per-parameter ratios that are 100-10,000× higher than Chinchilla's 20:1 recommendation across all training FLOPs scales from $10^{17}$ to $10^{25}$. The fact that Approach 2 recommends more aggressive overtraining than Approach 1 (which the paper notes) actually strengthens the argument: it means the distributional modeling in Approach 2, which correctly accounts for Jensen's inequality effects that the naive model ignores, does not pull the optimum back toward Chinchilla—it pushes it further into the overtrained regime. This is a diagnostic finding: the distributional correction, far from weakening the overtraining argument, strengthens it.
This triangulation approach is valuable beyond this paper. It establishes a template for how scaling laws research can build confidence in qualitative predictions: develop structurally distinct models, identify where they differ and where they agree, and use agreement on the sign and direction of effects (if not their precise magnitude) as evidence that the finding is robust to modeling choices. This is more sophisticated than the standard single-model-plus-ablation approach.
Innovation 3: Overtraining Is Not Just for Inference Efficiency—It Is Compute-Optimal Under Joint Budgets, With Quantifiable Ratios
The paper provides the first principled justification for overtraining as an optimal strategy under joint pretraining and test-time compute budgets, rather than as a pragmatic departure from optimality motivated by single-pass inference cost. This transforms overtraining from an "engineering hack" (which is how the Chinchilla-to-overtrained pipeline is often implicitly treated) into a prescribed outcome of a well-defined optimization problem.
What the field believed before. The prevailing narrative around overtraining, which the paper documents in Section 2 (Appendix A.3), is that models like Llama-2, Gemma, and OLMo are overtrained to reduce per-query serving cost. The logic: a smaller model trained on more data can achieve similar single-pass quality to a larger Chinchilla-optimal model, but costs less per forward pass. This is an inference-cost argument: you overtrain to make deployment cheaper, trading off some training efficiency (you spend more training FLOPs than Chinchilla would recommend for that model size) against inference savings. Under this logic, overtraining is a second-best strategy: the "true" optimum under single-pass evaluation is Chinchilla, but practical deployment constraints push you away from it.
The paper reveals that this narrative is incomplete and, for repeated-sampling deployments, potentially wrong. Under a joint budget that includes inference cost via the $k = C_{\text{inf}}/(2N)$ coupling, Chinchilla is not the optimum even if you only care about performance, not cost. The optimal model under the joint objective is overtrained because the coupling between model size and sample count creates a new tradeoff that Chinchilla's optimization doesn't see.
What is distinct about this finding. The paper demonstrates this concretely in Table 1 and Figure 3. At a pretraining budget of $C_{\text{train}} = 2.56 \times 10^{19}$ FLOPs and an inference budget of $C_{\text{inf}} = 2 \times 10^9$ FLOPs, the best overtrained model (37M parameters, trained far beyond the 20:1 ratio) outperforms the best Chinchilla-optimal model (which is 455M-901M parameters, depending on the task) across all eight tasks evaluated. For example, on LAMBADA, the overtrained 37M model achieves 49.90% pass@k vs. 27.30% for the Chinchilla-optimal 455M model—a near-doubling of accuracy. On synthetic simple reasoning, the 37M overtrained model gets 57.90% vs. 18.40% for the 901M Chinchilla model—a 3× improvement.
These are not marginal gains, and the mechanism is not simply that smaller models are cheaper. If it were just about cost, you would expect the Chinchilla-optimal model to outperform when given the same inference budget (since it has higher single-pass quality), with the smaller model winning only on cost-adjusted metrics. But the smaller overtrained model wins on raw pass@k accuracy under the same inference budget. This means the combination of overtraining (which improves per-sample quality) and the higher sample count (enabled by the smaller model size) more than compensates for the larger model's single-pass advantage. The joint optimization finds a configuration that neither pretraining-only nor inference-only optimization would discover.
Why this is a fundamental insight, not an incremental refinement. The paper is not just showing that overtraining helps—practitioners already suspected that. It is providing the first quantitative framework for determining how much to overtrain as a function of the joint budget. Figure 2 shows the optimal tokens-per-parameter ratio as a function of training FLOPs under T2 scaling: it is not a constant (like Chinchilla's 20:1) but varies with both $C_{\text{train}}$ and $C_{\text{inf}}$. The paper's isoFLOP curves (Figures 3, 6, 7) show the optimal frontier shifting continuously as the inference budget changes. This means the answer to "how much should I overtrain?" is not a fixed number but a function of the deployment context—specifically, of how heavily the model will be sampled.
This transforms overtraining from an empirical heuristic into a design parameter that can be optimized. A team deploying a model for a chatbot (low $C_{\text{inf}}$ per query, perhaps $k=1$ or small) should overtrain less than a team deploying a model for automated reasoning where hundreds of samples per query are typical. The paper provides the machinery to compute exactly how much less.
Innovation 4: The Inference Budget Coupling $k = C_{\text{inf}}/(2N)$ Creates a Diagnostic of When Test-Time Scaling Fails
The paper's analysis of the inference budget coupling reveals a diagnostic phenomenon that explains when and why test-time scaling fails to compensate for smaller models: the non-monotonic scaling behavior of Chinchilla-optimal models under inference correction. This is a conceptual contribution that provides clarity on a previously confused empirical landscape.
What was unclear before. The literature contains conflicting signals about whether smaller models with more test-time compute can match larger ones. Snell et al. (2024) and Brown et al. (2025) showed that they can, under certain conditions. But it was unclear under what conditions this works and when it breaks down. The field lacked a framework for predicting a priori whether a given model size and training recipe would benefit from test-time scaling, and how that benefit interacts with pretraining scale.
What the paper shows. The inference-corrected isoFLOP analysis in Figure 3 reveals a specific failure mode: standard Chinchilla-optimal scaling produces non-monotonic accuracy as a function of pretraining compute when evaluated under a fixed inference budget. On the left side of Figure 3 (Approach 1 and Approach 2, bottom row, "Inference-corrected" panels), the Chinchilla optimal frontier (black) initially improves with training FLOPs, but beyond a certain scale, accuracy decreases as more pretraining compute is invested. This means there exists a $C_{\text{train}}$ that is too large—the model becomes so big that the inference budget forces $k$ too low, and the single-pass quality gains from scaling $N$ fail to compensate for the lost samples.
This non-monotonicity is a direct consequence of the coupling $k = C_{\text{inf}}/(2N)$. As $C_{\text{train}}$ increases, Chinchilla recommends larger $N$, which reduces $k$ for fixed $C_{\text{inf}}$. The single-pass accuracy improvement from larger $N$ is logarithmic (power-law in loss, sigmoid in accuracy), while the pass@k benefit from additional samples follows its own power-law. When the marginal accuracy gain from increasing $N$ falls below the marginal accuracy loss from reducing $k$, further pretraining compute becomes counterproductive. This is a ceiling on the utility of pretraining compute under inference constraints that no prior scaling law captured.
Why this is a diagnostic contribution. The paper doesn't just observe this non-monotonicity; it uses it to validate the joint optimization framework. T2 scaling, which jointly optimizes $N$, $D$, and $k$, does not exhibit this non-monotonicity—it produces consistently improving accuracy with $C_{\text{train}}$ (red/blue frontiers in Figure 3, bottom panels). This means the non-monotonicity is a symptom of optimizing $(N, D)$ under the wrong objective (single-pass loss), and it disappears under the correct objective (inference-corrected loss or accuracy). The paper has thereby provided a clear diagnostic: if your scaling curves are non-monotonic under inference correction, your pretraining recipe is not optimized for your deployment strategy.
This insight has practical implications beyond the paper's specific models. It provides a test that practitioners can apply to their own scaling suites: evaluate models across a range of sizes under a fixed inference budget mimicking your deployment, and check whether larger pretraining budgets consistently improve inference-corrected accuracy. If they don't, your pretraining recipe needs adjustment—you are in the regime where T2 scaling would recommend overtraining.
Connection to prior confusion. The non-monotonicity explains why some prior work found that test-time scaling with smaller models matches larger ones, while other work found the opposite. The outcome depends on where in the $(N, D)$ space the comparison is made, relative to the inference budget. Comparisons made at scales below the non-monotonicity threshold will show test-time scaling succeeding; comparisons above it will show it failing. The paper's joint optimization framework provides the vocabulary to describe this threshold precisely.
Innovation 5: The Finding That Overtraining Survives Post-Training—But Is Attenuated—Reveals a Structural Tension
The paper's final experimental contribution is the demonstration that the overtraining prescription from T2 scaling persists after supervised fine-tuning, but is partially attenuated (Section 4.3, Figure 5, Table 2). This is not just an empirical robustness check—it reveals a structural tension between pretraining-for-repeated-sampling and pretraining-for-fine-tuning that has implications for how the full LLM pipeline should be designed.
What the paper shows. After fine-tuning on ARC-Easy, SciQ, and OpenBookQA, the optimal pretraining frontier under T2 scaling still shifts toward smaller, more overtrained models compared to Chinchilla—but the shift is somewhat subdued compared to the base model case (Figure 5). The overtrained checkpoints still outperform Chinchilla-optimal ones after fine-tuning (Table 2): on ARC-Easy, the best overtrained fine-tuned model achieves 5.60% vs. 1.50% for the Chinchilla fine-tuned model; on SciQ, 56.10% vs. 29.00%. But the relative advantage narrows compared to the base model comparisons in Table 1.
Why this is significant beyond the specific numbers. The paper connects this attenuation to a recent finding by Springer et al. (2025): overtrained models are harder to fine-tune. This creates a tension: T2 scaling recommends overtraining because it improves per-sample quality under repeated sampling, but overtraining makes the model less amenable to the fine-tuning that modern deployments invariably include. The paper's result that the T2 optimum survives but is attenuated after fine-tuning means that neither effect dominates entirely—there is a tradeoff, and the joint optimum lies somewhere between the base-model T2 optimum and the Chinchilla optimum.
This is a fundamentally new kind of tradeoff that the field has not previously had to grapple with. Previously, pretraining was optimized for single-pass loss (Chinchilla), and fine-tuning was an independent step whose success was evaluated post-hoc. The paper's finding implies that pretraining optimization should depend on the full pipeline: if you know you will fine-tune and then use repeated sampling, the optimal pretraining recipe is different from either the base-model T2 optimum or the standard Chinchilla optimum. The paper does not solve this joint optimization (it acknowledges this as future work in Section 5: "explicitly model the role of post-training in T2 scaling"), but it provides the first clear evidence that the tension exists and is quantitatively significant.
Why the tension matters for the field. This finding complicates the clean narrative that "you should overtrain if you plan to use repeated sampling." The correct narrative is: "you should overtrain, but less than the base-model T2 optimum if you plan to fine-tune, and the exact amount depends on the fine-tuning regime." This is a more nuanced prescription, and it connects two previously separate research threads—overtraining benefits (this paper, Sardana et al., 2023) and overtraining costs for fine-tuning (Springer et al., 2025)—into a single tradeoff space. Resolving this tradeoff quantitatively is a clear direction for future work, and the paper's framing provides the vocabulary and optimization structure needed to do so.
The attenuation also serves as an important boundary condition on the paper's main finding. The overtraining recommendation is strongest for base models used as-is with repeated sampling. If the deployment pipeline includes fine-tuning, the recommendation is directionally the same (overtrain relative to Chinchilla) but the magnitude is smaller. This prevents over-interpretation of the base-model results and sets realistic expectations for practitioners whose deployment pipelines include fine-tuning.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The paper evaluates on eight downstream tasks: four real benchmarks (LAMBADA OpenAI variant, ARC-Easy, SciQ, OpenBookQA) and four synthetic tasks (simple knowledge recall, multi-step arithmetic reasoning, commonsense causal reasoning, and spatial reasoning), each consisting of 1,000 fill-in-the-blank or short completion questions generated using GPT-5 and Claude Opus 4.6. The real tasks use their standard test splits; the synthetic tasks are described in Appendix E. Evaluation measures the negative log-likelihood (NLL) over the correct answer tokens. The paper does not report training set details for the real tasks beyond what is standard for each benchmark; Appendix E provides example formats for all tasks.
-
Base model(s). All experiments use a suite of transformer language models spanning 5M to 901M parameters, trained on the RefinedWeb dataset (Penedo et al., 2023) with a vocabulary size of 50,432. The checkpoint population includes 85 models from Porian et al. (2024)—which follow standard Chinchilla-optimal scaling with 20 tokens per parameter as the central recommendation—plus 21 additional overtrained checkpoints pretrained by the authors using identical hyperparameters but at substantially higher tokens-per-parameter ratios (shown in orange in Figure 8). The models use the OpenLM framework with the
hparams=base, warmup=short, decay=chinchillaconfiguration from Porian et al. (2024). The authors choose this scale (sub-1B parameters) to enable systematic, controlled experiments across many configurations; they argue the findings are expected to generalize but do not validate at larger scales. -
Metrics. The primary metrics are negative log-likelihood (NLL) on the correct answer tokens (for Approach 1 fitting and for the Chinchilla loss model in Approach 2) and pass@k accuracy (for Approach 2 fitting and for all downstream task evaluations). Pass@k is computed as the probability that at least one of
$k$independent samples produces the correct answer, averaged over all questions in the task. The per-question single-pass accuracy$p_i$is estimated from the model's likelihood of the correct answer tokens for that question. For a given checkpoint, NLL is the per-token negative log-likelihood over answer tokens, and pass@k is computed analytically from$p_i$using$\text{pass@k}_i = 1 - (1 - p_i)^k$. All aggregated results in the main text are macro averages across the eight tasks. -
Baselines. The paper compares against three baselines: standard Chinchilla scaling (Hoffmann et al., 2022), which optimizes
$N$and$D$for single-pass NLL under$C_{\text{train}}$only, producing a 20 tokens-per-parameter recommendation; Chinchilla-optimal checkpoints evaluated with inference correction (the "status quo" where a Chinchilla model is used for test-time scaling but not trained for it); and the empirical Chinchilla-optimal checkpoint within the authors' checkpoint population, selected as the model minimizing single-pass NLL at a given pretraining budget. Within the checkpoint comparisons (Tables 1 and 2), the Chinchilla optimal checkpoint is that which minimizes single-pass NLL at$C_{\text{train}} = 2.56 \times 10^{19}$, yielding model sizes of 455M, 611M, or 901M parameters depending on the task. -
Generation budget / compute accounting. Training compute is measured as
$C_{\text{train}} \approx 6ND$FLOPs, following the standard approximation from Kaplan et al. (2020). Inference compute is measured as$C_{\text{inf}} = 2Nk$FLOPs per query, where$2N$is the approximate FLOPs for a single forward pass per token, and$k$is the number of independent samples drawn. For the inference-corrected analyses in Figures 2, 3, 6, and 7,$k$is set to$C_{\text{inf}} / (2N)$for each model, meaning smaller models are allocated more samples under the same inference budget. The specific values of$C_{\text{inf}}$used are$2 \times 10^9$(Figure 3, Tables 1–2) and ranges spanning from approximately$10^9$to$1.4 \times 10^{11}$(Figures 6–7 in Appendix B). For the FLOPs-matched comparisons extrapolated in Figure 2,$C_{\text{inf}} = 1.4 \times 10^{11}$FLOPs is used. -
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported in the paper. The scaling law parameters are fit to the full checkpoint population with no held-out validation split during fitting (the extrapolation check in Figure 4 uses overtrained checkpoints that were excluded from the initial Chinchilla fitting as a validation set, but this is a single held-out evaluation rather than a systematic cross-validation protocol). The optimization uses L-BFGS-B with hundreds of random restarts (500 for Approach 1, 50+ for Approach 2) to mitigate local minima, and the run with the lowest sum of squared errors is selected. No confidence intervals, bootstrap estimates, or uncertainty quantification are reported for the fitted parameters or the isoFLOP curves. The paper acknowledges this implicitly: the agreement between Approach 1 and Approach 2 serves as the primary robustness check in lieu of formal statistical validation.
Main Quantitative Results
RQ1: Should Pretraining Change If You Know Your Test-Time Scaling Budget?
Headline finding. When inference cost is accounted for via the $k = C_{\text{inf}}/(2N)$ coupling, both T2 approaches forecast optimal pretraining configurations that are dramatically smaller and more overtrained than what standard Chinchilla scaling prescribes. At an inference budget of $C_{\text{inf}} = 1.4 \times 10^{11}$ FLOPs (the approximate cost of one forward pass through the 70B Chinchilla model), the recommended tokens-per-parameter ratio is orders of magnitude higher than the canonical 20:1 across all training FLOPs scales from $10^{17}$ to $10^{25}$ (Figure 2, left panel).
Optimal pretraining allocations (Figure 2). The paper extrapolates its T2 scaling law fits from the empirical checkpoint range (up to ~$10^{19}$ training FLOPs) to $10^{25}$ FLOPs—extending far beyond the empirical coverage—and plots three quantities as functions of training FLOPs:
-
Tokens per parameter (Figure 2, left): Chinchilla predicts a roughly constant ~20 tokens per parameter, with the 20:1 rule of thumb overlaid as a dashed line. Approach 1 (blue) forecasts optimal ratios in the hundreds to low thousands at moderate compute scales, growing to well over
$10^5$at$10^{25}$FLOPs. Approach 2 (red) forecasts even more aggressive overtraining, with optimal ratios roughly an order of magnitude higher than Approach 1 across most of the compute range—reaching approximately$10^6$tokens per parameter at the high end. Both approaches predict the optimal ratio increases with training compute, rather than remaining constant. -
Optimal model size
$N^*$(Figure 2, middle): Chinchilla predicts model sizes growing roughly as$N^* \propto C_{\text{train}}^{0.5}$, reaching ~70B parameters at the hero run scale (annotated on the plot). Both T2 approaches forecast substantially smaller models at every compute level. At$10^{19}$FLOPs, Chinchilla recommends ~500M parameters (consistent with the authors' own scale), while T2 recommends models in the range of$10^7$to$10^8$parameters—roughly 4–40× smaller. At$10^{25}$FLOPs, Chinchilla would recommend ~70B, while T2 approaches recommend ~$10^8$to$10^9$parameters—still sub-billion despite five orders of magnitude more training compute. The optimal model size grows much more slowly under T2 than under Chinchilla because the inference budget coupling penalizes large models by reducing their allowed sample count. -
Optimal training tokens
$D^*$(Figure 2, right): Because T2 recommends smaller models, the optimal training token count at each compute level is actually lower for T2 than for Chinchilla (since$D = C_{\text{train}}/(6N)$, and$N$is smaller). However, the ratio$D/N$is dramatically higher. At$10^{19}$FLOPs, Chinchilla recommends$D^* \approx 2 \times 10^9$tokens; T2 recommends similar or slightly lower absolute token counts but on much smaller models, yielding the extreme tokens-per-parameter ratios.
Why the two approaches differ in magnitude. Approach 2 recommends more aggressive overtraining than Approach 1. The paper attributes this to the distributional correction in Approach 2: by explicitly modeling the spread of per-question difficulties via the Beta distribution, Approach 2 correctly captures the fact that the benefit of additional samples (larger $k$) is amplified when the per-question accuracy distribution has high variance—which happens for smaller, less capable models. Approach 1's additive form assumes the benefit of $k$ is independent of $N$ and $D$, which may underestimate the sampling benefit for small models and thus recommend a less extreme shift toward overtraining. Both agree on the direction, but Approach 2's distributional correction pushes further in that direction.
IsoFLOP analysis (Figure 3). The paper visualizes the pretraining landscape at 12 compute levels spanning $C_{\text{train}} = 1.2 \times 10^{16}$ to $2.6 \times 10^{19}$ FLOPs (the range of their empirical checkpoint grid), with $C_{\text{inf}} = 2 \times 10^9$. Each isoFLOP curve sweeps over model size $N$ while setting $D = C_{\text{train}}/(6N)$ to satisfy the pretraining constraint. Two rows are shown: the top row ("Standard Chinchilla") with $k = 1$ (single-pass), and the bottom row ("Inference-corrected") with $k = C_{\text{inf}}/(2N)$.
-
Top row (single-pass): Both Approach 1 (left) and Approach 2 (right) reproduce the familiar Chinchilla pattern: there is a clear minimum in NLL or maximum in accuracy at an intermediate model size, with both too-small and too-large models underperforming. The Chinchilla optimal frontier (black line connecting the minima/maxima across isoFLOP curves) follows the expected ~20 tokens-per-parameter trajectory.
-
Bottom row (inference-corrected): The isoFLOP curves change shape dramatically. For Approach 1 (left), the NLL curves now tilt upward on the right side—larger models incur higher loss because the inference budget forces
$k$too low. The optimal model size at each compute level shifts sharply leftward (smaller models). For Approach 2 (right), the accuracy curves show a similar pattern: the peaks shift to smaller models. The Chinchilla optimal frontier (black), when evaluated under inference correction, becomes non-monotonic—accuracy initially improves with$C_{\text{train}}$but then declines at higher compute levels because Chinchilla recommends models that are too large and cannot draw enough samples. The T2 optimal frontier (red for Approach 2), in contrast, is consistently monotonic and achieves higher accuracy than the Chinchilla frontier at every compute level by choosing smaller, more overtrained models with more samples.
The non-monotonicity of the Chinchilla frontier under inference correction is a key diagnostic. For example, at a high compute level (~$2.6 \times 10^{19}$ FLOPs), the Chinchilla-optimal model (around 500M–900M parameters, single-pass) achieves lower inference-corrected accuracy than a smaller model at a lower compute level, because the large model gets too few samples. T2 avoids this by jointly optimizing $N$, $D$, and $k$, never entering the regime where model size growth undermines sample count.
RQ2: Does T2 Scaling Extrapolate to Overtrained Checkpoints?
Headline finding. Both T2 approaches extrapolate to overtrained checkpoints with reasonable accuracy, and the empirical best overtrained checkpoint consistently outperforms the empirical Chinchilla-optimal checkpoint under a joint pretraining and inference budget, confirming that the overtraining prescription is not merely an artifact of parametric extrapolation.
Extrapolation accuracy (Figure 4). The paper fits both T2 approaches using only the 85 Chinchilla-optimal checkpoints from Porian et al. (2024)—which stop at tokens-per-parameter ratios well below the T2 optimum—and then evaluates the fitted models on the 21 held-out overtrained checkpoints (which extend to much higher ratios). Figure 4 shows predicted vs. observed scatter plots:
-
Approach 1 (NLL): Relative absolute error of 2.8% across the overtrained checkpoints. The predictions track the observed NLL values closely, with points clustered near the diagonal and no systematic over- or under-prediction visible. This indicates the additive power-law form
$E + A/N^\alpha + B/D^\beta + G/k^\gamma$remains well-behaved far from the Chinchilla optimum. -
Approach 2 (Accuracy): Relative absolute error of 8.4%. The predictions generally overestimate accuracy (points lie slightly above the diagonal), but the trend is preserved. The higher error for Approach 2 is expected because accuracy is a noisier and harder-to-predict quantity than NLL, and the two-stage fitting (Chinchilla loss fit, then Beta regression on top) compounds errors from both stages. The paper presents the 2.8% vs. 8.4% comparison as evidence that Approach 1 extrapolates better but that both approaches are usable.
Empirical checkpoint comparison (Table 1). At a fixed pretraining budget of $C_{\text{train}} = 2.56 \times 10^{19}$ FLOPs and inference budget of $C_{\text{inf}} = 2 \times 10^9$ FLOPs, the paper compares the best overtrained checkpoint (selected from the 21 overtrained models) against the best Chinchilla-optimal checkpoint (selected from the Porian et al. population at the same $C_{\text{train}}$). The results show that the overtrained checkpoint—which is always smaller (37M, 84M, or 149M parameters depending on the task) and more overtrained—outperforms the Chinchilla-optimal checkpoint (ranging from 455M to 901M parameters) across all eight tasks, often by large margins:
- LAMBADA OpenAI: 49.90% (37M overtrained) vs. 27.30% (455M Chinchilla) — a 22.6 percentage point advantage.
- Simple Reasoning (synthetic): 57.90% (37M overtrained) vs. 18.40% (901M Chinchilla) — a 39.5 point advantage, more than 3× higher accuracy.
- Commonsense Causal (synthetic): 8.10% (37M) vs. 1.40% (901M) — roughly 5.8× higher.
- ARC-Easy: 0.14% (149M) vs. 0.07% (611M) — a small absolute difference but a 2× relative improvement.
These results are striking because the overtrained model is up to 24× smaller in parameter count (37M vs. 901M) yet achieves substantially higher pass@k accuracy under the same joint budget. The mechanism: the smaller model gets more samples ($k = C_{\text{inf}}/(2N)$ is larger), and the overtraining ensures those samples are of sufficient quality that the compounding pass@k effect outweighs the larger model's single-pass advantage.
What the extrapolation does and does not validate. The extrapolation check confirms that the parametric forms remain well-behaved into the overtrained regime—the fitted curves do not diverge or produce nonsensical predictions. However, the overtrained checkpoints still lie within the same order of magnitude of model size (all sub-1B) and training FLOPs (up to $2.56 \times 10^{19}$) as the fitting data. The extrapolation is in tokens-per-parameter space (from ~20:1 to much higher ratios) but not in absolute scale. This means the validation supports the T2 recommendation of overtraining within the studied scale but does not test whether the scaling laws continue to hold at, say, 7B parameter models trained on trillions of tokens—the regime of practical interest for modern deployments.
RQ3: Does T2 Scaling Survive Post-Training?
Headline finding. The overtraining prescription from T2 scaling persists after both standard fine-tuning (FT) and supervised fine-tuning (SFT), though the optimal tokens-per-parameter ratio is somewhat subdued compared to base models. The best overtrained fine-tuned checkpoints still outperform the Chinchilla-optimal fine-tuned checkpoints across all three tasks with available training sets.
Post-training procedure. The paper fine-tunes the full checkpoint population (including all 21 overtrained models) on ARC-Easy, SciQ, and OpenBookQA—the three real tasks that have standard training sets. Each model is trained for 6 epochs to convergence using a batch size of 8 and a constant learning rate of $2 \times 10^{-5}$ on 4 NVIDIA A10 GPUs. Two variants are tested: standard fine-tuning (FT), which computes the next-token prediction loss over both the instruction and answer tokens, and supervised fine-tuning (SFT), which computes loss only over the answer tokens (excluding instruction tokens). All models are trained for the same number of steps, rather than scaling training based on FLOPs, since training proceeds to convergence.
Optimal frontier after post-training (Figure 5). The paper refits both T2 approaches to the fine-tuned checkpoints and plots the optimal tokens-per-parameter frontier as a function of training FLOPs, compared to the base-model T2 frontier and the Chinchilla frontier (black). For all three tasks and both fine-tuning variants:
-
The optimal frontier shifts toward smaller, more overtrained models compared to Chinchilla. The T2 frontiers (blue for Approach 1, red for Approach 2) lie substantially above the Chinchilla 20:1 recommendation across all compute scales, confirming that overtraining remains beneficial after fine-tuning.
-
The shift is subdued compared to base-model T2. The fine-tuned frontiers are lower (less overtrained) than the base-model T2 frontiers. The paper attributes this to the finding by Springer et al. (2025) that overtrained models are harder to fine-tune: the fine-tuning process partially erodes the per-sample quality advantage that overtraining provides, reducing (but not eliminating) the benefit of the joint optimization.
-
Approach 1 and Approach 2 remain consistent. Both approaches predict qualitatively similar post-training frontiers, with Approach 2 again recommending more aggressive overtraining than Approach 1 for all three tasks and both fine-tuning variants.
The per-task variation is notable. On ARC-Easy, the optimal overtraining ratio is relatively modest (tokens-per-parameter in the low hundreds at moderate compute), while on SciQ, the overtraining recommendation is much more aggressive (reaching thousands of tokens per parameter). This reflects that the benefit of repeated sampling depends on the task's difficulty distribution: tasks where the base model has more room to improve through additional samples benefit more from the smaller-model-more-samples strategy.
Empirical checkpoint comparison after post-training (Table 2). The paper selects the best overtrained fine-tuned checkpoint and the best Chinchilla-optimal fine-tuned checkpoint at $C_{\text{train}} = 2.56 \times 10^{19}$ and $C_{\text{inf}} = 2 \times 10^9$, and compares their pass@k accuracy:
- ARC-Easy (FT): 5.60% (149M overtrained) vs. 1.50% (901M Chinchilla). For SFT: 8.20% (37M) vs. 3.40% (455M). The overtrained models achieve 2.4×–3.7× higher accuracy.
- SciQ (FT): 56.10% (149M overtrained) vs. 29.00% (901M Chinchilla). For SFT: 66.80% (84M) vs. 57.60% (347M). The overtrained models lead by 9.2–27.1 percentage points.
- OpenBookQA (FT): 2.80% (37M overtrained) vs. 0.45% (901M Chinchilla). For SFT: 2.60% (37M) vs. 0.38% (901M). The overtrained models are 5.8×–6.8× better.
Comparing FT to SFT, SFT consistently achieves higher accuracy than FT for both overtrained and Chinchilla models, and the optimal overtrained model size under SFT tends to be smaller (37M–84M) than under FT (149M), suggesting that SFT's focus on answer tokens amplifies the benefit of overtraining relative to FT, which also trains on instruction tokens.
Performance improvement from fine-tuning itself. Comparing Table 2 to Table 1 (base models), fine-tuning uniformly improves accuracy across all tasks and model types. On SciQ, the best overtrained model goes from 1.20% (base, Table 1) to 56.10% (FT) to 66.80% (SFT)—a massive absolute gain. On ARC-Easy, base model accuracy is near zero (0.14% for the best overtrained base model), but SFT raises it to 8.20%. The Chinchilla models also benefit from fine-tuning but remain behind the overtrained models. This confirms that the overtraining advantage is not an artifact of base-model evaluation and persists when models are fine-tuned to the target task.
Ablation Studies and Robustness Checks
Multiple inference budgets (Figures 6 and 7, Appendix B). The per-task isoFLOP profiles in Appendix B sweep over multiple values of $C_{\text{inf}}$ for each task, showing the optimal frontier as a function of inference budget. As $C_{\text{inf}}$ increases, the optimal models become smaller and more overtrained—the larger the inference budget, the more samples can be drawn, and the stronger the case for a small overtrained model that maximizes per-sample quality. This trend is consistent across all eight tasks for both Approach 1 (Figure 6) and Approach 2 (Figure 7). The paper does not provide a quantitative sensitivity analysis (e.g., how much the optimal $D/N$ ratio changes per doubling of $C_{\text{inf}}$), but the qualitative direction is clear and monotonic.
Approach 1 vs. Approach 2 agreement (Figures 2, 3, 5, 6, 7). The paper presents the consistency between the two modeling approaches as the primary robustness check. Across every experimental setting—base models, fine-tuned models, all tasks, all inference budgets—both approaches predict qualitatively the same findings: shift toward smaller overtrained models, with the shift being larger for Approach 2 than Approach 1. The consistency is non-trivial because the approaches differ in what they model (NLL vs. accuracy), their parametric structure (additive vs. distributional), and their fitting procedures (single-stage vs. two-stage). Agreement between them suggests the finding is robust to these modeling choices.
Extrapolation to held-out overtrained checkpoints (Figure 4). As described in the RQ2 results, the paper validates that both approaches extrapolate beyond the Chinchilla-optimal training data into the overtrained regime with acceptable error (2.8% for Approach 1, 8.4% for Approach 2). This is a critical robustness check because the T2 optimization recommends configurations far outside the training distribution—without this validation, the recommendations would be purely extrapolative with no empirical support. The paper includes 21 overtrained checkpoints specifically for this purpose.
Sigmoid baseline vs. Beta distribution for Approach 2. Approach 2's Beta regression model is initialized from a sigmoid baseline—the naive model that maps mean accuracy through a sigmoid and computes pass@k without distributional correction. The final Beta model outperforms this baseline by correcting for the concavity overestimate, but the paper does not report the quantitative improvement of the Beta model over the sigmoid baseline (e.g., the relative error reduction). This is a missing ablation: it would be informative to see how much the distributional correction matters quantitatively for the fitting accuracy and for the resulting optimal frontier.
Two-stage vs. end-to-end fitting for Approach 2. Approach 2 fits the Chinchilla loss parameters first (using NLL) and then fits the Beta regression parameters (using pass@k accuracy) on top of the fixed loss model. The paper does not compare this to an end-to-end approach where all parameters are jointly optimized for pass@k prediction. The two-stage approach has the advantage of simplicity and modularity, but it means the Chinchilla loss parameters are optimized for NLL—not for pass@k accuracy on the downstream tasks. A joint fit might produce different loss parameters that are better tuned for the task distribution, potentially shifting the optimal frontier. The paper does not report an ablation testing this.
Inverse-variance weighting for Chinchilla loss fit. Appendix F mentions that Approach 2's Chinchilla loss fitting uses inverse-variance weighting across isoFLOP groups, giving more weight to checkpoints with lower measurement variance. The paper does not report results without this weighting or compare to unweighted fitting, so the sensitivity of the Chinchilla loss parameters to this weighting choice is unknown.
No ablation on the number of difficulty bins or the Beta distribution assumption. The paper uses a continuous Beta distribution to model per-question accuracies. It does not test alternative distributional assumptions (e.g., logit-normal, truncated normal, nonparametric) or discrete difficulty binning (e.g., 5 bins as in the reference example paper). The Beta distribution is justified by prior work (Kazdan et al., 2025), but within this paper, there is no comparison showing that the Beta assumption is better than alternatives or that the key findings are robust to the distributional choice.
Critical Assessment
Does T2 scaling genuinely demonstrate that pretraining decisions should change when you know your test-time scaling budget?
The paper's central claim is that "optimal pretraining decisions shift radically into the overtraining regime when accounting for inference cost" (abstract) and that "if you know your test-time scaling budget, you should significantly overtrain" (Section 4.1). The experiments provide compelling evidence for this claim within the studied regime, but the claim is broader than what the experiments actually test, and several important boundary conditions are either untested or under-explored.
What the experiments demonstrate. Within the sub-1B parameter, RefinedWeb-trained model family, evaluated on eight downstream tasks, the joint optimization of $N$, $D$, and $k$ under separate training and inference budgets consistently recommends models that are smaller and more overtrained than Chinchilla-optimal models, and these models empirically outperform Chinchilla-optimal models at the same joint budget (Table 1). The finding holds for both base models and fine-tuned models (Table 2), using two modeling approaches (Figure 2), and across multiple inference budget levels (Appendix B). The empirical validation using 21 overtrained checkpoints (Figure 4) confirms that the parametric models extrapolate reasonably to the overtrained regime, and that the performance advantage is real, not an artifact of curve-fitting.
What is not demonstrated. The experiments use a single model family (Porian et al., 2024 architecture and training recipe), a single pretraining dataset (RefinedWeb), and sub-1B parameter scale. The paper argues these models are representative and the scaling laws should generalize, but provides no evidence for generalization across architectures, datasets, or scales. Modern deployments that use repeated sampling (OpenAI o1, DeepSeek-R1) involve models in the tens to hundreds of billions of parameters trained on trillions of tokens—three to four orders of magnitude beyond the studied regime. Scaling laws are known to sometimes change functional form at larger scales (e.g., the appearance of emergent abilities, changes in the power-law exponents), and the paper does not address this risk.
The paper also does not test whether the overtraining recommendation is robust to the choice of the 6ND and 2N FLOPs approximations. At very small scales, constant-factor overheads in transformer implementations can deviate from these simple formulas, and at very large scales, model parallelism and communication costs modify the effective FLOPs-per-parameter. The paper's conclusions depend on the specific exchange rate between $N$ and $k$ imposed by $k = C_{\text{inf}}/(2N)$, which assumes perfect linear scaling of inference cost with parameter count. In practice, KV-cache overhead, batch size effects, and hardware utilization curves mean the effective cost per sample is not strictly proportional to $2N$, especially for small models where overhead dominates. This could shift the optimal $N$ somewhat, though probably not enough to reverse the qualitative overtraining finding.
The inference budget as a free parameter. The paper evaluates at specific values of $C_{\text{inf}}$ but does not systematically explore how the optimal $(N, D)$ varies with $C_{\text{inf}}$. The per-task isoFLOP profiles in Appendix B show multiple $C_{\text{inf}}$ levels, but these are presented as separate plots rather than summarized into a continuous relationship. A more complete analysis would show the optimal tokens-per-parameter ratio as a function of both $C_{\text{train}}$ and $C_{\text{inf}}$, producing a surface rather than a curve. Practitioners need to know: for my specific inference budget, exactly how much should I overtrain? The paper provides the machinery to answer this but does not produce the condensed answer.
Does T2 scaling extrapolate to overtrained checkpoints in a way that validates the parametric forms?
The extrapolation results in Figure 4 show relative errors of 2.8% (Approach 1) and 8.4% (Approach 2), which the paper presents as validation. This is reasonable evidence that the parametric forms do not break down catastrophically in the overtrained regime, but it is not a strong test of extrapolation.
Why the test is limited. The overtrained checkpoints are trained with the same architecture, same dataset, same optimizer, and same hyperparameters as the Chinchilla checkpoints—the only difference is the number of training tokens. This means the extrapolation is testing whether the parametric form $B/D^\beta$ continues to hold for larger $D$, not whether the overall scaling law generalizes to different training regimes. The checkpoints are also within the same absolute scale (sub-1B parameters, up to $2.56 \times 10^{19}$ FLOPs), so the extrapolation is in $D/N$ ratio space, not in absolute compute space. A stronger test would have been to train a substantially larger model (e.g., 3B parameters) in the overtrained regime predicted by T2 and check whether the scaling law extrapolates to that new scale—but this was computationally infeasible given the paper's resources, and the authors do not claim otherwise.
The asymmetry in error direction. For Approach 2, the predictions overestimate accuracy (Figure 4, right panel). This means the model is slightly optimistic about overtrained checkpoint performance, which could lead to overestimating the benefit of overtraining. The effect is small (8.4% relative error) but systematic, and the paper does not investigate whether this bias grows at more extreme overtrained ratios.
Does the post-training analysis genuinely show that T2 scaling survives fine-tuning?
The post-training results in Figure 5 and Table 2 convincingly demonstrate that overtraining remains beneficial after fine-tuning, but with an important nuance that the paper acknowledges but does not fully explore: the attenuation.
What is shown. The T2 optimal frontier after fine-tuning lies above the Chinchilla frontier—overtraining is still recommended. The empirical overtrained checkpoints outperform Chinchilla checkpoints after both FT and SFT (Table 2). This is clear and well-supported.
What is not fully characterized. The paper attributes the attenuation to the Springer et al. (2025) finding that overtrained models are harder to fine-tune, but it does not provide a mechanistic analysis of why this is the case in its setting (e.g., does fine-tuning wipe out the per-sample quality gains more for some tasks than others? Does the optimal fine-tuning duration differ between overtrained and Chinchilla models?). The paper uses a fixed 6 epochs of fine-tuning for all checkpoints; if overtrained models require different fine-tuning hyperparameters (learning rate, number of epochs) to reach their full potential, the fixed recipe could disadvantage them relative to what a properly tuned comparison would show. The paper's finding might therefore either understate or overstate the true post-training advantage of overtraining.
The missing experiment. A natural experiment would be: for each checkpoint, sweep over fine-tuning hyperparameters to find the best post-training performance, then compare the best overtrained model against the best Chinchilla model when both are optimally fine-tuned. This would separate the effect of overtraining on base model quality from its effect on fine-tunability. The paper's fixed-epoch protocol conflates these two effects. The paper acknowledges this implicitly by citing Springer et al. (2025) but does not attempt to disentangle the mechanisms.
Are there missing baselines or comparisons that would strengthen the paper?
Several comparisons that would strengthen the experimental analysis are absent:
-
Comparison to Sardana et al. (2023) recommendations. The paper positions itself as modernizing Sardana et al. (2023) by adding repeated sampling, but it does not implement the Sardana et al. recipe and compare its recommendations to T2. Since Sardana et al. also recommends smaller models when inference volume is high, it would be informative to see whether their single-pass-aware overtraining is sufficient or whether the repeated-sampling-aware overtraining from T2 provides additional gains beyond what Sardana et al. would prescribe.
-
Test-time scaling budget spent on larger models. The inference budget correction
$k = C_{\text{inf}}/(2N)$allocates more samples to smaller models. But what if you allocated the inference budget differently—for example, giving the large Chinchilla model a small number of samples (say,$k = 4$or$k = 8$) rather than the single sample implied by$k = C_{\text{inf}}/(2N)$with large$N$? The paper's inference budget allocation is determined by the budget constraint, but it does not explore whether a different allocation strategy (e.g., minimum$k$per query regardless of$N$) would change the optimal pretraining recipe. The comparison in Table 1 uses$k = C_{\text{inf}}/(2N)$for all models, which means the Chinchilla-optimal large model gets very few samples (since$C_{\text{inf}} = 2 \times 10^9$at 901M parameters yields$k \approx 1.1$). This is a fair budget-constrained comparison, but it doesn't test sensitivity to this allocation rule. -
Majority voting as a test-time scaling strategy. The paper focuses exclusively on pass@k without verification—any correct sample in
$k$attempts counts as success. In practice, repeated sampling is often paired with a verifier or majority voting to select among the$k$samples. The paper does not evaluate whether the overtraining recommendation changes under majority voting (where the benefit of additional samples beyond a certain point may saturate faster) or under verifier-guided selection. This is a legitimate scope limitation (the paper focuses on parallel repeated sampling), but it means the results apply directly only to oracle pass@k evaluation, not to the more common deployment scenario where a selection mechanism is used. -
Latency and wall-clock time. The paper measures compute in FLOPs but never discusses latency. Repeated sampling is embarrassingly parallel—all
$k$samples can be drawn simultaneously with sufficient hardware. The inference budget$C_{\text{inf}} = 2Nk$is a FLOPs budget, not a time budget. In practice, a small model drawing 100 samples in parallel may have similar wall-clock latency to a large model drawing 1 sample, assuming batch size can be scaled. This means the inference budget constraint might not bind in the way the paper assumes for latency-constrained applications. The optimal$(N, D, k)$under a latency constraint (where all samples must complete within time$T$) could look different from the optimum under a FLOPs constraint—the paper's framework does not distinguish these.
Are the synthetic tasks well-justified?
The paper uses four synthetic tasks generated by GPT-5 and Claude Opus 4.6, each with 1,000 examples. These tasks cover simple knowledge recall, multi-step arithmetic reasoning, commonsense causal reasoning, and spatial reasoning. The rationale (Section 4 opening) is that the tasks should be "simple enough for small base models" since all checkpoints have fewer than 1B parameters. However, the paper provides no validation of these synthetic tasks—no human evaluation of quality, no comparison to established benchmarks for the same skills, no analysis of whether the tasks contain artifacts that might inflate the apparent benefit of overtraining or test-time scaling. The synthetic tasks account for half the evaluation suite, and conclusions drawn from macro averages over all eight tasks are influenced by these unvalidated tasks. The dramatic improvement on synthetic simple reasoning (57.90% vs. 18.40% in Table 1) is the largest relative gain in the paper; if this task is poorly constructed or measures something other than genuine reasoning, the headline results overstate the practical benefit.
Overall assessment.
The experiments strongly support the directional claim that joint optimization of pretraining and inference budgets recommends more overtraining than Chinchilla scaling, and that overtrained models empirically outperform Chinchilla-optimal models under test-time scaling budgets. The evidence is internally consistent across two modeling approaches, eight tasks, base and fine-tuned models, and multiple inference budgets. However, the experiments are limited in scale and scope: sub-1B parameters, single architecture, single dataset, oracle pass@k evaluation without verifiers or majority voting, no latency analysis, and no comparison to the closest prior work (Sardana et al., 2023). The paper's recommendations should be treated as qualitatively validated at small scale but not yet quantitatively calibrated for the deployment regimes where they would have the most impact (7B–70B parameter models, trillions of training tokens, production inference pipelines). The paper is transparent about most of these limitations (Section 5), and the experiments are appropriate for establishing the phenomenon and motivating larger-scale validation.
6. Limitations and Trade-offs
The Entire Analysis Is Confined to Sub-1B Parameter Models Trained on a Single Dataset
The constraint. All experiments use a single model family—transformer checkpoints from Porian et al. (2024) plus the authors' extensions—trained on the RefinedWeb dataset (Penedo et al., 2023) with model sizes spanning 5M to 901M parameters and training FLOPs up to 2.56 × 10¹⁹. The paper acknowledges this scope limitation in Section 5:
"In future work, we plan to validate our prescribed overtraining recipes at larger scales."
However, the main-text claims (Figure 2, Tables 1–2) are stated without qualification about scale, and Figure 2 extrapolates the T2 scaling law fits to 10²⁵ FLOPs—roughly six orders of magnitude beyond the empirical range and into the regime of models like Chinchilla-70B and beyond. The extrapolation in Figure 2 is purely parametric: it assumes the power-law forms fitted on sub-1B models hold unchanged across those six orders of magnitude.
The consequence. Scaling laws are not guaranteed to maintain the same functional form across orders-of-magnitude extrapolation. The Chinchilla exponents (α, β) fitted on small models have been observed to shift at larger scales in some settings (Kaplan et al., 2020 vs. Hoffmann et al., 2022 found different exponents partly due to different scale ranges). More critically, the Beta regression parameters (θ₀ through θ₄) that map Chinchilla loss to per-question accuracy distributions in Approach 2 are fitted entirely on sub-1B models with loss values in a specific range. It is unknown whether the loss-to-accuracy sigmoid saturates at the same ceiling accuracy for much larger models, or whether the concentration parameter ν's dependence on loss changes functional form when models enter a regime where most questions have non-trivial accuracy. If the scaling exponents change or the Beta regression saturates differently at scale, the extrapolated optimal tokens-per-parameter ratios in Figure 2 could be off by orders of magnitude—potentially over-recommending or under-recommending overtraining relative to the true large-scale optimum.
A secondary consequence: all empirical validation of the T2 recommendation (Tables 1–2) occurs within the same scale regime. The overtrained checkpoints that outperform Chinchilla-optimal ones are 37M–149M parameters. The paper has not demonstrated that a 7B parameter model overtrained to the T2-prescribed ratio would outperform a Chinchilla-optimal 70B model under repeated sampling—this is a prediction of the scaling laws, not an empirical result.
Evidence in the paper. Figure 4 shows that the parametric fits extrapolate to overtrained checkpoints within the same scale regime (5M–901M parameters) with 2.8% (Approach 1) and 8.4% (Approach 2) relative error. This validates that the functional forms hold within the studied scale range when moving from Chinchilla-optimal to overtrained token ratios. However, it provides no evidence about extrapolation in absolute scale—the overtrained checkpoints are at the same parameter range as the fitting data, just at higher D/N ratios. Figure 2 explicitly extends the fits to 10²⁵ FLOPs and plots optimal N, D, and D/N curves across this range, but these are extrapolations with no empirical anchor points beyond ~2.6 × 10¹⁹ FLOPs. The paper does not flag uncertainty bands or confidence intervals on these extrapolated curves.
Mitigation status. The paper partially mitigates through the two-approach triangulation: Approach 1 and Approach 2 have different parametric structures and failure modes, so their agreement on the qualitative overtraining direction provides some confidence that the finding is not an artifact of a single extrapolation formula. However, both approaches share the same underlying Chinchilla loss model fitted to the same data, so they could both fail in correlated ways at larger scales. The paper explicitly calls for larger-scale validation as future work (Section 5) but does not perform any such validation in this study.
The Inference Budget Coupling Assumes Perfect Linear Scaling of Inference Cost with Parameter Count
The assumption. The entire joint optimization framework rests on the inference cost model C_inf = 2Nk, where 2N approximates the FLOPs for a single forward pass per token. This assumption is embedded in the coupling equation k = C_inf / (2N) that drives the overtraining result: smaller models get more samples because they cost exactly proportionally less per forward pass. The paper inherits this 2N approximation from the standard scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) and applies it without modification or validation for the specific model sizes and hardware setups used.
The consequence. In practice, the effective cost per token of a transformer forward pass is not strictly proportional to N for several reasons. First, there are constant-overhead components (embedding lookups, final layer projections) that do not scale with model depth and become proportionally larger for very small models—meaning small models are relatively more expensive per parameter than the 2N approximation assumes. Second, hardware utilization varies with model size: very small models may not saturate GPU compute units, reducing effective throughput per FLOP, while very large models may be bottlenecked by memory bandwidth or communication rather than raw FLOPs. Third, when batching multiple samples, KV-cache memory usage scales with batch size and sequence length in ways that can constrain throughput independently of parameter count.
If the true inference cost scales sub-linearly with N (i.e., small models have higher overhead per parameter), then the budget coupling k = C_inf / (2N) overestimates how many extra samples a small model gets relative to a large one. This would weaken the overtraining recommendation—the sample-count advantage of small models would be smaller than the paper assumes, pushing the optimal N back toward larger values. Conversely, if the cost scales super-linearly with N (e.g., due to communication overheads at very large scales), the overtraining recommendation would be even stronger than the paper finds. The magnitude and even the sign of this correction is unknown and likely hardware-dependent.
A second consequence: the paper compares models at vastly different parameter counts (37M vs. 901M) and allocates inference samples proportionally (k ≈ 27 for the 37M model vs. k ≈ 1.1 for the 901M model at C_inf = 2 × 10⁹). If the 37M model's per-token inference overhead is 2× higher than the 2N approximation predicts (e.g., due to fixed embedding costs dominating at small scales), the effective k would be closer to 13–14 rather than 27, substantially changing the pass@k calculation. The paper does not report actual measured inference FLOPs or wall-clock times for any checkpoint.
Evidence in the paper. The paper provides no measurements of actual inference cost and no validation of the 2N approximation for its specific models and hardware. The FLOPs accounting in Section 3.3 is stated as a definition:
"C_inf = 2Nk"
without empirical validation or sensitivity analysis. Appendix C mentions training hyperparameters and hardware (bfloat16 mixed precision, OpenLM framework) but does not profile inference cost. The paper does not test how sensitive the optimal (N, D) recommendations are to the coefficient in the inference cost model—for example, whether using 3N or 1.5N instead of 2N would meaningfully shift the optimal tokens-per-parameter ratio.
Mitigation status. Not addressed. The paper does not discuss the limitations of the 2N approximation, does not test sensitivity to the inference cost coefficient, and does not report measured inference costs. The assumption is taken as standard from the scaling laws literature, but its validity at the extreme model size ratios the paper recommends (where overhead effects are most pronounced) is untested. A sensitivity analysis varying the inference cost coefficient would partially mitigate this concern by showing whether the qualitative overtraining finding survives under different cost models, but no such analysis is performed.
Difficulty Estimation Cost for the Beta Regression Model Could Dominate the Inference Budget Being Optimized
The constraint. Approach 2 requires estimating the per-question accuracy distribution (modeled as a Beta distribution) as a function of the Chinchilla loss. To fit the Beta regression parameters (θ₀ through θ₄), the paper needs per-question single-pass accuracy measurements p_i for every checkpoint in the population—106 checkpoints × 8 tasks × hundreds to thousands of questions per task, each requiring a forward pass to compute the model's likelihood of the correct answer. Additionally, to compute the empirical pass@k values used as regression targets, the paper needs to sample multiple values of k and compute the expected pass@k from the per-question accuracies. This fitting cost is amortized over the scaling law development and is not part of the per-query deployment cost—it is a one-time research expense.
However, a deployment of the T2 framework in practice would face a related cost: to determine the optimal (N, D) for a new model family, dataset, or task distribution, one would need to train a population of checkpoints, measure per-question accuracies across the target tasks, and refit the scaling laws. This is the same cost that any Chinchilla-style scaling law requires for a new setting—training dozens of models at different scales—so it is not a new cost introduced by T2. But the paper does not discuss whether the Beta regression component of Approach 2 requires more evaluation data per checkpoint than a standard Chinchilla fit (which only requires aggregate NLL), or whether the per-question accuracy measurements introduce additional variance that requires larger evaluation sets for stable fitting.
The consequence. The practical barrier to adopting T2 scaling for a new model family is the standard Chinchilla scaling cost (training a grid of checkpoints) plus the additional requirement of evaluating per-question accuracies on each downstream task of interest. For organizations that already perform Chinchilla-style scaling studies, the marginal cost of adding T2 is modest—they already have the checkpoints and can run the per-question evaluations. For organizations that do not, the full cost of a T2 scaling study may be prohibitive. The paper does not provide guidance on the minimum checkpoint population or evaluation set size needed for reliable T2 fits, making it difficult for practitioners to assess the cost.
Evidence in the paper. The checkpoint grid (Figure 8) uses 106 checkpoints (85 from Porian et al., 2024 plus 21 overtrained). The evaluation tasks use between 1,000 examples (synthetic tasks) and the standard test set sizes for the real benchmarks (LAMBADA: number not specified in the paper but typically ~5,000; ARC-Easy: ~2,400; SciQ: ~1,000; OpenBookQA: ~500). The paper does not report the total evaluation FLOPs spent computing per-question accuracies, nor does it analyze how the T2 fits degrade with fewer checkpoints or smaller evaluation sets. The fitting procedures (Appendix F) use the full checkpoint population with no ablation on population size.
Mitigation status. Not addressed. The paper treats the checkpoint population and per-question evaluations as given, without analyzing their cost or providing guidance on budget-constrained T2 fitting. This is a reasonable scope choice for a research paper establishing the scaling phenomenon, but it limits the immediate practical applicability. Future work would need to characterize the data requirements for reliable T2 fitting—minimum number of checkpoints, minimum evaluation examples per task, and whether aggregated pass@k measurements (computed from a modest number of samples per checkpoint rather than exact per-question likelihoods) suffice.
The Post-Training Analysis Uses a Fixed Fine-Tuning Protocol That May Disadvantage Overtrained Models
The constraint. The fine-tuning experiments in Section 4.3 and Appendix D use a fixed protocol: all checkpoints are fine-tuned for 6 epochs with a batch size of 8 and a constant learning rate of 2 × 10⁻⁵, on 4 NVIDIA A10 GPUs. The paper states:
"We allocate the same number of training steps to each checkpoint, rather than scaling training based on FLOPs, since we ultimately train to convergence."
This protocol assumes that all checkpoints—ranging from 37M to 901M parameters, and from Chinchilla-optimal to heavily overtrained—reach their peak fine-tuned performance at exactly 6 epochs. It does not sweep over learning rates, numbers of epochs, or other hyperparameters per checkpoint.
The consequence. The paper cites Springer et al. (2025) for the finding that "overtrained models are harder to fine-tune." If this is true in the present setting, then the fixed 6-epoch protocol may systematically disadvantage overtrained models in one of two ways: (1) overtrained models may require more epochs to reach their full fine-tuned potential (since they are "harder to fine-tune"), in which case the fixed protocol understates their true performance; or (2) overtrained models may overfit faster during fine-tuning (since they start from a stronger base performance on the pretraining distribution), in which case 6 epochs may be too many and the fixed protocol understates their performance by overshooting. Without sweeping fine-tuning hyperparameters per checkpoint, the paper cannot distinguish these effects from the genuine performance ceiling.
The paper's finding that the T2 overtraining recommendation is "subdued" after fine-tuning (Figure 5) could therefore reflect either: (a) a genuine reduction in the benefit of overtraining due to the fine-tuning process equalizing model quality, or (b) a suboptimal fine-tuning protocol that leaves overtrained models further from their achievable peak than Chinchilla models. The paper's interpretation favors (a), citing Springer et al. (2025), but the experimental design cannot rule out (b).
A related concern: the paper's fine-tuning trains to convergence on the downstream task training sets, which for ARC-Easy, SciQ, and OpenBookQA are relatively small (ARC-Easy: ~2,200 training examples; SciQ: ~11,700; OpenBookQA: ~5,000). On such small datasets, the variance in fine-tuned performance across random seeds is likely substantial, and the fixed-protocol single-run evaluation may not reliably rank checkpoints. The paper does not report multiple fine-tuning runs with different random seeds or provide confidence intervals on the post-training results.
Evidence in the paper. Table 2 reports the best overtrained and Chinchilla-optimal checkpoint performances after FT and SFT at C_train = 2.56 × 10¹⁹ and C_inf = 2 × 10⁹. The post-training results consistently show overtrained checkpoints outperforming Chinchilla-optimal ones, which suggests that any protocol-induced disadvantage is not severe enough to reverse the qualitative finding. However, the magnitude of the advantage could be misestimated. Figure 5 shows that the optimal tokens-per-parameter ratio is lower after fine-tuning than for base models, but whether this attenuation is partly an artifact of the fixed fine-tuning protocol is untested. The paper acknowledges the Springer et al. (2025) finding in Section 4.3:
"The finding that it is subdued is consistent with prior work showing that overtrained models are harder to fine-tune"
but does not test whether fine-tuning hyperparameter optimization would recover some of the lost advantage.
Mitigation status. Partially addressed through SFT vs. FT comparison. The paper tests two fine-tuning objectives (FT, which trains on instruction + answer, and SFT, which trains on answer only) and finds that both preserve the overtraining advantage. This shows the finding is not specific to one fine-tuning objective. However, it does not address the hyperparameter sweep concern—both FT and SFT use the same fixed 6-epoch, fixed learning rate protocol. The paper does not suggest that future work should sweep fine-tuning hyperparameters per checkpoint, framing the attenuation as a genuine effect (citing Springer et al., 2025) rather than a potential artifact.
Pass@k Without Verification Overestimates the Practical Benefit of Repeated Sampling
The constraint. The paper evaluates all models using oracle pass@k: a question is counted as correct if any of the k independent samples produces the correct answer. This assumes a perfect verification mechanism—a verifier that can look at k candidate answers and identify the correct one with 100% accuracy. In practice, repeated sampling deployments almost always rely on imperfect selection mechanisms: majority voting (which can select a wrong answer if the plurality is incorrect), reward model scoring (which can be fooled by plausible-but-wrong answers), or unit test execution for code (which may have incomplete coverage). The paper does not evaluate any of these practical selection mechanisms; it assumes access to an oracle verifier.
The consequence. Oracle pass@k is an upper bound on the performance achievable with any practical selection mechanism. The benefit of additional samples (larger k) is likely overstated by oracle pass@k relative to majority voting or verifier-guided selection, because those methods have non-zero error rates in identifying the correct answer among the k candidates. The error rate of the selection mechanism interacts with k: if the verifier has a fixed probability of selecting the wrong answer regardless of k, then increasing k may not help as much as oracle pass@k predicts. More subtly, if the verifier's error rate increases with k (because larger candidate pools contain more plausible-looking wrong answers), then there may be an optimal k beyond which additional samples actually reduce accuracy—a phenomenon the oracle pass@k metric cannot capture.
The overtraining recommendation from T2 scaling depends critically on the shape of the pass@k curve: how much does additional sampling improve success probability? If the true pass@k under a practical verifier saturates faster than oracle pass@k (because the verifier fails to identify correct answers in large candidate pools), then the benefit of having a smaller model with more samples is reduced, and the optimal pretraining recipe would shift back toward larger models with fewer samples. The paper's recommendation may therefore be calibrated for an idealized deployment scenario that does not exist in practice.
Evidence in the paper. The paper does not evaluate any verifier or selection mechanism. The pass@k metric is defined in Section 2:
"For a single problem i with per-sample success probability p_i, the probability of at least one answer in k attempts being correct is pass@k_i = 1 - (1 - p_i)^k."
This is the oracle pass@k definition. The paper never mentions majority voting, reward models, or verifier-guided selection as alternatives, and does not discuss the gap between oracle pass@k and practical achievable accuracy with imperfect verification. The related work section (Appendix A.2) mentions verification-based test-time scaling (Saad-Falcon et al., 2025) but the paper's own experiments do not use it.
Mitigation status. Not addressed. The paper's scope is explicitly limited to parallel repeated sampling without discussing verification. Section 5 mentions "account for transformer-specific inference cost models" as future work but does not mention adding verification to the framework. This is a legitimate scope choice for establishing the core scaling phenomena, but it means practitioners cannot directly apply the T2 recommendations to deployments that use majority voting or reward model verification without additional analysis of how the verification gap modifies the optimal pretraining recipe.
The Checkpoint Population and Evaluation Scale Limit the Statistical Reliability of the Optimal Frontier
The constraint. The empirical foundation for all T2 fits is a grid of 106 checkpoints spanning approximately one order of magnitude in model size (5M to 901M parameters) and roughly three orders of magnitude in training FLOPs (1.25 × 10¹⁶ to 2.56 × 10¹⁹). The crucial validation in Table 1 compares the best overtrained checkpoint against the best Chinchilla-optimal checkpoint at a single pretraining budget (C_train = 2.56 × 10¹⁹). However, the checkpoint grid is relatively sparse at this budget level: the Chinchilla-optimal candidates are the larger models (455M, 611M, 901M), and the overtrained candidates are the smaller models (37M, 84M, 149M). The "best" is selected from a small number of candidates at each end of the spectrum, and there are no checkpoints at intermediate sizes or intermediate tokens-per-parameter ratios that could reveal whether the performance surface has local optima or plateaus.
The consequence. The finding that "the best overtrained checkpoint outperforms the best Chinchilla-optimal checkpoint" could be sensitive to the specific checkpoints available. If the Chinchilla-optimal candidate at this budget happened to be a particularly poor checkpoint (e.g., due to training instability or an unfortunate position in the loss landscape) and the overtrained candidate happened to be particularly good, the comparison would overstate the benefit of overtraining. More fundamentally, the discrete nature of the checkpoint grid means the paper is selecting the best from a coarse set rather than identifying the true optimum. The continuous T2 scaling law fits smooth over this discreteness and predict the optimal N and D, but the empirical validation in Table 1 uses the actual checkpoints, not the predicted optima. If the best Chinchilla-optimal checkpoint in the grid is not actually at the Chinchilla optimum for that compute budget (because the grid resolution is limited), the comparison is not a clean test of T2 vs. Chinchilla.
Additionally, the paper does not report confidence intervals on the optimal frontier positions in Figures 2, 3, 5, 6, or 7. The Chinchilla loss fitting and Beta regression fitting produce point estimates without uncertainty quantification. Given the relatively small number of checkpoints (106) spread across a large (N, D) space, there is likely substantial uncertainty in the fitted parameters—particularly in the extrapolated regime. Without error bars or confidence bands, practitioners cannot assess whether the difference between Approach 1 and Approach 2's recommendations (which can be an order of magnitude in D/N ratio) is statistically distinguishable or within the fitting uncertainty.
Evidence in the paper. The checkpoint grid is visualized in Figure 8, showing the tokens-per-parameter ratio for each cell. The grid has good coverage near the Chinchilla optimum (including checkpoints both above and below the 20:1 ratio) but sparser coverage in the heavily overtrained regime (the orange cells, which are concentrated at specific (N, D) combinations trained for this work). The paper does not report standard errors, confidence intervals, or bootstrap uncertainty for any fitted parameter or optimal frontier. The fitting procedures (Appendix F) minimize sum of squared errors and select the best-fitting restart, but do not characterize the distribution of parameter estimates across restarts or across bootstrapped resamples of the checkpoint population.
Mitigation status. The two-approach triangulation partially mitigates the point-estimate concern by showing that two structurally different models converge on the same qualitative finding. If the checkpoint grid were so sparse or noisy that the fits were unreliable, it would be unlikely for both approaches to agree by chance. However, this does not provide quantitative uncertainty bounds. The paper does not discuss checkpoint grid resolution, the number of checkpoints needed for reliable T2 fits, or how the results would change with a denser or differently distributed grid. The extrapolation validation in Figure 4 provides evidence that the fits are well-behaved for the held-out overtrained checkpoints, but this is a point-prediction evaluation, not an uncertainty quantification.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper introduces a structural shift in how the field should think about scaling laws, moving from a two-stage paradigm where pretraining and inference are optimized independently to a joint optimization framework where they are tightly coupled. This is not an incremental refinement—it is a reframing of the scaling problem that changes what counts as "optimal" and reinterprets existing practices as rational responses to a previously unarticulated optimization objective.
Pre-T2: two independent problems. Before this work, the field operated with an implicit separation of concerns. Pretraining scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) told you how to allocate training compute between model size and training tokens to minimize single-pass loss. Test-time scaling laws (Snell et al., 2024; Brown et al., 2025) told you how many samples to draw from a fixed model to maximize pass@k. The two communities shared a platform—pretrained models—but not an optimization objective. The consequence was a structural blind spot: Chinchilla-optimal models were never evaluated on whether they were optimal for being sampled from hundreds of times, because the optimization criterion used to train them did not include that downstream usage. Practitioners who overtrained models (Llama-2, Gemma, OLMo) were seen as deliberately departing from optimality for cost reasons—a pragmatic compromise, not a principled recipe.
Post-T2: one joint problem. The paper demonstrates that when you fold inference cost into the optimization through the coupling k = C_inf / (2N), the optimal pretraining recipe shifts dramatically—by orders of magnitude in the tokens-per-parameter ratio—away from Chinchilla. This makes overtraining not a departure from optimality but the optimal strategy under a more complete specification of the problem. The paper thereby transforms the field's narrative around overtraining: it is no longer "overtraining is for inference efficiency, at the cost of training efficiency," but rather "overtraining is compute-optimal when you account for how the model will actually be used."
This reframing resolves a tension that was hiding in plain sight. The Chinchilla 20:1 tokens-per-parameter recommendation was widely cited but increasingly ignored in practice, yet there was no principled alternative. The field lacked a vocabulary to explain why practitioners were departing from Chinchilla other than "inference is expensive." T2 provides that vocabulary: the departure is optimal under a joint training-inference budget, and the magnitude of the departure should be a function of the inference intensity. The paper's Figure 2 gives a concrete answer to "how much should I overtrain?" that depends on the joint budget—a question practitioners previously answered by guesswork.
A new diagnostic: non-monotonic Chinchilla scaling under inference correction. One of the paper's most clarifying contributions is the demonstration that Chinchilla-optimal models, when evaluated under a fixed inference budget with k = C_inf / (2N), produce non-monotonic accuracy as a function of pretraining compute (Figure 3, inference-corrected panels). Investing more pretraining compute can reduce accuracy because the larger model it recommends eats into the inference budget and reduces the number of samples below a critical threshold. This is a diagnostic that the pretraining recipe is mismatched to the deployment strategy: if your scaling curves are non-monotonic under inference correction, you are optimizing the wrong objective. T2 scaling produces consistently monotonic improvement because it jointly optimizes N, D, and k under the full budget. This diagnostic gives the field a concrete test for whether a pretraining recipe is deployment-aware: evaluate models across a range of sizes under a fixed inference budget mimicking intended usage, and check for monotonicity.
Reconciling conflicting intuitions about small models with more samples. Prior work showed that smaller models with more test-time compute can match larger models (Snell et al., 2024; Brown et al., 2025), but these findings were treated as an empirical curiosity—an observation about what can happen, not a prescription about what should happen. T2 bridges this gap by showing that the observation falls out of a joint optimization framework: the smaller model gets more samples because the inference budget allows it, and the per-sample quality improvement from overtraining amplifies the pass@k compounding. What was an empirical observation becomes a predicted optimum. This transforms the conversation from "small models with more samples can sometimes compete" to "small overtrained models with more samples should be what you train if you know your deployment strategy."
Research directions become more and less attractive. T2 makes several directions newly attractive. First, overtraining-aware pretraining recipes become a design space rather than a post-hoc fix—how do architecture choices, data mixtures, and optimizer hyperparameters interact with the optimal D/N ratio under test-time scaling? Second, joint pretraining-inference scaling laws at scale become a high-priority empirical target—the paper establishes the phenomenon at sub-1B scale, and replicating it at 7B–70B scale with modern architectures and datasets would determine whether the recommended overtraining ratios hold or shift at scale. Third, verifier-aware T2 scaling becomes a natural extension—since practical deployments use imperfect verifiers rather than oracle pass@k, the interaction between verifier quality, model size, overtraining, and sample count becomes a tractable optimization problem that T2 provides the framework for.
Conversely, T2 makes several existing practices less attractive. The sequential approach of "train Chinchilla-optimal, then figure out test-time scaling separately" is shown to be suboptimal—it leaves performance on the table compared to joint optimization. The practice of publishing a single scaling law fit (for single-pass loss) without specifying the intended inference regime becomes underspecified—a Chinchilla fit for a model deployed with greedy decoding is genuinely different from one intended for heavy repeated sampling. And the habit of comparing models at equal parameter count or equal training FLOPs without inference correction becomes misleading—Table 1 shows that a 37M overtrained model can outperform a 901M Chinchilla model under the same joint budget, meaning raw size comparisons without accounting for inference strategy are not just incomplete but potentially inverted.
A qualification on magnitude. The paper establishes the direction of the shift convincingly but the magnitude remains uncertain at practical scales. The extrapolated tokens-per-parameter ratios in Figure 2 (reaching 10^6 at 10^25 FLOPs for Approach 2) are parametric projections from sub-1B models and should not be taken as calibrated recommendations for frontier-scale training. The paper's contribution is the framework and the qualitative finding, not the specific numbers at extrapolated scales. The most important landscape change is that the question "how should I train my model?" now has a new required input: "how will you sample from it at deployment?"
Follow-Up Research This Work Enables
Large-scale replication of T2 scaling laws at 1B–70B parameters. The paper's entire empirical foundation is sub-1B models trained on RefinedWeb. The most urgent follow-up is to determine whether the overtraining recommendation holds, shifts, or disappears at the scales where modern deployments actually operate. A strong follow-up would train a Chinchilla-style grid at 1B, 3B, 7B, and 30B parameter scales—each with checkpoints spanning from Chinchilla-optimal to heavily overtrained (100×–1000× the 20:1 ratio)—on a standard pretraining corpus like C4 or the Pile, evaluate on reasoning and knowledge benchmarks with inference budgets mimicking typical deployment (e.g., C_inf values corresponding to 4, 16, 64, and 256 samples at each model size), and fit both T2 approaches to determine whether the optimal D/N ratio scales with C_train according to the same power-law trend as the sub-1B fit, or whether the exponents change. The key measurements would be: (a) whether the non-monotonicity of the Chinchilla frontier under inference correction appears at larger scales with the same severity, (b) whether Approach 1 and Approach 2 continue to agree on the direction and approximate magnitude of overtraining, and (c) whether the extrapolated curves from the sub-1B fit actually predict the large-scale optima, or whether there is a systematic bias that requires refitting. This experiment is expensive but essential—without it, the T2 framework remains an intriguing small-scale phenomenon rather than a deployment-ready tool.
Verifier-aware joint scaling with imperfect selection mechanisms. The paper evaluates oracle pass@k, which assumes perfect identification of the correct answer among k candidates. In practice, repeated sampling uses majority voting or a learned verifier (reward model), both of which have error rates that interact with k and N. A natural extension would be to model pass@k under majority voting: for a given distribution of per-question accuracies (already modeled by the Beta distribution in Approach 2), the probability that the correct answer receives a plurality among k samples can be computed from order statistics of correlated Bernoulli trials (since samples from the same model on the same question are not independent in their correctness, but the paper's framework assumes independent samples). A follow-up would fit a modified T2 model where accuracy is defined as P(plurality is correct | N, D, k) instead of 1 - (1 - p)^k, using the same Beta distribution over per-question p_i but with a different aggregation function. The key question: does the optimal (N, D) shift back toward larger models under majority voting (because majority voting saturates faster with k than oracle pass@k, reducing the sample-count advantage of small models), or does it remain similarly overtrained? And does the answer depend on the difficulty distribution—are there tasks where majority voting and oracle pass@k diverge enough to recommend qualitatively different pretraining recipes? This would directly address the gap between the paper's idealized evaluation and practical deployment.
Interaction between overtraining and test-time search strategies beyond parallel sampling. The paper focuses on parallel repeated sampling (independent draws). Modern reasoning systems (OpenAI o1, DeepSeek-R1) combine parallel sampling with sequential refinement (chain-of-thought, tree search, iterative revision). The interaction between overtraining and search strategy is unexplored. A follow-up could extend the T2 framework to model k not as the number of independent samples but as a budget of inference tokens that can be split between parallel exploration (breadth) and sequential refinement (depth). The key extension: instead of k = C_inf / (2N) representing independent samples, model k = (parallel_samples × average_sequence_length) and let the allocation between parallel and sequential tokens be an additional optimization variable. The question: does overtraining remain optimal when inference compute is spent on longer reasoning chains rather than more independent samples? If overtraining primarily helps per-sample quality, and sequential refinement is a way to improve quality within a single sample, then the two strategies might be substitutes—overtraining might be less beneficial when sequential refinement is the primary test-time strategy. Alternatively, if overtraining improves the model's ability to benefit from sequential refinement (e.g., by learning better intermediate reasoning steps), they could be complements. This experiment would connect T2 to the tree-search and chain-of-thought scaling literature (Snell et al., 2024; Madaan et al., 2023) and determine whether the overtraining prescription is specific to parallel sampling or generalizes across test-time strategies.
Training data quality and composition interact with the optimal overtraining ratio. The paper trains all checkpoints on a single dataset (RefinedWeb). In practice, pretraining data mixtures are carefully engineered, and data quality is known to affect scaling law parameters (Goyal et al., 2024). A follow-up could investigate whether the optimal D/N ratio from T2 depends on data quality and composition. The hypothesis: higher-quality data might reduce the benefits of overtraining because per-sample quality saturates faster—you hit diminishing returns on additional tokens sooner when every token is informative. Conversely, lower-quality or more diverse data might increase the benefits of overtraining because the model needs more tokens to extract the same signal. A concrete experiment: pretrain two parallel checkpoint grids at the same sub-1B scale, one on raw web data (RefinedWeb) and one on a filtered, high-quality subset (e.g., Wikipedia + books + code), fit T2 scaling to both, and compare the optimal D/N ratios. If the ratio is substantially lower for the high-quality data, then the T2 prescription is not a universal number but depends on data curation strategy—an important nuance for practitioners.
Negative result: stress-testing T2 on tasks where base model accuracy is near zero. The paper's hardest tasks (ARC-Easy for base models in Table 1: 0.14% vs. 0.07%) show near-zero pass@k for both overtrained and Chinchilla models. T2 recommends overtraining because it improves per-sample quality, which is then amplified by repeated sampling. But if the base model's per-sample accuracy is essentially zero on a task, no amount of overtraining or repeated sampling helps—the pass@k remains zero because (1 - 0)^k = 0 and 1 - 0^k = 0. This suggests a boundary condition: T2's overtraining recommendation is only relevant for tasks within the model's capability range, where p_i > 0 for at least some questions. A valuable negative-result experiment would be to fit T2 on a task where scaling model size yields zero accuracy until a threshold is crossed (a classic emergent ability pattern), and test whether T2 erroneously recommends an overtrained small model that never crosses the threshold. If T2 fails in this regime—recommending a small model that can never succeed, when a larger model past the emergence threshold would succeed with even a single sample—it would delineate a clear failure mode and a necessary condition for T2's applicability: the base model's single-pass accuracy must be non-trivially above zero on the target task distribution. This would prevent over-application of T2 to tasks that fundamentally require scale.
Adaptive inference budget allocation as a function of query difficulty. The paper allocates k = C_inf / (2N) uniformly across all queries. In practice, queries vary in difficulty, and a smarter system would allocate more samples to harder queries and fewer to easier ones—similar to the adaptive test-time compute strategies in Snell et al. (2024). T2 could be extended to model difficulty-conditioned sample allocation: estimate per-query difficulty from a small number of initial samples (or from query features), then set k_i adaptively within the total inference budget constraint E_i[2N k_i] ≤ C_inf. The optimization would now include a difficulty estimator and an allocation policy, and the pretraining recipe (N, D) would be jointly optimized with the allocation strategy. The question: does adaptive allocation change the optimal overtraining ratio? If the system can allocate more samples to hard queries (where repeated sampling helps most) and fewer to easy queries (where one sample suffices), the average benefit of additional samples might be higher, potentially strengthening the case for small overtrained models. Or, if adaptive allocation means the inference budget is spent more efficiently, the optimal N might shift back toward larger models because the sample budget needed to achieve good performance is lower. This connects T2 to the difficulty-conditioned compute-optimal scaling framework and would produce a unified theory of pretraining, overtraining, and adaptive test-time scaling.
Practical Applications and Downstream Use Cases
Pretraining recipe design for reasoning-focused model families. The most immediate practical application is for teams designing foundation models intended for reasoning-heavy deployment. If a model will be queried with repeated sampling—as is standard for math, code, and scientific reasoning applications—T2 provides a principled alternative to the Chinchilla 20:1 rule of thumb. Concretely, at the paper's scale of C_train ≈ 2.6 × 10^19 FLOPs and C_inf ≈ 2 × 10^9 FLOPs per query, T2 recommends models in the 37M–149M parameter range with tokens-per-parameter ratios far exceeding 20:1, rather than Chinchilla's recommendation of 455M–901M parameters. A team training a reasoning model at larger scale (e.g., 7B parameters with access to trillions of training tokens) could use T2 scaled up to determine their optimal configuration: instead of training a Chinchilla-optimal 7B model on ~140B tokens, T2 might recommend training a 1.5B model on ~1.4T tokens (a 10× increase in tokens-per-parameter) if heavy repeated sampling is planned. The cost savings in inference—a 1.5B model is ~5× cheaper per forward pass than a 7B model, allowing ~5× more samples under the same inference budget—can then be reinvested into pass@k gains. The paper's framework provides the specific parametric forms and fitting procedures needed to estimate these numbers for a given model family and task distribution, turning a qualitative intuition ("overtrain for inference") into a quantitative prescription.
Cost-efficient deployment of small models with repeated sampling for batch evaluation. Organizations that run large-scale batch inference—evaluating thousands of candidate solutions, generating synthetic training data, or scoring model outputs—face a tradeoff between per-query cost and per-query accuracy. T2's finding that a 37M overtrained model can achieve 49.90% pass@k vs. 27.30% for a 455M Chinchilla model on LAMBADA (Table 1) under the same joint budget has direct cost implications. If the batch job consists of 1 million queries at C_inf = 2 × 10^9 FLOPs each, the overtrained 37M model would cost the same total inference FLOPs as the Chinchilla 455M model but would deliver nearly 1.8× more correct answers. Alternatively, if the acceptable accuracy threshold is fixed at 27%, the 37M model could achieve it with substantially fewer samples (lower k) and therefore lower per-query cost, since its per-sample quality after overtraining means fewer retries are needed. For organizations running these batch jobs at scale, the savings compound: switching from a Chinchilla-optimal model family to a T2-optimal one at the same total compute budget could mean either higher accuracy for the same cost or lower cost for the same accuracy, with the optimal choice determined by the downstream tolerance for errors. The T2 framework provides the scaling curves needed to compute this tradeoff quantitatively for a specific task distribution.
Guiding the "distillation from large models" pipeline. A common practice is to train a large teacher model, generate high-quality outputs through repeated sampling from it, and distill those outputs into a smaller student model. T2 suggests a more efficient alternative: instead of using a large Chinchilla-optimal teacher, use a moderately-sized overtrained teacher that achieves higher pass@k under the same joint budget. For example, Table 1 shows that a 37M overtrained model achieves 57.90% on synthetic simple reasoning vs. 18.40% for a 901M Chinchilla model—the overtrained model produces correct answers at 3× the rate under the same total compute. Using the overtrained model as the teacher would generate more correct training examples for the student at the same cost, improving the student's quality. More provocatively, T2 suggests that the "small student" might itself be a T2-optimal model—that is, the distillation target might not be a tiny model trained on synthetic data, but a small overtrained model trained on real (and synthetic) data at a T2-prescribed ratio from the start. This collapses the two-stage teacher-student pipeline into a single-stage T2 training recipe, potentially simplifying the deployment architecture while achieving similar or better performance per dollar.
When to Prefer This Method
The paper implicitly defines a tradeoff between T2-optimal pretraining (joint optimization of N, D, and k under training and inference budgets) and the status-quo approach (Chinchilla-optimal pretraining followed by separate test-time scaling decisions). The conditions under which T2 is preferable can be inferred from the paper's findings:
-
Prefer T2 scaling when the model will be deployed with substantial repeated sampling (large
C_infper query relative to the per-sample inference cost). The paper shows that T2's overtraining recommendation strengthens asC_infincreases—the larger the inference budget, the more samples small models can draw, and the stronger the case for overtraining. The status quo (Chinchilla + test-time scaling) produces non-monotonic scaling under largeC_inf(Figure 3), while T2 avoids this failure mode. -
Prefer T2 scaling when deploying at small-to-moderate model sizes for cost-sensitive applications. The paper's empirical validation at 37M–149M parameters shows that overtrained models in this range substantially outperform much larger Chinchilla-optimal models (up to 901M) under the same joint budget. For on-device deployment, edge computing, or applications where per-query infrastructure cost is the binding constraint, T2 provides a recipe for maximizing accuracy within a tight inference budget.
-
Prefer the status quo when the model will be deployed primarily with single-sample or few-sample queries (
k ≈ 1). AsC_infapproaches2N(i.e.,k → 1), the inference correction term in the T2 loss vanishes, and the optimization reduces to standard Chinchilla scaling. In this regime, T2 offers no benefit over Chinchilla, and the additional complexity of joint optimization is unnecessary. -
Prefer the status quo when the target task is fundamentally outside the base model's capability range, as indicated by near-zero single-pass accuracy even after overtraining. T2's mechanism relies on overtraining improving per-sample quality, which is then amplified by repeated sampling. If even the overtrained model has
p_i ≈ 0for nearly all questions, thenpass@k ≈ 0regardless ofk, and no amount of joint optimization helps. In this regime, scaling model size (which Chinchilla recommends) is the only path forward, even if it means fewer inference samples. -
Prefer T2 scaling when you control both pretraining and deployment, allowing the joint optimization to be realized end-to-end. If pretraining and deployment are managed by separate teams with separate budgets (a common organizational pattern), the coupling
k = C_inf / (2N)requires coordination that may not be feasible—the pretraining team optimizes for single-pass loss underC_train, and the deployment team optimizeskunderC_infgiven whatever model they receive. In such split-incentive settings, the status quo sequential approach may be the only practically implementable one, even though it is jointly suboptimal.
