Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data

ArXiv: 2510.03264

🎯 Pitch

Introducing reasoning data early—during pretraining, not just fine-tuning—creates an irreversible 19% average accuracy gain that later-stage training cannot recover, and naively scaling up SFT data actually erases those benefits by 5%. The core insight is an asymmetric principle: pretraining craves diversity, while SFT demands quality, and models pretrained on diverse reasoning patterns compound their advantage through every subsequent training stage.

---

1. Executive Summary

This paper conducts the first systematic study of how reasoning data—varying in scale, diversity, and quality—affects LLM performance when introduced at different stages of training, using an 8B parameter hybrid Mamba-transformer model pretrained from scratch on 1 trillion tokens. The central contribution is the discovery of an asymmetric principle for optimal data allocation: pretraining benefits most from broad diversity in reasoning patterns (yielding an 11% average gain with diverse data over narrow high-quality data), while supervised fine-tuning is dominated by data quality (yielding a 15% average gain with high-quality reasoning traces over large-scale mixed-quality data). The paper demonstrates that front-loading reasoning data into pretraining creates a compounding advantage that post-training alone cannot recover—models pretrained with reasoning data achieve a 19% average gain on expert-level benchmarks after reinforcement learning, with the gap widening at each subsequent training stage rather than closing. Critically, the paper establishes that naive scaling of SFT data can be actively harmful, washing away pretraining benefits with a 5% average drop in math accuracy when doubling mixed-quality data, while a marginal 0.4% addition of high-quality reasoning-intensive examples consistently improves performance—underscoring that SFT is a phase of targeted refinement whose effectiveness is fundamentally constrained by the quality of the pretraining foundation.

2. Context and Motivation

The Core Gap: We Don't Know When to Introduce Reasoning Data

The fundamental question this paper addresses is deceptively simple: given a fixed budget of reasoning-intensive training data, should you spend it during pretraining or during supervised fine-tuning? This matters because the prevailing paradigm for building reasoning-capable LLMs treats these stages as largely independent—pretraining produces a generalist base model on broad web text, and post-training layers reasoning capabilities on top using curated instruction-formatted data. Yet the actual practice in frontier model development is increasingly opaque, with companies like OpenAI, Anthropic, and DeepMind incorporating reasoning-style data into mid-training or even late pretraining stages without publicly characterizing what they do, why, or what tradeoffs are involved.

This gap is significant for several practical and scientific reasons:

• Compute allocation decisions: Pretraining is orders of magnitude more expensive than SFT. If reasoning data introduced during pretraining yields compounding benefits that SFT cannot recover, then organizations making decisions about how to structure their training pipelines need to front-load investment in data curation earlier. Conversely, if pretraining reasoning data provides no durable advantage, the cheaper SFT-only approach is preferable.

• Scientific understanding of reasoning emergence: We don't fundamentally understand whether reasoning is a capability that must be "baked into" the model's core representations during the massive unsupervised phase, or whether it can be effectively "bolted on" as a surface-level skill after the fact. This has implications beyond engineering—it speaks to how language models represent and acquire complex cognitive capabilities.

• The opacity problem: As the authors note in the abstract, "due to the opaqueness of pretraining corpora in most frontier models, the effect of reasoning data introduced at different phases of pre- and/or post-training is relatively less reported in the scientific literature." The community lacks public, systematic evidence about what the major labs are actually doing, making it difficult for researchers outside those labs to build on or critique those approaches.

Conflicting Signals in Existing Literature

The paper is motivated by a genuine tension in the emerging literature. On one hand, several recent works have found that introducing reasoning supervision before SFT—during "mid-training" phases—yields substantial benefits:

• Wang et al. (2025) (Octothinker) introduce a mid-training phase continuing pretraining on a small but high-quality reasoning dataset before SFT and RLVR, reporting "substantial downstream gains, particularly in mathematics benchmarks" (Section 6). Yet their corpus is heavily math-centric, making it unclear whether the gains come from scale, complexity, or domain diversity.

• Essential AI et al. (2025) similarly explore introducing "long chain-of-thought data into intermediate checkpoints" during pretraining, suggesting that early reasoning exposure matters. But their intervention is limited to small amounts of data and doesn't systematically vary quality, diversity, or quantity.

• Gandhi et al. (2025) inject algorithmically generated "cognitive behavioral" reasoning traces during mid-training and show improvements after RL. This reinforces the promise of early reasoning supervision but is "restricted to small datasets and narrow tasks, leaving open questions about scalability, diversity, and phase-specific allocation" (Section 6).

On the other hand, the post-training literature has developed a sophisticated understanding of what makes reasoning SFT effective—but this understanding may not transfer to pretraining:

• Zhou et al. (2023) (LIMA) famously showed that small amounts of high-quality instruction data can be sufficient for alignment, suggesting quality matters more than quantity in post-training. The paper's findings (Section 5, Table 5) confirm this for SFT but show the opposite pattern for pretraining, where diversity and scale dominate.

• Prabhakar et al. (2025), Luo et al. (2025a), and Huan et al. (2025) have independently observed that reasoning-centric post-training yields strong gains on math but "the improvement on science is minimal." The current paper observes the opposite pattern after reasoning-rich pretraining (Section 4, Table 2): SFT amplifies pretraining gains most prominently in science domains, suggesting that early reasoning exposure builds domain-general logical representations that late-stage SFT cannot replicate.

The Catch-Up Hypothesis: An Unresolved Question

Perhaps the most pressing unresolved question is whether an intensive SFT phase can compensate for a weak pretraining foundation. This is what the paper calls the "catch-up hypothesis": can a model pretrained without reasoning data match the performance of a reasoning-aware counterpart by simply undergoing more aggressive post-training?

The answer matters enormously. If catch-up is possible, then the expensive, logistically complex task of curating reasoning data for pretraining is unnecessary—organizations can simply invest in better SFT recipes. If catch-up is not possible, then pretraining data strategy becomes the decisive factor in final model capability, and the entire pipeline must be designed with pretraining choices as the primary constraint.

Prior work provides no clear answer. Cheng et al. (2024) study "instruction pretraining" by converting raw text into short QA pairs during pretraining, and demonstrate gains over vanilla pretraining after instruction tuning. However, their setup does not target reasoning-intensive domains, and critically, they do not assess whether "these marginal pretraining gains persist once models undergo reasoning-heavy SFT and reinforcement learning" (Section 6). Their work establishes that pretraining data matters, but leaves open whether the advantage is durable under aggressive post-training.

Similarly, Liang et al. (2025) explore aligning instruction-tuning data with the pretraining distribution to improve SFT outcomes. While complementary in spirit, their intervention is applied only during SFT and does not address whether reasoning-specific supervision at the pretraining stage provides sustained benefits that survive multiple rounds of post-training.

The Overfitting Concern: Could Early Reasoning Data Hurt?

A countervailing hypothesis, grounded in the continual fine-tuning literature, is that introducing reasoning data early might cause the model to overfit or overspecialize, reducing its ability to generalize during subsequent training phases. Luo et al. (2025b) provide evidence of catastrophic forgetting during sequential fine-tuning, and Gupta et al. (2025) find that "overspecialization of reasoning during post-training can be detrimental." If the model internalizes the specific reasoning patterns in its pretraining data too rigidly, it might lose the flexibility needed to absorb new patterns during SFT, or perform well on tasks outside the pretraining reasoning distribution.

The paper directly confronts this concern (Section 1, third bullet; Section 4; Appendix C, Figure 2), and the results refute it: models pretrained with reasoning data not only avoid overfitting but actually amplify their advantage through each subsequent training stage. The SFT phase reveals latent benefits from high-quality pretraining data (the ℳ_{LMQ} model gains an additional 4.25% over ℳ_{LDQ} only after SFT, despite showing minimal immediate advantage at the pretraining stage—Table 4), and using the same reasoning data in both pretraining and SFT reinforces rather than overwrites learning (Figure 2). This challenges the prevailing assumption that data redundancy between stages is wasteful or harmful.

The Data Strategy Vacuum: No Principled Framework for Allocation

Prior to this work, there existed no systematic framework for deciding how to allocate reasoning data across training stages. The existing literature offered piecemeal guidance:

• For SFT: Use high-quality, long-chain-of-thought data; less is often more (Zhou et al., 2023; Zhao et al., 2024). Quality matters more than quantity.

• For mid-training: Inject small amounts of high-quality reasoning data before SFT; it seems to help, but the optimal composition, scale, and domain diversity are unknown (Wang et al., 2025; Gandhi et al., 2025).

• For full pretraining: Virtually no public guidance exists. The major labs don't disclose their pretraining data compositions for reasoning models, and the computational cost of end-to-end pretraining experiments has prevented academic investigation.

This vacuum means that organizations building reasoning models are operating largely on intuition. Should they invest in curating diverse reasoning data for pretraining, or save budget for higher-quality SFT? Should the same data appear in both stages or should the stages be complementary? What happens when reasoning data in pretraining is noisy—does it help (by providing broad exposure) or hurt (by teaching spurious patterns)? These questions have direct financial implications: pretraining data decisions are among the most expensive choices in the LLM pipeline, and getting them wrong wastes millions of dollars in compute.

How This Paper Positions Itself

The paper positions itself as the first systematic, large-scale analysis of reasoning data across the entire training pipeline (Section 6). It is not proposing a new method or architecture, but rather providing the empirical foundation for a principled data strategy. The key conceptual framework is the optimization problem stated in Equation 2 (Section 2):

$\max_{\mathcal{D}_{\text{res}}^{\text{PT}}, \mathcal{D}_{\text{res}}^{\text{SFT}}} \mathcal{P}(\mathcal{D}_{\text{res}}^{\text{PT}}, \mathcal{D}_{\text{res}}^{\text{SFT}}) \quad \text{s.t.} \quad \mathcal{B} = |\mathcal{D}_{\text{res}}^{\text{PT}}| + |\mathcal{D}_{\text{res}}^{\text{SFT}}|$

where $\mathcal{B}$ is the total budget of reasoning data available, and the goal is to maximize final downstream accuracy $\mathcal{P}$ after RLVR. This framing makes explicit what prior work left implicit: there is a tradeoff between allocating reasoning data to pretraining (where it shapes core representations but is expensive to incorporate) versus SFT (where it is cheaper to apply but may be less impactful).

The paper's contribution is to characterize this tradeoff empirically along three critical dimensions—diversity, quality, and quantity—across three training stages (pretraining, SFT, RLVR) using a fully crossed experimental design. The four pretrained models (ℳ_{base}, ℳ_{SHQ}, ℳ_{LDQ}, ℳ_{LMQ}) vary the reasoning data exposure during pretraining, and each is subsequently fine-tuned on multiple SFT data configurations to disentangle phase-specific effects. This design enables the paper to answer questions that prior work couldn't: whether catch-up is possible (Section 4, Table 4), whether overfitting is a concern (Section 4, Appendix C), whether diversity and quality have phase-dependent utility (Section 5, Tables 1 and 5), and whether pretraining data has latent effects unlocked by subsequent alignment (Section 5, Table 4).

The paper also explicitly connects to the reinforcement learning literature by carrying all experiments through to a GRPO-based RLVR stage (Section 2.3, Phase 3). This is critical because the ultimate metric of interest—final downstream accuracy on expert-level tasks—is measured after RL. A pretraining data choice that looks beneficial at the base model stage but washes out during RL would be practically irrelevant. By demonstrating that pretraining advantages compound through RL (Table 3: ℳ_{LMQ} + SFT + RL achieves 56.66% average vs. 37.92% for the baseline, with a 39.32% leap on AIME competition math), the paper establishes that early reasoning data decisions have enduring, amplified consequences that cascade through the entire training pipeline.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

This paper is a controlled experimental study, not a new method proposal—the "system" being built is a systematic experimental framework for comparing how reasoning data injected at different training stages shapes the final capabilities of an LLM. It solves the allocation problem of deciding when to introduce reasoning-intensive data into a multi-stage training pipeline by constructing a fully-crossed design where four models are pretrained with different reasoning data compositions, then each is fine-tuned under multiple SFT configurations, and finally all are pushed through reinforcement learning, with every combination evaluated on a broad set of downstream benchmarks.

3.2 Big-picture architecture (diagram in words)

The experimental infrastructure has five major components:

1. General Pretraining Corpus ($\mathcal{D}_{\text{base}}$) — a 6.2T-token dataset of high-quality Common Crawl, mathematics, and code sources (from NVIDIA, 2025b) that serves as the backbone for all pretraining runs, comprising 80% of the pretraining mix.

2. Reasoning Data Catalog ($\mathcal{D}_{\text{res}}$) — four distinct reasoning-focused datasets varying along three axes (diversity, quality, quantity) that are injected at controlled proportions into either the pretraining mix or the SFT stage.

3. Pretraining Pipeline — pretrains an 8B hybrid Mamba-transformer model from scratch for 1 trillion tokens using a 80/20 blend of $\mathcal{D}_{\text{base}}$ and a selected $\mathcal{D}_{\text{res}}$ variant, producing four distinct base models ($\mathcal{M}_{\text{base}}$, $\mathcal{M}_{\text{SHQ}}$, $\mathcal{M}_{\text{LDQ}}$, $\mathcal{M}_{\text{LMQ}}$).

4. Supervised Fine-Tuning (SFT) Pipeline — takes each of the four base models and fine-tunes them on one of several SFT data configurations, producing the crossed set of models used to disentangle pretraining effects from SFT effects.

5. Reinforcement Learning (RLVR) Pipeline — applies Group Relative Policy Optimization (GRPO) with verifiable rewards using the nemotron-crossthink dataset on top of the SFT models, producing the final models on which the ultimate downstream accuracy $\mathcal{P}$ is measured.

Information flows sequentially: pretraining data blending → pretraining for 1T tokens → SFT on 4.8M reasoning samples → GRPO-RLVR for 1 epoch → evaluation on reasoning benchmarks (AIME, MATH-500, GSM8K, MMLU, MMLU-Pro, GPQA-Diamond, LiveCodeBench, IFEval). The key scientific output is not a single model but a matrix of accuracy measurements that reveals how pretraining data choices interact with SFT data choices across multiple training stages.

3.3 Roadmap for the deep dive

• First, the formal optimization framework (Equations 1 and 2) that defines "optimal allocation" as maximizing final downstream accuracy under a fixed reasoning data budget, since this is the conceptual lens through which all experiments are designed.
• Second, the four reasoning datasets ($\mathcal{D}_{\text{SHQ}}$, $\mathcal{D}_{\text{LDQ}}$, $\mathcal{D}_{\text{LMQ}}$, $\mathcal{D}_{\text{ALF}}$), their construction, and what dimension of "data quality," "data diversity," or "data complexity" each is intended to isolate, because these datasets are the independent variables in every experiment.
• Third, the pretraining pipeline—model architecture, training hyperparameters, how reasoning data is blended into $\mathcal{D}_{\text{base}}$, and what the four base models represent—since this is where the central intervention (front-loading vs. not) occurs.
• Fourth, the SFT pipeline—how each base model is fine-tuned, what data configurations are swept, and how the crossed design enables testing the catch-up hypothesis, the overfitting hypothesis, and the asymmetric allocation principle.
• Fifth, the RLVR pipeline—GRPO configuration, verifiable rewards, and why running through RL matters for claims about compounding advantage.
• Sixth, the evaluation rubric—what is measured, on which benchmarks, at which stages, and how metrics are aggregated into domain-level averages—since the paper's conclusions rest on careful metric definitions.

3.4 Detailed, sentence-based technical breakdown

This is primarily an empirical analysis paper whose core idea is that the optimal allocation of reasoning data between pretraining and SFT follows an asymmetric principle—diversity and scale dominate in pretraining, while quality dominates in SFT—and that front-loading reasoning data into pretraining creates a compounding advantage that later stages cannot recover.

---

The Formal Optimization Framing

The paper frames its investigation as solving an explicit optimization problem: given a fixed total budget of reasoning data $\mathcal{B}$, how should that budget be split between pretraining and SFT to maximize final model accuracy on downstream reasoning tasks?

Let $\mathcal{D}_{\text{res}}^{\text{PT}}$ denote the reasoning data introduced during pretraining, and $\mathcal{D}_{\text{res}}^{\text{SFT}}$ denote the reasoning data used during supervised fine-tuning. The objective is:

$\mathcal{P}(\mathcal{D}_{\text{res}}^{\text{PT}}, \mathcal{D}_{\text{res}}^{\text{SFT}}) = \mathbb{E}_{t \sim \mathcal{T}} \left[ \text{Acc}\left( f_{\theta_{\text{SFT}}}(t) \right) \right]$

where $\mathcal{T}$ is a set of downstream reasoning tasks spanning math, science, code, and general reasoning, $\theta_{\text{SFT}}$ are the model parameters after both pretraining and SFT (and, implicitly, RLVR—the paper measures final accuracy at the end of the full pipeline), and $\text{Acc}(\cdot)$ is the task-specific accuracy metric.

What it computes: the expectation (average) of the model's accuracy across all tasks in the evaluation suite $\mathcal{T}$, using the model that results from applying a specific pretraining data recipe and a specific SFT data recipe. The function $\mathcal{P}$ is the central quantity the paper aims to characterize—it maps a pair of data allocation decisions to a single scalar representing final model capability.

Why this form: the expectation over $\mathcal{T}$ ensures the metric captures general reasoning capability rather than overfitting to any single domain. The explicit conditioning on both $\mathcal{D}_{\text{res}}^{\text{PT}}$ and $\mathcal{D}_{\text{res}}^{\text{SFT}}$ makes clear that the goal is understanding the interaction between the two stages, not optimizing either in isolation. The model parameters $\theta_{\text{SFT}}$ are themselves the result of a two-stage training process:

$\theta_{\text{PT}} = \arg\min_{\theta} \mathbb{E}_{(x, y) \sim \mathcal{D}_{\text{base}} \cup \mathcal{D}_{\text{res}}^{\text{PT}}} \mathcal{L}_{\text{LM}}(f_\theta(x), y)$

$\theta_{\text{SFT}} = \arg\min_{\theta} \mathbb{E}_{(x, y) \sim \mathcal{D}_{\text{res}}^{\text{SFT}}} \mathcal{L}_{\text{SFT}}(f_\theta(x), y)$

where $\mathcal{L}_{\text{LM}}$ is the standard autoregressive language modeling loss (next-token prediction) used in pretraining, and $\mathcal{L}_{\text{SFT}}$ is the supervised fine-tuning loss applied to the reasoning-formatted data. The key structural insight is that $\theta_{\text{PT}}$ is the initialization for $\theta_{\text{SFT}}$, meaning pretraining data choices influence SFT outcomes through the path of the optimization trajectory, not just through final parameter values.

The budget-constrained optimization problem is then:

$\max_{\mathcal{D}_{\text{res}}^{\text{PT}}, \mathcal{D}_{\text{res}}^{\text{SFT}}} \mathcal{P}(\mathcal{D}_{\text{res}}^{\text{PT}}, \mathcal{D}_{\text{res}}^{\text{SFT}}) \quad \text{s.t.} \quad \mathcal{B} = |\mathcal{D}_{\text{res}}^{\text{PT}}| + |\mathcal{D}_{\text{res}}^{\text{SFT}}|$

What this constrains: the total amount of reasoning data (measured in tokens or examples, denoted by $|\cdot|$) used across both stages cannot exceed some fixed budget $\mathcal{B}$. This forces a tradeoff: every additional token of reasoning data allocated to pretraining must come at the expense of reasoning data available for SFT, and vice versa.

Why this form: without the budget constraint, the trivial "solution" would be to use maximum reasoning data in both stages. The constraint makes the allocation problem real—it forces the experimenter to confront the marginal utility of reasoning data in each phase. This framing is analogous to compute-optimal scaling laws for pretraining (where the budget is FLOPs and the decision is model size vs. data quantity), but applied to the qualitatively different question of phase-specific data allocation. The constraint also mirrors practical deployment: organizations have finite resources to curate reasoning data, and must decide how to deploy it across training stages.

The paper does not solve this optimization analytically but instead maps the function $\mathcal{P}$ empirically through a grid of experiments: four pretraining data configurations crossed with multiple SFT data configurations, with each combination evaluated after RLVR. The resulting performance matrix reveals the approximate shape of $\mathcal{P}$ and yields qualitative principles (diversity in pretraining, quality in SFT) that approximate the optimum.

---

The Reasoning Datasets ($\mathcal{D}_{\text{res}}$): Independent Variables

The paper curates four distinct reasoning-focused datasets that serve as the controlled independent variables across all experiments. Each dataset is designed to isolate a specific dimension of variation—scale, diversity, quality, or reasoning depth—while keeping other properties as constant as possible.

Large-Scale, Diverse Data ($\mathcal{D}_{\text{LDQ}}$). This is the "quantity-over-quality" dataset, drawn from the Nemotron-Pretraining-SFT-v1 dataset (NVIDIA, 2025b). It contains 336 billion tokens with a domain composition of approximately 56% math, 17% code, and 27% science and general-purpose reasoning. The dataset includes tasks ranging from simple Q&A to multi-turn dialogues.

What it isolates: the effect of broad coverage and sheer scale at the expense of carefully curated quality. The data is described as having "heterogeneous quality and reasoning depth, reflecting large-scale real-world availability." This means the dataset contains both high-quality examples with detailed reasoning traces and lower-quality examples with short, noisy, or superficial responses. By using this dataset, the paper can test whether exposing the model to a wide variety of reasoning patterns—even if many are imperfect—builds a robust foundation.

Why 336B tokens: this is a massive scale relative to typical SFT datasets (which are measured in millions of examples, not billions of tokens). The contrast with the 1.2M-example $\mathcal{D}_{\text{SHQ}}$ is intentional—it represents the extreme of sacrificing quality for quantity.

Small-Scale, High-Quality Data ($\mathcal{D}_{\text{SHQ}}$). Drawn from the dataset of Guha et al. (2025), this corpus contains 1.2 million carefully curated examples with a domain composition of 71% math, 21% code, and 8% science. The dataset is characterized by "long chain-of-thought traces from strong teacher models" with an average answer length exceeding 10,000 tokens.

What it isolates: the effect of curated quality and reasoning depth in the absence of broad diversity. The dataset is smaller (by token count, dramatically smaller than $\mathcal{D}_{\text{LDQ}}$), less diverse in domain coverage, but each example provides a rich, detailed reasoning trace generated by a capable teacher model.

Why long CoT: the paper hypothesizes that longer reasoning chains "serve as richer supervisory signals, encouraging models to internalize structured multi-step inference rather than surface-level heuristics" (Appendix C). The 10k+ token average length means the model sees the intermediate reasoning steps, not just final answers.

Large-Scale, Mixed-Quality Data ($\mathcal{D}_{\text{LMQ}}$). This is constructed as the straightforward set union: $\mathcal{D}_{\text{LMQ}} = \mathcal{D}_{\text{LDQ}} \cup \mathcal{D}_{\text{SHQ}}$. It preserves the large-scale coverage of $\mathcal{D}_{\text{LDQ}}$ while injecting a fraction of curated, high-quality reasoning traces from $\mathcal{D}_{\text{SHQ}}$.

What it isolates: the marginal benefit of adding quality to quantity at the pretraining stage. Since $\mathcal{D}_{\text{LMQ}}$ contains everything in $\mathcal{D}_{\text{LDQ}}$ plus the high-quality examples from $\mathcal{D}_{\text{SHQ}}$, any performance difference between $\mathcal{M}_{\text{LMQ}}$ and $\mathcal{M}_{\text{LDQ}}$ can be attributed to the addition of high-quality data. This tests whether quality matters on top of diversity during pretraining, or whether diversity alone saturates the benefit.

Answer-Length Filtered Data ($\mathcal{D}_{\text{ALF}}$). This is constructed as a subset of $\mathcal{D}_{\text{LDQ}}$ (the large, diverse, mixed-quality dataset) by retaining only examples where the answer length exceeds 4,096 tokens. The filtering is based on the principle that longer responses "often correspond to more complex CoT reasoning" (Section 2.2).

What it isolates: the effect of reasoning depth/complexity independent of other quality dimensions. $\mathcal{D}_{\text{ALF}}$ is drawn from the same underlying distribution as $\mathcal{D}_{\text{LDQ}}$ (so it shares the same sources, domains, and generation processes) but filters for examples with longer, presumably more detailed reasoning traces. The dataset is approximately 50 times smaller than $\mathcal{D}_{\text{LDQ}}$—representing only ~2% of the original corpus—but is "highly skewed toward domains with inherently deeper reasoning (75% math)."

Why length as a proxy for quality: this is a deliberate methodological choice that enables a clean ablation. Rather than relying on a separate quality classifier (which would introduce its own biases and training data requirements), the paper uses a simple, reproducible heuristic. The 4,096-token threshold is chosen empirically; longer responses in $\mathcal{D}_{\text{LDQ}}$ disproportionately come from math problems requiring multi-step derivations.

A variant $\mathcal{D}'_{\text{ALF}}$ is also used, constructed by scaling $\mathcal{D}_{\text{ALF}}$ with $\mathcal{D}_{\text{SHQ}}$—adding only "a marginal increase in dataset size (0.4% more samples)" while concentrating the additional data in high-quality reasoning traces.

Design rationale for the four-dataset structure. The paper's core question—whether diversity matters more in pretraining or SFT—requires datasets that vary independently along the diversity and quality axes. $\mathcal{D}_{\text{LDQ}}$ is high-diversity, mixed-quality. $\mathcal{D}_{\text{SHQ}}$ is low-diversity, high-quality. $\mathcal{D}_{\text{LMQ}}$ is high-diversity, mixed-quality with a quality spike. $\mathcal{D}_{\text{ALF}}$ is moderate-diversity, filtered for reasoning depth. By testing all combinations of pretraining with these datasets and SFT with these datasets, the paper can observe the performance surface and infer whether the optimal strategy is symmetric (use the same type of data in both stages) or asymmetric (use different types).

---

Phase 1: Pretraining Pipeline

The pretraining phase is where the paper's central intervention occurs: the controlled injection of reasoning data into the base model's foundational training. All models are pretrained from scratch—not fine-tuned from an existing checkpoint—to ensure that differences in final capability are attributable to the pretraining data composition rather than inherited from a shared initialization.

Model Architecture. The base architecture is an 8B-parameter hybrid transformer with a mixture of Mamba 2 layers (Dao & Gu, 2024), standard self-attention layers, and feed-forward network (FFN) layers. This architecture, from NVIDIA (2025a), is chosen to strike "a balance between computational feasibility and the capacity to learn complex reasoning patterns" (Section 2.1).

Why 8B parameters: this size is large enough to exhibit non-trivial reasoning behavior (enabling meaningful evaluation on benchmarks like AIME and GPQA-Diamond) but small enough that pretraining four variants from scratch for 1T tokens is computationally tractable within the paper's resource constraints. A 70B or 405B model would make the fully-crossed experimental design prohibitively expensive.

Why hybrid Mamba-attention: the paper does not argue for this architectural choice on scientific grounds—it is inherited from the NVIDIA Nemotron family. The architectural details are provided for reproducibility but are not the focus of the research question. The key requirement is that all four model variants share identical architecture, initialization, and training hyperparameters, differing only in pretraining data composition.

Pretraining Data Blending. For all four models, the pretraining mixture consists of 80% $\mathcal{D}_{\text{base}}$ (the general-purpose 6.2T-token corpus from NVIDIA, 2025b) and 20% of the selected reasoning dataset $\mathcal{D}_{\text{res}}$. The base corpus "provides broad coverage of languages and technical domains, serving as the backbone of all pretraining experiments" (Section 2.2).

The 80/20 split is a design decision: it ensures that the bulk of pretraining tokens come from a shared, consistent distribution, while the 20% reasoning injection is large enough to have a measurable effect but small enough that the model does not become a pure reasoning specialist. If reasoning data comprised 50% or more of the mix, the pretraining would effectively be a different distribution entirely, making it harder to attribute effects specifically to the reasoning data's early introduction versus simply more reasoning tokens overall.

The four base models. The paper constructs four distinct models by varying $\mathcal{D}_{\text{res}}^{\text{PT}}$:

• $\mathcal{M}_{\text{base}}$: pretrained with no reasoning data—only $\mathcal{D}_{\text{base}}$. This is the control condition, representing the conventional paradigm where reasoning is introduced only during post-training.
• $\mathcal{M}_{\text{SHQ}}$: pretrained with $\mathcal{D}_{\text{base}}$ + $\mathcal{D}_{\text{SHQ}}$ (small-scale, high-quality, low-diversity). This tests whether curated quality matters at the pretraining stage in the absence of broad diversity.
• $\mathcal{M}_{\text{LDQ}}$: pretrained with $\mathcal{D}_{\text{base}}$ + $\mathcal{D}_{\text{LDQ}}$ (large-scale, diverse, mixed-quality). This tests whether sheer scale and domain breadth matter at the pretraining stage, even when individual examples may be noisy.
• $\mathcal{M}_{\text{LMQ}}$: pretrained with $\mathcal{D}_{\text{base}}$ + $\mathcal{D}_{\text{LMQ}}$ (the union of diverse and high-quality data). This tests whether adding quality to diversity yields further gains, or whether diversity alone saturates the benefit.

The notation $\mathcal{M}_{\text{res}}$ in subsequent results denotes the aggregate average across the three reasoning-pretrained models ($\mathcal{M}_{\text{SHQ}}$, $\mathcal{M}_{\text{LDQ}}$, $\mathcal{M}_{\text{LMQ}}$), used when the paper wants to summarize "the effect of having any reasoning data in pretraining" without distinguishing between data types.

Pretraining hyperparameters. All four models are pretrained from scratch for 1 trillion tokens using 512 H100 80GB SXM5 GPUs. The optimizer is AdamW (Loshchilov & Hutter, 2019) with $\beta_1 = 0.9$, $\beta_2 = 0.95$, and weight decay of 0.1. The learning rate follows a schedule with maximum value $3 \times 10^{-4}$ and minimum value $3 \times 10^{-6}$ (a cosine decay from max to min over the 1T-token run). The batch size is 6 million tokens, and the context length is 8,192 tokens. An 8-way tensor and pipeline parallelism strategy distributes the model across the 512 GPUs.

Why 1 trillion tokens: this is a substantial but not extreme pretraining budget for an 8B model. It ensures the model is well-trained and has converged reasonably, but it is not so large that the cost of running four variants becomes prohibitive. The choice also aligns with practical considerations: many deployed models in this size class are trained on 1T–2T tokens, so the experimental setup mirrors real-world conditions.

Immediate evaluation at the base model stage. After pretraining, each of the four models is evaluated on a set of general-purpose benchmarks using LM Eval Harness (Gao et al., 2024). This evaluation is distinct from the later SFT evaluation suite—it focuses on general capabilities rather than reasoning depth, including:

• General Purpose Reasoning (gprPT avg): ARC-Challenge (0-shot), HellaSwag (0-shot), WinoGrande (0-shot), RACE (0-shot). These measure commonsense reasoning, reading comprehension, and linguistic understanding.
• Math Reasoning (mathPT avg): GSM8K (8-shot) and MATH-500 (4-shot). These measure mathematical problem-solving ability in the few-shot prompted setting, testing whether the base model can perform math without SFT.
• Science Reasoning (sciencePT avg): MMLU (5-shot) and MMLU-Pro (5-shot). These test domain knowledge and reasoning across professional and academic subjects.
• Code Reasoning (codePT avg): HumanEval and MBPP (both 0-shot, with EvalPlus variants and sanitized generations). Code metrics report avg@32 (average pass@1 across 32 generations per prompt), which provides a more stable estimate than single-sample pass@1.

The base model evaluation establishes the immediate, pre-SFT effect of reasoning data in pretraining, answering: does reasoning data help even before the model is explicitly fine-tuned for reasoning tasks?

---

Phase 2: Supervised Fine-Tuning Pipeline

The SFT phase is where the paper's crossed experimental design becomes fully realized. Each of the four base models is fine-tuned on one of several SFT data configurations, producing a matrix of models that enables isolating pretraining effects (by comparing SFT-matched models with different pretraining) from SFT effects (by comparing pretraining-matched models with different SFT data).

SFT data configurations. The paper sweeps several SFT data recipes, each using 4.8 million reasoning samples (approximately 4.8M examples) drawn from the reasoning datasets:

• SFT[$\mathcal{D}_{\text{SHQ}}$]: fine-tuning on the small, high-quality dataset. This is the primary SFT configuration used for most comparisons, as it represents the "best practice" of using curated, long-CoT data for post-training.
• SFT[$\mathcal{D}_{\text{LDQ}}$]: fine-tuning on the large-scale, diverse, mixed-quality dataset. This tests whether broad data diversity is beneficial at the SFT stage, or whether it actually degrades performance relative to curated data.
• SFT[$\mathcal{D}_{\text{LMQ}}$]: fine-tuning on the combined mixed-quality corpus.
• SFT[$\mathcal{D}_{\text{ALF}}$] and SFT[$\mathcal{D}'_{\text{ALF}}$]: fine-tuning on the length-filtered subsets, testing the effect of reasoning depth.
• SFT[2× $\mathcal{D}_{\text{LDQ}}$]: double the amount of the diverse, mixed-quality data (9.6M samples), testing whether simply scaling up SFT data quantity can compensate for the lack of reasoning pretraining.
• SFT[2×] for $\mathcal{M}_{\text{base}}$: the baseline model fine-tuned with twice the SFT data, directly testing the "catch-up hypothesis"—can more aggressive SFT make the baseline match models that had reasoning data in pretraining?

SFT hyperparameters. All SFT runs use AdamW with $(\beta_1, \beta_2) = (0.9, 0.95)$, weight decay 0.01, warmup ratio 0.05, learning rate $5 \times 10^{-6}$, batch size 512, and context length 32,768 tokens. The learning rate is an order of magnitude smaller than pretraining ($5 \times 10^{-6}$ vs. $3 \times 10^{-4}$), reflecting the standard practice of using gentler updates during fine-tuning to avoid catastrophic forgetting of pretrained knowledge.

Why 4.8M samples: this is a substantial but not extreme SFT budget. Prior work (Zhou et al., 2023) has shown that even 1K high-quality examples can be sufficient for alignment, but the paper's goal is to study the interaction between pretraining and SFT, which requires a large enough SFT phase that the model actually undergoes meaningful adaptation. 4.8M samples ensures the SFT signal is strong enough to observe differential effects based on pretraining.

Why 32k context length: the $\mathcal{D}_{\text{SHQ}}$ dataset contains examples with average answer lengths exceeding 10K tokens, and some individual examples may exceed standard 4K or 8K context windows. The 32K context ensures that these long reasoning traces are not truncated, allowing the model to learn from complete chain-of-thought sequences.

Testing the catch-up hypothesis. The most critical SFT experiment is the comparison of $\mathcal{M}_{\text{base}} + \text{SFT}[2\times]$ against $\mathcal{M}_{\text{SHQ}} + \text{SFT}$, $\mathcal{M}_{\text{LDQ}} + \text{SFT}$, and $\mathcal{M}_{\text{LMQ}} + \text{SFT}$. If the catch-up hypothesis were true, doubling the SFT data for the baseline would allow it to match or exceed the reasoning-pretrained models, which receive only the standard amount of SFT data. The paper reports (Table 4) that doubling SFT improves the baseline from 26.62% to 34.01% average, but this enhanced baseline still trails $\mathcal{M}_{\text{SHQ}} + \text{SFT}$ at 37.33%—and falls substantially short of $\mathcal{M}_{\text{LDQ}} + \text{SFT}$ at 46.70% and $\mathcal{M}_{\text{LMQ}} + \text{SFT}$ at 50.95%. The catch-up hypothesis is thus refuted.

Testing the overfitting hypothesis. The SFT design also tests whether pretraining with reasoning data causes overfitting that reduces the model's ability to absorb SFT data. If overfitting were occurring, we would expect the reasoning-pretrained models to show smaller gains from SFT than the baseline, or even to degrade. The paper reports the opposite (Table 2): the reasoning-pretrained group ($\mathcal{M}_{\text{res}} + \text{SFT}$) achieves 35.92% average versus 26.62% for the baseline—a 9.3 percentage point gap that is larger than the 8.35 point gap at the base model stage (61.05% vs. 52.70% from Table 1). The pretraining advantage widens, not narrows, through SFT.

Testing the asymmetric allocation principle. The fully-crossed SFT design—where reasoning-pretrained models are fine-tuned on both high-quality and mixed-quality SFT data—directly tests whether the optimal data strategy differs by stage. Table 5 shows that $\mathcal{M}_{\text{res}} + \text{SFT}[\mathcal{D}_{\text{SHQ}}]$ achieves 44.99% average, while the same models fine-tuned on $\mathcal{D}_{\text{LDQ}}$ achieve only 31.54%—a 13.45 point gap. Meanwhile, at the pretraining stage, $\mathcal{M}_{\text{LDQ}}$ (diverse, mixed-quality) dramatically outperforms $\mathcal{M}_{\text{SHQ}}$ (narrow, high-quality) by 9.09 points (64.09% vs. 54.98% from Table 1). The patterns are opposite: diversity wins in pretraining, quality wins in SFT.

SFT evaluation suite. Unlike the base model evaluation, which focused on general capabilities, the SFT evaluation suite targets reasoning-centric benchmarks using NeMo-Skills:

• Math Reasoning (mathSFT avg): GSM8K, MATH-500, AIME-2024, AIME-2025. The inclusion of AIME (competition math) is critical—these problems are substantially harder than GSM8K and MATH-500, testing whether pretraining reasoning data provides benefits on expert-level tasks.
• Science Reasoning (scienceSFT avg): MMLU, MMLU-Pro, GPQA-Diamond. GPQA-Diamond is a graduate-level Q&A benchmark designed to be resistant to memorization.
• Code Reasoning (codeSFT avg): LiveCodeBench, which tests complex code generation and reasoning.
• Instruction Following (insSFT avg): IFEval, which tests the model's ability to follow explicit instructions—a capability that may trade off against reasoning depth (as shown in Appendix C, Table 9, where length-filtered data improves reasoning but slightly degrades instruction following).

Metrics are reported as Pass@1 (the fraction of trials where the first generated answer is correct), averaged over 16 runs for AIME-2024 and AIME-2025 (which have high variance due to few problems), and 4 runs for all other benchmarks.

---

Phase 3: Reinforcement Learning with Verifiable Rewards (RLVR)

The RLVR phase serves two purposes: it tests whether the benefits of reasoning-rich pretraining survive and compound through an additional training stage that fundamentally changes the model's generation behavior, and it produces the final models on which the paper's headline results (the 19% gain on expert-level benchmarks) are measured.

Why RLVR matters. SFT teaches the model to imitate reasoning patterns from the training data, but it does not directly optimize for correct answers. RLVR with verifiable rewards (e.g., checking whether the final answer to a math problem matches the ground truth) provides a training signal that directly rewards correctness, potentially amplifying the latent reasoning capabilities built during pretraining. If pretraining reasoning data truly builds better internal representations, those representations should enable more effective RLVR—the model should be able to explore more productively and converge to higher-accuracy policies.

RL Algorithm. The paper uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024), implemented via the veRL framework. GRPO is a variant of PPO designed for reasoning tasks; it groups rollouts from the same prompt and uses relative advantages within each group, which reduces variance compared to population-wide advantage estimation.

RL Configuration. The RL phase runs for one epoch on the nemotron-crossthink dataset (Akter et al., 2025), which the paper describes as having "shown to be effective to enhance reasoning across diverse domains" (Section 2.3). The learning rate is constant at $1 \times 10^{-6}$ (an order of magnitude lower than the SFT learning rate), batch size is 128, PPO mini-batch size is 128, and context length is 8,192 tokens. Each step samples 128 prompts with 8 rollouts per prompt (1,024 total generations per step), using temperature = 1.0 and top-p = 1.0 (no sampling restrictions beyond temperature). A KL penalty coefficient of 0.001 controls the divergence from the SFT policy—small enough to allow substantial policy improvement but large enough to prevent the model from collapsing to degenerate strategies.

Why 8 rollouts per prompt: this provides the group for GRPO's relative advantage computation. With 8 rollouts per prompt, the algorithm can estimate the relative quality of each generation by comparing it to the average of the other 7 rollouts for the same prompt, which is more meaningful than comparing to rollouts from different prompts whose difficulty may differ.

Why one epoch: the paper's focus is on the comparative effect of pretraining choices, not on squeezing maximum performance through extensive RL. One epoch provides enough signal to observe whether pretraining advantages compound (they do) without introducing confounding factors from differential RL convergence rates across models.

RL evaluation. The final models are evaluated on the same reasoning benchmarks as in the SFT phase: AIME-2024, AIME-2025, MATH-500, GSM8K, MMLU, MMLU-Pro, GPQA-Diamond, and LiveCodeBench. The RL evaluation uses the same Pass@1 metric and the same averaging (16 runs for AIME, 4 runs for others). The results in Table 3 show the headline finding: $\mathcal{M}_{\text{LMQ}} + \text{SFT} + \text{RL}$ achieves 56.66% average accuracy versus 37.92% for $\mathcal{M}_{\text{base}} + \text{SFT} + \text{RL}$—an 18.74 percentage point gap, with AIME-24 showing a 32.92 point gap (45.21% vs. 12.29%) and AIME-25 showing a 17.92 point gap (33.96% vs. 16.04%).

The compounding dynamic. The paper argues that the RL advantage is not simply additive to the SFT advantage but represents a true compounding effect: the reasoning-rich pretraining enables the model to explore more productively during RL, generating rollouts that are more likely to contain correct reasoning paths, which in turn provides better training signal, which further improves the policy. This is why the gap grows at each stage—from 8.35 points at base model, to 9.30 points at SFT, to 18.74 points after RL. Each stage amplifies the advantage established in the previous stage.

---

Evaluation Infrastructure and Metric Aggregation

The paper's conclusions depend on careful metric definitions and aggregation across diverse benchmarks. The evaluation framework is designed to measure domain-level reasoning capability rather than performance on any single benchmark.

Domain-level averages. For each training stage, the paper aggregates individual benchmark accuracies into domain-level averages:

• mathPT avg and mathSFT avg: averaged over the math benchmarks (base: GSM8K, MATH-500; SFT/RL: adds AIME-24, AIME-25). Individual benchmarks are equally weighted.
• sciencePT avg and scienceSFT avg: averaged over science benchmarks (base: MMLU, MMLU-Pro; SFT/RL: adds GPQA-Diamond).
• codePT avg and codeSFT avg: base evaluation averages HumanEval, HumanEvalPlus, MBPP, and MBPP-sanitized; SFT/RL evaluation uses LiveCodeBench alone.
• gprPT avg (base only): averaged over ARC-C, RACE, WinoGrande, HellaSwag.
• insSFT avg (SFT/RL only): IFEval.

Overall average. The final scalar metric used throughout the paper for high-level comparisons is the unweighted average of the domain-level averages, referred to as "Average" in all tables. For the base model: (mathPT avg + sciencePT avg + codePT avg + gprPT avg) / 4. For SFT/RL: (mathSFT avg + scienceSFT avg + codeSFT avg + insSFT avg) / 4.

Why unweighted average: this treats each domain as equally important, preventing any single benchmark (e.g., a particular math dataset with many more evaluation examples than a science dataset) from dominating the metric. The tradeoff is that domains with fewer benchmarks (like code at the SFT stage, where only LiveCodeBench is used) have their individual benchmark's variance less smoothed out.

Base vs. SFT/RL evaluation suites differ. The base model evaluation uses general-purpose benchmarks (ARC-C, HellaSwag, WinoGrande, RACE, HumanEval, MBPP) to measure broad capability, while the SFT/RL evaluation shifts to reasoning-intensive benchmarks (AIME, GPQA-Diamond, LiveCodeBench). This reflects the different scientific questions at each stage: the base model evaluation asks "does reasoning pretraining harm general capabilities?", while the SFT/RL evaluation asks "does reasoning pretraining amplify specialized reasoning ability?" The gprPT avg domain (general purpose reasoning) disappears after the base model stage, and insSFT avg (instruction following) appears only in SFT/RL.

$\mathcal{M}_{\text{res}}$ notation. When the paper reports results for $\mathcal{M}_{\text{res}}$ (e.g., "$\mathcal{M}_{\text{res}} + \text{SFT}$ achieves 35.92%"), this denotes the arithmetic mean across the three reasoning-pretrained variants ($\mathcal{M}_{\text{SHQ}}$, $\mathcal{M}_{\text{LDQ}}$, $\mathcal{M}_{\text{LMQ}}$). This aggregation is used when the goal is to summarize "the average effect of having any reasoning data in pretraining" without distinguishing between data types. The individual model results are always available in the detailed tables (e.g., Table 1 breaks down each of the four base models on every benchmark, and Table 8 in the appendix does the same for SFT models).

Pass@1 with multiple runs. To reduce variance in small benchmarks, the paper averages Pass@1 over multiple evaluation runs: 16 runs for AIME-2024 and AIME-2025 (each with only 30 problems), and 4 runs for larger benchmarks (MATH-500 has 500 problems, GSM8K has ~1,300 test problems, MMLU has ~14,000 questions). The multi-run averaging provides more stable estimates, particularly for AIME where a model might get 3/30 correct in one run and 7/30 in another simply due to sampling variance.

Code evaluation detail. For the base model code evaluation, the paper uses avg@32 (average pass@1 across 32 independent generations per prompt) rather than single-sample pass@1. This is motivated by the high variance in code generation—a single sample may fail due to a minor syntax error even when the model understands the algorithm, and avg@32 provides a more accurate measure of the model's true code generation capability. The EvalPlus variants (Liu et al., 2023) extend the original test suites with additional test cases to catch false positives, and "sanitized" generations refers to post-processing that normalizes formatting before evaluation.

4. Key Insights and Innovations

Innovation 1: The Asymmetric Allocation Principle — A Phase-Dependent Data Strategy

The paper's most fundamental intellectual contribution is not a method or a metric, but a principle: optimal data strategy is fundamentally asymmetric across training phases. Prior to this work, the prevailing intuition—where it existed at all—treated data strategy as phase-agnostic. The literature on post-training had converged on the view that data quality is paramount, famously crystallized in LIMA's demonstration that a mere 1,000 high-quality examples can suffice for effective alignment (Zhou et al., 2023). Meanwhile, the pretraining literature had long operated under the assumption that scale and diversity are what matter, but this was a claim about general web text, not about reasoning data specifically, and it was never tested in a phase-comparative framework where the same data characteristics were evaluated in both stages under controlled conditions.

The paper upends the implicit assumption of symmetry by demonstrating that the answer to "what data is best?" depends entirely on when you ask. In pretraining, the model pretrained on large-scale, diverse but mixed-quality data (ℳ_{LDQ}) achieves a 9.09 percentage point gain over the model pretrained on small-scale, high-quality but narrow data (ℳ_{SHQ})—64.09% vs. 54.98% average base model accuracy (Table 1). Diversity and scale dominate. But at the SFT stage, the pattern inverts: fine-tuning reasoning-pretrained models on the large, diverse corpus 𝒟_{LDQ} yields only 31.54% average, while fine-tuning the same models on the small, high-quality corpus 𝒟_{SHQ} yields 44.99%—a 13.45 point gap in the opposite direction (Table 5). Quality dominates.

What makes this innovative is that it resolves a latent tension in the literature without ever being proposed as a resolution. Why did LIMA find that quality is all that matters, while pretraining practitioners obsess over data filtering at the trillion-token scale? Because they were operating in different phases. The paper doesn't merely report that "diversity helps in pretraining" (which would be an unsurprising confirmation); it demonstrates that the marginal utility of diversity flips sign between phases, making diversity actively harmful during SFT while being essential during pretraining. This is a phase interaction effect, not a main effect, and it could only be discovered through the fully-crossed experimental design.

The significance extends beyond empirical findings to a conceptual reframing of how the field should think about data strategy. Rather than asking "what is the best reasoning dataset?", the question becomes "what is the best reasoning dataset for this phase?" This shifts the optimization problem from a single-dimensional search over data characteristics to a multi-stage allocation problem where data that is suboptimal for one purpose may be optimal for another. It is a fundamentally different way of thinking about the training pipeline—not as a sequence of independent stages each optimizing locally, but as an integrated system where stage-specific data choices interact to determine final capability.

The evidence is anchored in the contrasting patterns of Table 1 and Table 5, but the insight itself is the asymmetry, not any single number.

---

Innovation 2: The Latent Effect — Pretraining Data as "Sleeping" Capability Activated by SFT

The paper's second conceptual innovation is the discovery that high-quality pretraining data can have latent effects—benefits that are invisible or negligible at the base model stage but are "unlocked" by subsequent alignment. This challenges the standard evaluation paradigm in which pretraining data choices are judged by their immediate impact on base model benchmarks. If the community had evaluated the ℳ_{LMQ} model only at the base model stage, it would have concluded that adding high-quality data to an already-diverse pretraining mix provides minimal marginal benefit: ℳ_{LMQ} achieves 64.07% average versus ℳ_{LDQ}'s 64.09%—a difference of -0.02 percentage points, essentially zero (Table 1). The natural conclusion would be that once you have sufficient diversity, quality doesn't add value in pretraining.

But this conclusion is wrong—or at least, it misses what's actually happening inside the model. After SFT, ℳ_{LMQ} + SFT achieves 50.95% average, while ℳ_{LDQ} + SFT achieves 46.70%—a 4.25 percentage point gap that emerges during alignment (Table 4). The high-quality data didn't improve the model's immediate performance; it installed a latent capacity that only became measurable after the model was fine-tuned to express it.

This is a genuinely novel diagnostic concept. The field has long understood that pretraining builds representations and SFT extracts them, but this has been a vague intuition rather than an empirically demonstrated mechanism with a clear signature: a null effect at pretraining time that becomes a significant positive effect after SFT. The paper identifies and names this phenomenon, giving it empirical grounding through the ℳ_{LMQ} vs. ℳ_{LDQ} comparison.

Why is this significant beyond the specific result? Because it implies that evaluating pretraining data choices based solely on base model benchmarks is systematically misleading. The standard practice of reporting base model perplexity, downstream few-shot accuracy, or even reasoning benchmarks after pretraining may fail to capture the most important effects of data curation decisions—effects that are latent until alignment reveals them. This has practical implications for how pretraining data ablations should be conducted: studies that stop at the base model stage may incorrectly conclude that certain data choices don't matter, when in fact those choices are determining the ceiling of what post-training can achieve.

The mechanism is likely representational: high-quality reasoning traces, even when they constitute a small fraction of the pretraining mix, may shape the model's internal representations in ways that create more favorable conditions for SFT to build upon—perhaps by establishing richer connectivity patterns for multi-step reasoning circuits, or by reducing interference between reasoning and non-reasoning representations. The paper does not mechanistically explain the latent effect (that would require interpretability tools), but it establishes the phenomenon's existence and provides a clean experimental signature that future work can probe. This is an example of a finding that is more important for the question it opens (what other pretraining data choices have latent effects?) than for the specific numerical gain it reports.

The evidence is Table 4, specifically the ℳ_{LMQ} vs. ℳ_{LDQ} post-SFT gap emerging from a near-zero pretraining gap.

---

Innovation 3: The "Catch-Up Refutation" as a Foundational Negative Result

The paper's third innovation is a negative result with structural implications: SFT cannot compensate for a weak pretraining foundation, no matter how aggressively it is scaled. This is not simply a finding that reasoning-rich pretraining helps (which might be true but incremental); it is a finding that reasoning-poor pretraining imposes a hard ceiling on final capability that post-training alone cannot breach.

The experiment is elegantly simple: take the baseline model ℳ_{base} (no reasoning data in pretraining), double its SFT data from the standard 4.8M samples to 9.6M samples, and compare it to the reasoning-pretrained models receiving only the standard SFT allocation. If SFT could compensate, the doubled baseline should at least match the weakest reasoning-pretrained model. It doesn't: ℳ_{base} + SFT[2×] achieves 34.01% average, while ℳ_{SHQ} + SFT (the weakest reasoning-pretrained variant, pretrained on the small, narrow, high-quality dataset) achieves 37.33% (Table 4). The doubled baseline also falls far short of ℳ_{LDQ} + SFT at 46.70%.

What makes this a genuine innovation rather than a confirmation of the obvious is that it establishes a causal ordering that was previously unestablished. The field knew that pretraining data matters (Cheng et al., 2024 showed that instruction-pretraining helps) and that SFT data matters (Zhou et al., 2023 showed quality is key), but it didn't know whether these were substitutes or complements. Could a model that missed reasoning data in pretraining "catch up" by simply training harder during SFT? The answer is a clear no, and the magnitude matters: even with 2× the SFT data, the baseline loses by 3.32 points to the weakest reasoning-pretrained model, and by 12.69 points to the model pretrained with diverse reasoning data.

This finding has architectural implications for how training pipelines should be designed. If SFT were compensatory, organizations could build generalist base models and then specialize them for reasoning through aggressive post-training—a strategy that would be cheaper (since base models could be shared across applications) and more flexible (since post-training can be customized per domain). The catch-up refutation implies that this strategy is fundamentally limited: reasoning capability must be built into the model from the earliest stages of training, and post-training serves to amplify existing foundations rather than create them from scratch. This shifts pretraining from a relatively generic phase (produce a general-purpose base model) to a strategically decisive phase where choices about data composition have irreversible consequences for final capability.

The result also connects to the overfitting concern that motivated the paper's research questions (Section 1). The fact that the reasoning-pretrained models not only maintain their advantage but widen it through SFT (Table 2: a 9.3 point gap vs. 8.35 at the base model stage) demonstrates that early reasoning exposure does not cause the brittleness or overspecialization that the continual fine-tuning literature might predict (Luo et al., 2025b; Gupta et al., 2025). Instead, it creates a more capable substrate that absorbs new SFT data more effectively. This reframes reasoning pretraining from a potential risk (overfitting) to a necessary investment (foundation-building), which is a significant conceptual shift.

The evidence is Table 4, specifically the ℳ_{base} + SFT[2×] row versus the ℳ_{SHQ} + SFT row and the broader pattern across Tables 1–3 showing the gap widening at each stage.

---

Innovation 4: Naive SFT Scaling as Actively Harmful — A Cautionary Principle

The paper's fourth innovation is a cautionary finding that challenges the intuitive but incorrect assumption that scaling reasoning data in SFT is monotonic—that more data, even if somewhat noisy, can't hurt and might help. The evidence shows it can and does hurt, specifically in the reasoning domains where improvement is most desired.

There are two demonstrations of this principle. The first is the direct comparison of SFT data quality in Table 5: reasoning-pretrained models fine-tuned on the large, diverse, mixed-quality dataset 𝒟_{LDQ} (or its union with high-quality data, 𝒟_{LMQ}) achieve worse math reasoning (30.91% and 32.28% mathSFT avg, respectively) than the baseline model ℳ_{base} fine-tuned on the high-quality 𝒟_{SHQ} (42.79% mathSFT avg). The reasoning-pretrained models—which should have a head start—actually underperform the baseline when given low-quality SFT data. The SFT data doesn't just fail to help; it actively erases the pretraining advantage.

The second demonstration is the scaling experiment in Table 6: doubling the amount of diverse but mixed-quality SFT data (𝒟_{LDQ} to 2× 𝒟_{LDQ}) yields negligible average improvement (+0.15 percentage points, from 32.84% to 32.99%) but causes a 4.92 percentage point drop in math accuracy. The extra data is not neutral—it is destructive to the very capability the SFT is supposed to enhance. In contrast, adding a targeted 0.4% increase in data that is specifically high-quality and reasoning-intensive (𝒟_{ALF} augmented with 𝒟_{SHQ} to form 𝒟'_{ALF}) yields consistent improvements, with mathSFT avg rising from ~28% (with 𝒟_{LDQ} alone) to ~61% (with 𝒟'_{ALF}). The conclusion is stark: naive scaling of SFT data with mixed quality is worse than scaling nothing at all.

Why is this intellectually distinctive? The field has long understood that data quality matters for SFT (Zhao et al., 2024; Prabhakar et al., 2025), but this was typically framed as "quality is sufficient" (LIMA showed small high-quality datasets work well) or "quality is more efficient" (you get more improvement per example). The paper's finding goes further: it shows that low-quality SFT data is not merely inefficient but counterproductive—it can actively destroy capabilities built during pretraining. This is a stronger claim than "quality matters" and has different practical implications. Under the weaker claim, one might scale a mixed-quality dataset on the assumption that the good examples will help and the bad ones will be harmless noise. Under the paper's finding, the bad examples are actively harmful, so scaling without stringent quality filtering is a losing strategy.

This finding also illuminates why the asymmetric allocation principle (Innovation 1) holds. Diversity is beneficial in pretraining because the model is learning broad representations from scratch, and even noisy reasoning patterns help establish the structural priors for multi-step inference. But during SFT, the model is in a regime of targeted refinement—it's adjusting an already-structured representation, and noisy data introduces conflicting gradients that can disrupt delicate reasoning circuits that pretraining established. The phase-dependent response to data quality is not an arbitrary empirical observation but reflects a fundamental difference in what each training stage is doing to the model's representations.

The evidence is anchored in Table 6 (the 2× scaling result showing math degradation) and Table 5 (the SFT quality comparison showing reasoning-pretrained models underperforming the baseline when given low-quality SFT data). Together, these establish the cautionary principle with both within-model and across-model evidence.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The pretraining backbone is a 6.2T-token general-purpose corpus from NVIDIA (2025b), drawn from high-quality Common Crawl, mathematics, and code sources. The reasoning datasets injected at controlled proportions are: (1) 𝒟_{LDQ}, a 336B-token large-scale diverse corpus from the Nemotron-Pretraining-SFT-v1 dataset with composition approximately 56% math, 17% code, 27% science and general reasoning; (2) 𝒟_{SHQ}, a 1.2M-example high-quality corpus from Guha et al. (2025) with 71% math, 21% code, 8% science and average answer length exceeding 10K tokens; (3) 𝒟_{LMQ}, the set union of 𝒟_{LDQ} and 𝒟_{SHQ}; and (4) 𝒟_{ALF}, a length-filtered subset of 𝒟_{LDQ} retaining only examples with answer length exceeding 4,096 tokens, representing approximately 2% of the original corpus. All four reasoning datasets are in question-answer format with varying quality, diversity, and reasoning depth. SFT uses 4.8M samples drawn from these datasets. Evaluation benchmarks span four categories: math reasoning (GSM8K, MATH-500, AIME-2024, AIME-2025), science reasoning (MMLU, MMLU-Pro, GPQA-Diamond), code reasoning (HumanEval, MBPP, MBPP-sanitized at base model stage; LiveCodeBench at SFT/RL stage), general-purpose reasoning (ARC-Challenge, HellaSwag, WinoGrande, RACE at base model stage only), and instruction following (IFEval at SFT/RL stage only).

• Base model. All experiments use an 8B-parameter hybrid transformer with a mixture of Mamba 2 layers (Dao & Gu, 2024), standard self-attention layers, and FFN layers, from the NVIDIA Nemotron family (NVIDIA, 2025a). Each variant is pretrained from scratch for 1 trillion tokens on 512 H100 80GB SXM5 GPUs. This size is chosen to balance "computational feasibility and the capacity to learn complex reasoning patterns" (Section 2.1): large enough to exhibit non-trivial reasoning behavior on benchmarks like AIME and GPQA-Diamond, but small enough that the fully-crossed experimental design (four pretraining variants × multiple SFT configurations × RLVR) remains tractable.

• Metrics. The primary metric is Pass@1 accuracy (the fraction of trials where the first generated answer is correct), averaged over multiple evaluation runs to reduce variance: 16 runs for AIME-2024 and AIME-2025 (each with 30 problems), and 4 runs for all other benchmarks. For base model code evaluation specifically, the paper reports avg@32 (average Pass@1 across 32 independent generations per prompt) using EvalPlus variants with sanitized generations (Liu et al., 2023), providing a more stable estimate of code generation capability than single-sample Pass@1. Individual benchmark accuracies are aggregated into domain-level averages (mathSFT avg, scienceSFT avg, codeSFT avg, insSFT avg, gprPT avg) by taking the unweighted mean of the constituent benchmarks. The overall "Average" metric reported in all tables is the unweighted mean of the domain-level averages. For the base model stage: (mathPT avg + sciencePT avg + codePT avg + gprPT avg) / 4. For SFT/RL stages: (mathSFT avg + scienceSFT avg + codeSFT avg + insSFT avg) / 4. Base model evaluations are conducted with LM Eval Harness (Gao et al., 2024); SFT/RL evaluations use NeMo-Skills.

• Baselines. The central baseline is ℳ_{base}, pretrained for 1T tokens on 𝒟_{base} alone with no reasoning data injected during pretraining, then fine-tuned on 𝒟_{SHQ} (the high-quality SFT corpus) using the standard 4.8M-sample SFT recipe, and finally processed through GRPO-RLVR. This represents the conventional paradigm where reasoning is introduced exclusively during post-training. For the catch-up hypothesis test (Table 4), an enhanced baseline ℳ_{base} + SFT[2×] is constructed by doubling the SFT data to 9.6M samples while still using no reasoning data in pretraining. At the pretraining evaluation stage (Table 1), ℳ_{base} serves as the sole reference point against which all three reasoning-pretrained variants are compared. For the SFT data quality ablation (Table 5), the baseline comparison is ℳ_{base} + SFT[𝒟_{SHQ}] achieving 29.92% average (note: this differs from the 26.62% in Table 2 because Table 5 reports the single-model baseline rather than the aggregate ℳ_{base} + SFT group average from the fully-crossed design). The ℳ_{res} notation—used in Tables 1, 2, and 5—denotes the arithmetic mean across the three reasoning-pretrained models (ℳ_{SHQ}, ℳ_{LDQ}, ℳ_{LMQ}), serving as a summary statistic for "the average effect of having any reasoning data in pretraining."

• Generation budget / compute accounting. The paper measures training compute primarily through token and sample counts rather than FLOPs, since the controlled variable is data allocation rather than model size or architecture. All four base models are pretrained on exactly 1 trillion tokens with the same 80/20 blend ratio (ensuring equal pretraining compute), with the independent variable being the 20% reasoning data composition. SFT is standardized at 4.8M samples across all configurations, with the exception of the doubled-SFT baseline (ℳ_{base} + SFT[2×]) at 9.6M samples and the 𝒟'_{ALF} experiment adding only 0.4% more samples. RLVR uses one epoch on the nemotron-crossthink dataset with 128 prompts per step and 8 rollouts per prompt, producing 1,024 generations per step. Evaluation compute is not factored into the budget, consistent with standard practice. The paper does not report wall-clock time for any stage, though the pretraining phase uses 512 H100 GPUs—a substantial but not extreme allocation for an 8B model trained on 1T tokens.

• Cross-validation / statistical protocol. No formal cross-validation is used. The paper relies on multiple evaluation runs (16 for AIME, 4 for other benchmarks) to produce mean accuracy estimates and reduce sampling variance, but does not report standard deviations, confidence intervals, or statistical significance tests for any comparison. The experimental design is a single-trial fully-crossed grid where each pretraining variant is trained once and then fine-tuned under multiple SFT configurations. Since pretraining from scratch is the cost bottleneck (4 models × 1T tokens each), replication with different random seeds for weight initialization or data ordering is not performed. The absence of variance estimates means that small differences between models—particularly the −0.02 point gap between ℳ_{LMQ} and ℳ_{LDQ} at the base model stage (Table 1)—should be interpreted cautiously, as they may reflect noise rather than true performance equivalence. The RLVR stage runs for exactly one epoch in all cases, so differences in RL convergence rate across models could confound the final comparisons; the paper does not report learning curves to verify that all models have converged.

Main Quantitative Results

Immediate Foundational Gains from Reasoning Data in Pretraining

The base model evaluation (Table 1) establishes the immediate, pre-SFT effect of injecting reasoning data into pretraining. The headline result: every model pretrained with reasoning data meaningfully outperforms ℳ_{base} (52.70% overall average), with the largest gains coming from diverse, large-scale data rather than curated quality.

• ℳ_{LDQ} (large-scale diverse, mixed-quality) achieves 64.09% overall—a +11.39 percentage point gain over ℳ_{base}. The improvement is concentrated in math (+28.39 points: 75.56% vs. 47.17% mathPT avg) and code (+9.05 points: 49.94% vs. 40.89% codePT avg), with moderate gains in science (+7.25 points: 54.38% vs. 47.13%). General-purpose reasoning (gprPT avg) shows minimal sensitivity (+0.85 points), consistent across all reasoning-pretrained variants.

• ℳ_{LMQ} (union of large-scale diverse + small high-quality) achieves 64.07% overall—essentially identical to ℳ_{LDQ} (−0.02 points). This is the critical null result that establishes the latent effect: adding high-quality data on top of diversity provides negligible immediate benefit. The per-domain pattern mirrors ℳ_{LDQ} closely (mathPT avg: 72.37% vs. 75.56%; sciencePT avg: 54.49% vs. 54.38%; codePT avg: 52.60% vs. 49.94%, with the code advantage being the one notable positive gap of +2.66 points).

• ℳ_{SHQ} (small-scale high-quality, low-diversity) achieves 54.98% overall—a modest +2.28 point gain over ℳ_{base}, substantially trailing ℳ_{LDQ} by 9.11 points. The quality-only pretraining provides a notable math lift (+5.43 points: 52.60% vs. 47.17%) but minimal science improvement (−0.23 points: 46.90% vs. 47.13%) and modest code gain (+3.43 points: 44.32% vs. 40.89%).

The domain-level pattern is revealing: math and code—domains demanding structured multi-step reasoning—show the largest sensitivity to pretraining reasoning data, with mathPT avg nearly doubling from 47.17% (ℳ_{base}) to 75.56% (ℳ_{LDQ}). Science shows a smaller but meaningful gain. General-purpose reasoning is essentially flat, with all four models clustered within 1.2 points (75.63% to 76.83% gprPT avg), confirming that commonsense and reading comprehension tasks—which test broad world knowledge rather than structured inference—are minimally affected by reasoning data injection during pretraining.

The detailed per-benchmark breakdown (Appendix A, Table 7) reveals that the math gains are driven by large improvements on both GSM8K (ℳ_{LDQ}: 82.71% vs. ℳ_{base}: 59.74%, a +22.97 point gap) and MATH-500 (ℳ_{LDQ}: 68.40% vs. ℳ_{base}: 34.60%, a +33.80 point gap). Interestingly, ℳ_{LMQ} shows stronger GSM8K performance (85.14%) than ℳ_{LDQ} (82.71%) but weaker MATH-500 (59.60% vs. 68.40%)—a crossover pattern suggesting the high-quality data may help with grade-school-level structured problems while not providing additional benefit for competition-level math at the base model stage.

Pretraining Advantage Is Maintained and Amplified Post-SFT

The SFT evaluation (Table 2) addresses two questions simultaneously: whether SFT can "catch up" to reasoning-rich pretraining, and whether reasoning pretraining causes overfitting that diminishes subsequent gains. The headline result: SFT amplifies the pretraining advantage rather than closing it.

• The aggregate ℳ_{res} + SFT group (averaging across all three reasoning-pretrained models after SFT on 𝒟_{SHQ}) achieves 35.92% overall versus 26.62% for ℳ_{base} + SFT—a +9.30 percentage point gap. This is larger than the +8.35 point gap at the base model stage (61.05% vs. 52.70% from Table 1), meaning the pretraining advantage has widened through SFT rather than narrowed. If SFT were compensatory (the catch-up hypothesis), we would expect the gap to shrink or reverse; the observed widening refutes catch-up directly.

• The domain-level decomposition reveals that the SFT-amplified gap is most pronounced in science (+13.85 points: 34.77% vs. 20.92% scienceSFT avg), followed by code (+9.66 points: 16.75% vs. 7.09% codeSFT avg), then math (+6.13 points: 40.61% vs. 34.48% mathSFT avg), and instruction following (+7.54 points: 51.52% vs. 43.98%). The science domain gap is particularly notable because prior work (Prabhakar et al., 2025; Luo et al., 2025a; Huan et al., 2025) had found that "reasoning-centric post-training to be most effective on math domains, the improvement on science is minimal." The paper observes the opposite pattern here: the largest relative gains from reasoning pretraining emerge in science after SFT, suggesting that "pretraining with reasoning data does more than teach facts; it helps the model develop effective internal representations for abstract and logical structures to enhance problem solving ability across domains" (Section 4).

The detailed per-benchmark breakdown in Appendix B (Table 8) shows the individual model variants after SFT on 𝒟_{SHQ}:

The monotonic improvement from ℳ_{SHQ} to ℳ_{LDQ} to ℳ_{LMQ} (42.32% → 52.28% → 56.14%) demonstrates that both diversity and quality in pretraining contribute to post-SFT capability, with the strongest model combining both. Notably, ℳ_{LMQ} + SFT shows a particularly large code advantage over ℳ_{LDQ} + SFT (35.55% vs. 28.57% LiveCodeBench, a +6.98 point gap), suggesting that the high-quality pretraining data's latent benefit is strongest in code reasoning—a domain where structured step-by-step reasoning traces may be especially valuable.

The AIME results (Table 8) underscore the pretraining advantage on expert-level math: ℳ_{LMQ} + SFT achieves 41.88% on AIME-24 and 33.12% on AIME-25, versus 8.12% and 11.88% for ℳ_{base} + SFT—gaps of +33.76 and +21.24 percentage points respectively. These are massive differences, representing the gap between near-random performance and meaningful problem-solving capability on competition-level math.

Pretraining Strategy Dictates Final Accuracy on Expert-Level Tasks (RLVR)

The RLVR results (Table 3) represent the terminal evaluation—the final downstream accuracy after the complete training pipeline—and they reveal that pretraining choices have compounding, not diminishing, effects. Only the two extreme pretraining variants are compared: ℳ_{base} (no reasoning pretraining) and ℳ_{LMQ} (the strongest variant, combining diversity and quality), both after SFT on 𝒟_{SHQ} and GRPO-RLVR.

• ℳ_{LMQ} + SFT + RL achieves 56.66% overall average versus 37.92% for ℳ_{base} + SFT + RL—an +18.74 percentage point gap (Table 3). This is roughly double the +9.30 point gap observed after SFT alone (Table 2), confirming that the pretraining advantage compounds through RL rather than being washed out.

• On math reasoning benchmarks specifically: ℳ_{LMQ} achieves 87.13% on MATH-500 (vs. 72.05% for baseline, +15.08 points), 93.07% on GSM8K (vs. 83.83%, +9.24 points), 45.21% on AIME-24 (vs. 12.29%, +32.92 points), and 33.96% on AIME-25 (vs. 16.04%, +17.92 points). The AIME-24 improvement of +32.92 points is the single largest gap in the entire paper, representing a near-quadrupling of accuracy on one of the most challenging math benchmarks available.

• On science and code: ℳ_{LMQ} achieves 31.69% on GPQA-Diamond (vs. 28.16%, a modest +3.53 point gap), 72.91% on MMLU (vs. 41.10%, a massive +31.81 point gap), 56.91% on MMLU-Pro (vs. 36.69%, +20.22 points), and 32.43% on LiveCodeBench (vs. 13.16%, +19.27 points). The MMLU improvement is particularly striking—a 31.81 point gain on a broad 57-subject benchmark that tests factual knowledge and reasoning across domains, suggesting that reasoning-rich pretraining improves not just math-specific circuits but domain-general knowledge integration.

The compounding dynamic—8.35 point gap at base model → 9.30 point gap after SFT → 18.74 point gap after RL—implies that each training stage acts as an amplifier of pretraining quality. RLVR, which directly optimizes for correct answers via verifiable rewards, provides the strongest amplification because it can exploit the richer internal representations built during pretraining. A model with better initial reasoning circuits can explore more productively during RL, generating rollouts with a higher probability of containing correct solutions, which provides better training signal, which further improves the policy. This virtuous cycle does not exist (or exists much more weakly) for the baseline model, whose weak pretraining foundation limits the quality of RL exploration.

The Catch-Up Hypothesis Is Refuted

Table 4 provides the direct test of whether SFT can compensate for absent reasoning pretraining. ℳ_{base} + SFT[2×]—the baseline model with doubled SFT data—achieves 34.01% overall average. This is a +7.39 point improvement over the standard ℳ_{base} + SFT at 26.62% (Table 2), confirming that more SFT data does help the baseline. However, this enhanced baseline still falls short of every reasoning-pretrained model:

• ℳ_{SHQ} + SFT at 37.33%: the weakest reasoning-pretrained variant (small-scale, high-quality, low-diversity pretraining) outperforms the doubled-SFT baseline by +3.32 percentage points, despite the baseline having consumed 2× the SFT data.
• ℳ_{LDQ} + SFT at 46.70%: the diverse pretraining variant outperforms the doubled baseline by +12.69 points.
• ℳ_{LMQ} + SFT at 50.95%: the combined variant outperforms by +16.94 points.

The domain-level decomposition reveals that the doubled baseline's math improvement (48.05% mathSFT avg) brings it close to ℳ_{SHQ} + SFT (50.52%) but still trails ℳ_{LDQ} + SFT at 60.79% by a substantial 12.74 points. More revealing is the instruction-following domain: ℳ_{base} + SFT[2×] achieves only 32.70% insSFT avg, which is actually lower than the standard ℳ_{base} + SFT at 43.98% (Table 2, 30.59% in the single-model breakdown of Table 8)—a degradation of 11.28 points (or reversal of ordering depending on which baseline figure is used). This suggests that scaling mixed-quality SFT data may particularly harm the model's ability to follow instructions, possibly because noisy examples introduce conflicting patterns for task formatting and response structure.

The catch-up refutation is not marginal—the gap between the doubled baseline and even the weakest reasoning-pretrained model (+3.32 points) exceeds the entire improvement gained from doubling SFT data (+7.39 points relative to standard SFT, or roughly +4.09 points relative to the Table 8 single-model baseline of 35.52%). This means that adding 4.8M extra SFT samples to the baseline provides less benefit than adding reasoning data to pretraining with a small, narrow dataset, establishing a strong causal ordering: pretraining data composition is more impactful than SFT data scale.

SFT Is Dominated by Data Quality, Not Diversity

Table 5 provides the clearest evidence for the asymmetric allocation principle. Reasoning-pretrained models (ℳ_{res}) are fine-tuned on three different SFT data configurations, and the results invert the pattern observed at the pretraining stage:

The gap between 𝒟_{SHQ} and 𝒟_{LDQ} at the SFT stage is ~13.45 points in favor of the small, high-quality corpus. At the pretraining stage (Table 1), the pattern was reversed: the large, diverse corpus (ℳ_{LDQ}) beat the small, high-quality corpus (ℳ_{SHQ}) by 9.09 points. The optimal data characteristic flips sign between phases.

Critically, reasoning-pretrained models fine-tuned on 𝒟_{LDQ} achieve worse math reasoning (32.28% mathSFT avg) than the baseline ℳ_{base} fine-tuned on 𝒟_{SHQ} (42.79% mathSFT avg, from the Table 8 breakdown)—a gap of −10.51 points favoring the baseline model. This means that low-quality SFT data does not merely fail to help reasoning-pretrained models; it erases their pretraining advantage and leaves them worse off than a model that never saw reasoning data in pretraining at all. The mixed-quality SFT data actively damages the reasoning circuits that pretraining established.

The instruction-following domain shows the opposite effect: 𝒟_{LDQ} and 𝒟_{LMQ} achieve 54.61% and 56.41% insSFT avg respectively, substantially outperforming 𝒟_{SHQ} at 43.56% (and also outperforming ℳ_{base} + SFT[𝒟_{SHQ}] at 30.59% from Table 8). This suggests a domain tradeoff: large-scale diverse SFT data improves broad instruction-following capability at the expense of deep reasoning, while high-quality long-CoT SFT data improves deep reasoning at the expense of instruction-following breadth. This tradeoff between reasoning depth and instruction-following generality is an important nuance to the asymmetric allocation principle: "quality dominates SFT" is true for reasoning tasks specifically but not for all capabilities.

Naive SFT Scaling Is Ineffective and Harmful

Table 6 examines two strategies for scaling SFT data on top of ℳ_{LDQ} (the model pretrained with large-scale diverse data):

Doubling mixed-quality data: ℳ_{LDQ} + SFT[2× 𝒟_{LDQ}] achieves 32.99% overall, essentially identical to ℳ_{LDQ} + SFT[𝒟_{LDQ}] at 32.84%—a negligible +0.15 point gain from doubling the SFT data. But the domain-level decomposition reveals a hidden cost: mathSFT avg drops from 28.38% to 23.46%, a −4.92 point degradation. Science and code show small improvements (+4.43 and +1.59 points respectively), but these do not offset the math regression. The extra data is not neutral noise; it is actively harmful to mathematical reasoning, the very capability the SFT is supposed to enhance. The paper attributes this to "noisy or shallow reasoning data" that "may dilute the useful signal" (Section 5).

Targeted scaling of high-quality data: ℳ_{LDQ} + SFT[𝒟_{ALF}] (replacing 𝒟_{LDQ} with the length-filtered subset retaining only examples with answer length >4,096 tokens) achieves 42.66% overall—a +9.82 point gain over using the full 𝒟_{LDQ} at 32.84%. The math improvement is dramatic: mathSFT avg jumps from 28.38% to 60.95%, a +32.57 point gain. Science improves by +12.07 points (35.22% → 47.29%), and code more than doubles (10.16% → 22.54%). However, instruction following degrades from 57.61% to 39.87%, confirming the breadth-vs-depth tradeoff observed in Table 5.

Adding a marginal 0.4% of extra high-quality data: ℳ_{LDQ} + SFT[𝒟'_{ALF}] (which scales 𝒟_{ALF} with 𝒟_{SHQ}, adding only 0.4% more samples) achieves 43.04% overall, a further +0.38 point gain. Math improves to 61.61% (+0.66 points over 𝒟_{ALF} alone). The key takeaway is not the magnitude of this improvement but its efficiency: a 0.4% increase in dataset size, strategically targeted at high-quality reasoning traces, yields measurable gains, while a 100% increase in mixed-quality data yields none and actively harms math. This asymmetry in the marginal utility of data—positive and large for quality, zero or negative for mixed-quality quantity—is the paper's strongest evidence that SFT scaling must be quality-driven rather than quantity-driven.

Ablation Studies and Robustness Checks

The latent effect of high-quality pretraining data: The near-zero gap between ℳ_{LMQ} (64.07%) and ℳ_{LDQ} (64.09%) at the base model stage (Table 1) versus the +4.25 point gap after SFT (50.95% vs. 46.70% in Table 4) demonstrates that high-quality pretraining data can have effects invisible at the base model stage but "unlocked" by alignment. This latent effect is not symmetric: the high-quality data's benefit emerges only after the model is explicitly fine-tuned to express reasoning capabilities. The absence of this pattern in the base model evaluation confirms that standard pretraining benchmarks (perplexity, few-shot accuracy, commonsense reasoning) would fail to capture this effect, making it a methodological caution for how pretraining ablations should be evaluated.

Diversity vs. quality in pretraining (Table 1): ℳ_{LDQ} (diverse, mixed-quality) achieves 64.09% overall versus ℳ_{SHQ} (narrow, high-quality) at 54.98%—a +9.11 point advantage for diversity. The domain decomposition shows that diversity's advantage spans all reasoning domains: math (+22.96 points), science (+7.48 points), code (+5.62 points). The ℳ_{LMQ} result (64.07%, adding quality on top of diversity) confirms that quality alone provides minimal additional immediate benefit when diversity is already present, but Table 4 shows this quality investment pays off after SFT (+4.25 points). This establishes a two-stage value proposition for pretraining data: diversity provides immediate base-model gains, while quality provides latent gains activated by post-training.

Reasoning depth as a quality marker in SFT (Appendix C, Table 9): ℳ_{LDQ} + SFT[𝒟_{ALF}] (filtered for answer length >4,096 tokens) achieves 42.71% overall versus 32.84% for ℳ_{LDQ} + SFT[𝒟_{LDQ}]—a +9.87 point gain despite 𝒟_{ALF} being approximately 50× smaller (~2% of the original corpus). Math improves most dramatically (+32.57 points), followed by science (+12.28 points) and code (+12.38 points). Instruction following degrades (−17.74 points), establishing the depth-vs-breadth tradeoff. This ablation isolates "reasoning trace length" as a specific marker of SFT data quality, consistent with the broader finding that longer chain-of-thought examples provide richer supervisory signals for structured multi-step inference.

Data redundancy between pretraining and SFT (Appendix C, Figure 2): The model exposed to 𝒟_{SHQ} in both pretraining and SFT (ℳ_{SHQ} + SFT[𝒟_{SHQ}]) significantly outperforms the baseline that saw 𝒟_{SHQ} only during SFT (ℳ_{base} + SFT[𝒟_{SHQ}]), with the Figure 2 bar chart showing consistent gains across math, science, code, and instruction-following categories. This directly refutes the concern that "using the same reasoning data in both pretraining and SFT leads to catastrophic forgetting or brittle overfitting" (Appendix C). The paper hypothesizes that the two phases serve different functions: pretraining slowly integrates reasoning data into core representations alongside diverse knowledge, while SFT acts as reinforcement on an already-prepared foundation. The top-performing ℳ_{LMQ} model further amplifies this effect by leveraging its diverse pretraining exposure to more effectively capitalize on the repeated high-quality signal.

Answer-length filtering as a quality heuristic (𝒟_{ALF} construction): The 𝒟_{ALF} dataset, constructed by applying a simple 4,096-token answer-length threshold to the large mixed-quality 𝒟_{LDQ} corpus, serves as a robustness check on the "quality matters in SFT" finding. Since 𝒟_{ALF} is drawn from the same underlying distribution as 𝒟_{LDQ} (same sources, domains, generation processes) and differs only in the length/complexity of individual examples, the large performance gap between SFT[𝒟_{LDQ}] (32.84%) and SFT[𝒟_{ALF}] (42.71%) isolates reasoning depth as a causal factor rather than a correlate of other quality dimensions. The 75% math bias introduced by this filter is a confound—𝒟_{ALF} is not just longer but domain-skewed—but the strong math improvement (+32.57 points) relative to the more modest science and code gains suggests that length is acting as a genuine quality signal beyond domain composition.

The ℳ_{res} aggregate as a robustness summary: Throughout Tables 1, 2, and 5, the paper reports ℳ_{res}—the arithmetic mean of the three reasoning-pretrained variants—as a summary statistic. In Table 1, ℳ_{res} achieves 61.05% versus ℳ_{base} at 52.70% (+8.35 points). In Table 2, ℳ_{res} + SFT achieves 35.92% versus ℳ_{base} + SFT at 26.62% (+9.30 points). In Table 5, ℳ_{res} + SFT[𝒟_{SHQ}] achieves 44.99% versus ℳ_{base} + SFT[𝒟_{SHQ}] at 29.92% (+15.07 points). The ℳ_{res} aggregate consistently shows positive effects from reasoning pretraining regardless of the specific data composition, confirming that the benefit is robust to the choice of reasoning dataset and not dependent on a particular quality-diversity tradeoff. The widening gap from +8.35 (base) to +9.30 (SFT) to a projected larger gap after RL (extrapolating from Table 3's ℳ_{LMQ} comparison) further confirms the compounding dynamic is not an artifact of a single variant.

Negative result: ReST^EM-style revision model degradation (not in this paper): This paper does not include a negative result analogous to the ReST^EM revision model degradation described in the reference example. The paper's primary negative results are the SFT data scaling failure (Table 6: doubling mixed-quality data causes −4.92 point math drop) and the instruction-following degradation when emphasizing reasoning depth (Tables 5, 9). No ablation attempts an alternative training methodology (e.g., different RL algorithm, different pretraining curriculum, different architecture) that fails, which limits the paper's ability to characterize the robustness of its findings to implementation choices.

Critical Assessment

On the Claim: "Front-loading reasoning data into pretraining creates a durable, compounding advantage"

What was demonstrated: Tables 1–3 convincingly show that models pretrained with reasoning data (ℳ_{res}) outperform the baseline (ℳ_{base}) at every stage: +8.35 points at the base model stage (Table 1), +9.30 points after SFT (Table 2), and +18.74 points after RLVR for the strongest variant (Table 3, ℳ_{LMQ} vs. ℳ_{base}). The gap widens monotonically across stages, consistent with compounding. The catch-up refutation (Table 4) shows that even 2× SFT data on the baseline cannot close the gap to the weakest reasoning-pretrained model.

Where the evidence supports the claim: The widening gap from base model to SFT to RL is clean and internally consistent. The claim that SFT "amplifies" the pretraining advantage is supported by the +9.30 point post-SFT gap exceeding the +8.35 point base-model gap. The compounding claim is supported by the +18.74 point post-RL gap substantially exceeding the post-SFT gap. The claim that "post-training alone cannot recover" the advantage is directly supported by Table 4.

Where caution is warranted: The RLVR comparison in Table 3 only includes two models (ℳ_{LMQ} and ℳ_{base}), not the full set of pretraining variants. We do not know whether ℳ_{LDQ} or ℳ_{SHQ} would also show compounding through RL, or whether the compounding effect is specific to the ℳ_{LMQ} configuration. The RLVR stage runs for only one epoch, and the paper does not report convergence curves—if ℳ_{LMQ} converges faster during RL (which is plausible given its better initial representations), the one-epoch comparison may overstate the long-run gap if both models would eventually converge to similar accuracy. Conversely, if ℳ_{base} converges faster but to a lower asymptote, the one-epoch comparison may understate the gap. Without learning curves, the "compounding" interpretation is confounded with the possibility of differential RL efficiency.

Additionally, the SFT and RLVR stages for all models use the same hyperparameters (learning rate, batch size, number of epochs) optimized for the baseline or for general practice—not tuned per-variant. It is possible that reasoning-pretrained models would benefit from different SFT hyperparameters (e.g., lower learning rate to avoid disrupting established reasoning circuits), and the gap would be even larger with variant-specific tuning. Conversely, the baseline might benefit from different hyperparameters as well. The paper's fixed-hyperparameter design is appropriate for a controlled comparison but means the reported gaps are conditional on the specific hyperparameter choices.

On the Claim: "The optimal data strategy is asymmetric—prioritize diversity in pretraining and quality in SFT"

What was demonstrated: Table 1 shows ℳ_{LDQ} (diverse, mixed-quality) > ℳ_{SHQ} (narrow, high-quality) by +9.09 points at the base model stage. Table 5 shows SFT[𝒟_{SHQ}] (small, high-quality) > SFT[𝒟_{LDQ}] (large, diverse, mixed-quality) by +13.45 points at the SFT stage. The asymmetry is empirically visible: diversity dominates in one phase, quality in the other.

Where the evidence supports the claim: The phase reversal is clear and does not depend on marginal differences. The +9.09 point pretraining gap and the +13.45 point SFT gap are both large effects unlikely to be explained by evaluation noise. The finding that 𝒟_{LDQ} SFT actively erases reasoning-pretrained advantages (Table 5: ℳ_{res} + SFT[𝒟_{LDQ}] at 31.54% vs. ℳ_{base} + SFT[𝒟_{SHQ}] at 29.92%) strengthens the case that low-quality SFT data is not merely ineffective but harmful, making the asymmetric principle practically consequential.

Where caution is warranted: The claim is established for the specific datasets used, but the dimensions of "diversity" and "quality" are confounded with other dataset properties. 𝒟_{LDQ} has 336B tokens; 𝒟_{SHQ} has 1.2M examples with 10K+ token average answer length. These differ not just in diversity and quality but in scale, domain composition (56% vs. 71% math), average response length (~550 vs. ~10K tokens), source models, and data collection methodology. The term "diversity" conflates multiple factors: domain breadth, task format variety, source diversity, and sheer token count. The term "quality" conflates answer length, teacher model capability, and curation effort. The asymmetric principle is most accurately stated as: "the specific large-scale diverse dataset 𝒟_{LDQ} is better than the specific small-scale high-quality dataset 𝒟_{SHQ} during pretraining, and vice versa during SFT." Whether this generalizes to other instantiations of "diverse" and "high-quality" datasets is not tested.

A stronger test would systematically vary one dimension at a time: hold quality constant and vary diversity, hold diversity constant and vary quality, at both stages. The current dataset design pairs (high-diversity, mixed-quality) against (low-diversity, high-quality), which confounds the two dimensions. The 𝒟_{LMQ} dataset partially addresses this (adding quality to diversity) but only for the pretraining stage. A more complete design would include a (low-diversity, low-quality) dataset and a (high-diversity, high-quality) dataset, though constructing the latter might be infeasible.

On the Claim: "Naive scaling of SFT data is ineffective and harmful"

What was demonstrated: Table 6 shows that doubling 𝒟_{LDQ} for SFT yields negligible average improvement (+0.15 points) and causes a −4.92 point drop in math accuracy. In contrast, adding 0.4% more data via 𝒟'_{ALF} yields consistent improvements across reasoning domains.

Where the evidence supports the claim: The math degradation from doubling mixed-quality data is the strongest direct evidence that naive SFT scaling can be harmful. The contrast between doubling mixed-quality data (neutral/harmful) and adding a small amount of high-quality data (beneficial) is striking and practically important.

Where caution is warranted: The finding is established only for ℳ_{LDQ} as the pretrained base and 𝒟_{LDQ} as the SFT data being scaled. We don't know whether doubling 𝒟_{SHQ} (high-quality SFT data) would also be harmful, neutral, or beneficial—the harmful effect might be specific to the mixed-quality nature of 𝒟_{LDQ} rather than to SFT scaling in general. The paper implies that quality is the key variable (hence the contrast with 𝒟'_{ALF}), but the ablation that would cleanly test this—doubling 𝒟_{SHQ} for SFT and observing whether math degrades—is not performed. If doubling high-quality data also degrades math, the mechanism would be overfitting rather than noise dilution, which would have different practical implications.

Also, the SFT data scaling is from 4.8M to 9.6M samples—a 2× increase. The paper does not explore whether 3×, 4×, or 10× scaling of mixed-quality data causes monotonic degradation or whether there is a U-shaped curve where very large amounts of mixed-quality data eventually help through sheer exposure. The practical recommendation "don't blindly scale SFT data" is supported for the 2× regime but may not hold for orders-of-magnitude larger scales.

On the Claim: "High-quality pretraining data can have a latent effect unlocked by SFT"

What was demonstrated: ℳ_{LMQ} and ℳ_{LDQ} have near-identical base model performance (64.07% vs. 64.09%), but after SFT ℳ_{LMQ} leads by +4.25 points (Table 4: 50.95% vs. 46.70%).

Where the evidence supports the claim: The emergence of a gap where none previously existed is the clean signature of a latent effect. The +4.25 point gap after SFT is substantial relative to the near-zero gap at the base model stage. The opportunity to evaluate this was created by the fortunate empirical outcome that ℳ_{LMQ} and ℳ_{LDQ} performed almost identically at the base model stage—if they had differed, the latent effect would be harder to isolate.

Where caution is warranted: The −0.02 point gap at the base model stage is not necessarily evidence of "no difference." Without standard deviations, we cannot distinguish between "these two models are truly equivalent" and "these two models differ by a small amount that this experiment cannot reliably detect." If the true base model gap were +0.5 points in favor of ℳ_{LMQ} with high variance, the post-SFT gap of +4.25 points would still represent amplification but the "zero at pretraining → positive after SFT" narrative would be weakened. The claim of a latent effect is compelling but would be strengthened by replication: does the same pattern appear when comparing other dataset pairs that produce near-identical base model performance?

Additionally, the paper does not explain why the latent effect occurs. Is it because the high-quality data installs reasoning patterns that are only exercised when the model is prompted with instruction-formatted queries (which base model evaluation does not use)? Is it because the high-quality data reduces interference between reasoning and non-reasoning representations, which only matters when the model is fine-tuned to emphasize reasoning? The paper provides the empirical signature without mechanistic understanding, which makes the finding suggestive rather than explanatory.

On the Overall Experimental Design

Strengths: The fully-crossed design (four pretraining variants × multiple SFT configurations × RLVR) is extremely thorough for a pretraining-scale study, where each base model variant costs hundreds of thousands of GPU-hours. The inclusion of RLVR as a terminal evaluation stage is critical for establishing that pretraining effects are durable under aggressive optimization. The per-benchmark reporting in appendices (Tables 7, 8) provides transparency. The data diversity/quality/quantity taxonomy is conceptually clear and enables precise formulation of the asymmetric allocation principle.

Weaknesses and missing experiments:

1. Single model family, single architecture, single scale. All results are for an 8B hybrid Mamba-transformer model from the NVIDIA Nemotron family. It is unclear whether the asymmetric principle generalizes to pure transformer architectures, larger scales (70B+), or models from different families with different pretraining data distributions and inductive biases. At 8B parameters, the model may be in a regime where pretraining data effects are more pronounced because the model has not yet saturated its capacity for reasoning—a 405B model might show different phase-dependent patterns.

2. No variance reporting. The absence of standard deviations, confidence intervals, or statistical tests means that small differences (the −0.02 point gap between ℳ_{LMQ} and ℳ_{LDQ}, the +0.15 point gain from doubling SFT data, the +0.38 point gain from 𝒟'_{ALF}) cannot be meaningfully interpreted. For a paper whose central claims involve comparisons of numerical differences—sometimes small, sometimes large—this is a significant limitation. The multiple evaluation runs (16 for AIME, 4 for others) reduce within-benchmark variance but do not capture between-run variance from different weight initializations or data orderings, which are not replicated.

3. RLVR limited to two models. Only ℳ_{LMQ} and ℳ_{base} are compared after RLVR (Table 3). The exclusion of ℳ_{LDQ} and ℳ_{SHQ} from the RL stage means we cannot observe whether the latent effect (ℳ_{LMQ} > ℳ_{LDQ}) persists or amplifies through RL, or whether the diversity advantage (ℳ_{LDQ} > ℳ_{SHQ}) does. Given that Table 3 is the paper's headline finding (the 19% gain claim), the incomplete RL coverage is a notable gap.

4. No SFT data quality scaling for the baseline. The paper shows that doubling mixed-quality SFT data harms reasoning-pretrained models (Table 6) and that doubling high-quality SFT data helps the baseline (Table 4: ℳ_{base} + SFT[2×] improves to 34.01%). But it does not test whether doubling high-quality SFT data helps reasoning-pretrained models, or whether the harmful scaling effect of mixed-quality data also applies to the baseline. The interaction between pretraining condition and SFT scaling strategy remains partially characterized.

5. Difficulty-based analysis is absent. Unlike some reasoning papers that break down results by problem difficulty, this paper reports only domain-level averages and per-benchmark scores. We don't know whether the pretraining advantage is strongest on easy, medium, or hard problems—which would inform whether reasoning-rich pretraining primarily helps with routine reasoning (amplifying existing capability) or enables novel reasoning (extending capability). The AIME results hint at large gains on difficult problems, but this is not systematically analyzed.

6. The 20% reasoning data proportion is a single design point. All reasoning-pretrained models use an 80/20 blend of general and reasoning data. The paper does not explore whether 5%, 10%, or 50% reasoning data would produce different patterns. This matters because the asymmetric principle's practical implementation requires knowing not just which data to use but how much. If 5% diverse reasoning data already saturates the pretraining benefit, the practical recommendation changes from "allocate a large fraction of pretraining budget to diverse reasoning data" to "include a modest amount."

7. Overfitting assessment is limited to aggregate accuracy. The paper claims that reasoning pretraining does not cause overfitting (Section 4, Appendix C), but the evidence is that reasoning-pretrained models show larger post-SFT gains than the baseline. This is consistent with "no overfitting" but does not rule out subtler forms of overspecialization—e.g., reasoning-pretrained models might perform worse than the baseline on tasks very far from the reasoning distribution (toxic language detection, creative writing, translation), which are not evaluated. The gprPT avg domain (commonsense reasoning) shows insensitivity to reasoning pretraining, which is reassuring but not a comprehensive test of generalization.

8. The difficulty estimation insight from prior work is absent. The reference example paper emphasized that difficulty estimation is critical for adaptive test-time compute allocation. This paper does not engage with the question of whether different problem difficulties within the evaluation benchmarks respond differently to pretraining data choices, which would be a natural extension of the asymmetric principle.

Despite these limitations, the paper's core empirical findings—the widening gap across stages, the phase-dependent reversal of diversity vs. quality utility, the harm from naive SFT scaling, and the latent pretraining effect—are well-supported by the data presented. The experimental design, while not exhaustive, is appropriate for the paper's primary contribution: establishing the existence and qualitative nature of phase-dependent data allocation effects in reasoning-focused LLM training. The missing experiments represent opportunities for future work rather than threats to the paper's central conclusions.

6. Limitations and Trade-offs

The Latent Effect Mechanism Is Unexplained and Untested Across Conditions

The assumption or constraint. The paper demonstrates that adding high-quality data (𝒟_{SHQ}) on top of diverse pretraining data (𝒟_{LDQ}) yields near-zero immediate gain at the base model stage (64.07% for ℳ_{LMQ} vs. 64.09% for ℳ_{LDQ}, Table 1) but a substantial +4.25 percentage point advantage after SFT (50.95% vs. 46.70%, Table 4). The paper names this phenomenon a "latent effect" and describes it as pretraining data that "may act as a complementary amplifier, whose benefits emerge after alignment" (Section 5). However, the paper provides no mechanistic account of why this occurs. It speculates about "internal representations for abstract and logical structures" (Section 4) but conducts no analysis of model internals, attention patterns, or representation geometry to distinguish between competing explanations.

The consequence. Without mechanistic understanding, practitioners cannot predict which high-quality data will have latent effects, how much high-quality data is needed to trigger them, or whether the effect requires specific SFT conditions to unlock. The finding is a single empirical observation (one model pair, one SFT recipe, one evaluation) rather than a characterized phenomenon. If a team adds high-quality data to their pretraining mix and sees no immediate base model improvement, they cannot know whether they are observing a genuine latent effect that will pay off after SFT, or whether their high-quality data is simply inert. The decision to invest in expensive pretraining data curation with no immediate payoff requires confidence that the latent effect generalizes—confidence the paper's evidence does not support.

What evidence exists in the paper. Only one comparison demonstrates the latent effect: ℳ_{LMQ} vs. ℳ_{LDQ} after SFT on 𝒟_{SHQ} (Table 4). No additional dataset pairs, no alternative SFT recipes, and no variant with different proportions of high-quality data are tested. The ℳ_{res} aggregate in Table 5 (SFT[𝒟_{SHQ}] at 44.99%) combines all three reasoning-pretrained models, so the latent effect is averaged with the diversity effect and cannot be isolated. No interpretability analysis or probing experiment is conducted to characterize what the model learns from the high-quality pretraining data that only becomes useful after SFT.

Mitigation status. Not addressed. The paper presents the latent effect as a discovery rather than a solved mechanism. Section 2 and Section 5 describe the phenomenon without claiming to explain it. The authors do not propose future work specifically targeting mechanistic understanding of latent pretraining effects, though the finding implicitly opens this research direction. A practitioner reading the paper would know that latent effects exist in one case but would have no guidance for producing or predicting them in their own training pipelines.

---

Difficulty Estimation Cost Is Not Accounted For — And Analogous Overhead Exists Here

The assumption or constraint. The paper's experimental design requires pretraining four separate 8B-parameter models from scratch for 1 trillion tokens each on 512 H100 GPUs to characterize the function 𝒫(𝒟_{res}^{PT}, 𝒟_{res}^{SFT}). This is not an inference-time cost of a deployed method (the paper proposes no deployment-time method), but it constitutes the cost of deriving the paper's actionable guidance (the asymmetric allocation principle). An organization wanting to apply this principle to their own model family, architecture, scale, and data distribution would need to replicate something approximating the paper's fully-crossed experimental design—pretraining multiple model variants with different data compositions and evaluating the outcome surface. The paper provides no cheaper proxy: no scaling law that predicts phase-dependent data utility from smaller-scale experiments, no transfer learning result showing that the 8B findings predict behavior at 70B or 405B, and no lightweight diagnostic that estimates "reasoning data sensitivity" without full pretraining runs. Section 2.3 acknowledges the computational scale implicitly ("pretrain from scratch for 1 trillion tokens... using 512 H100 80GB SXM5 GPUs") but does not frame this as a limitation.

The consequence. The paper's primary practical output—the asymmetric principle ("diversity in pretraining, quality in SFT")—is derived from experiments whose total compute cost likely exceeds $1M in GPU-hours. A practitioner wanting to verify that the principle applies to their setting (different architecture, different base corpus, different scale, different reasoning datasets) faces a cost that may rival or exceed the training budget they are trying to optimize. The paper does not provide the kind of lightweight heuristic that would make the principle actionable without large-scale replication. This is analogous to the difficulty estimation problem flagged in the reference example paper: the guidance is valuable but the cost of obtaining it is unaccounted for in the headline claims.

What evidence exists in the paper. The paper implicitly acknowledges the cost by limiting the study to a single architecture and scale (8B hybrid Mamba-transformer, Section 2.1): "This size strikes a balance between computational feasibility and the capacity to learn complex reasoning patterns." The choice suggests that even at 8B, the fully-crossed design was pushing computational limits, and larger scales were infeasible. No ablation tests whether a cheaper experimental design (e.g., pretraining only to 250B or 500B tokens, using a smaller model, using fewer SFT configurations) would recover the same qualitative findings, which would be the natural route to making the approach more practical.

Mitigation status. Not addressed. The paper does not propose smaller-scale proxy experiments, data utility predictors, or scaling laws that would reduce the cost of applying its framework to new settings. Section 7 (Conclusion) frames the contribution as providing "a principled guide for strategically allocating data across the entire training pipeline" without acknowledging the cost of deriving that guide. Future work on predicting phase-dependent data utility from cheaper signals (e.g., data mixture optimization at small scale, representation similarity metrics, pretraining loss curvature) would directly address this limitation.

---

The Asymmetric Principle Is Established for One Dataset Pair, Not for Abstract Dimensions

The assumption or constraint. The paper's central claim—that diversity dominates in pretraining while quality dominates in SFT—is operationalized through exactly two datasets: 𝒟_{LDQ} (336B tokens, ~56% math, ~550-token average answer length, from Nemotron-Pretraining-SFT-v1) and 𝒟_{SHQ} (1.2M examples, ~71% math, ~10K-token average answer length, from Guha et al., 2025). These datasets differ along multiple dimensions simultaneously: scale (336B tokens vs. ~12B tokens equivalent), domain composition (56% vs. 71% math), answer length (~550 vs. ~10K tokens), teacher model quality, curation methodology, and task format distribution. The paper's design contrasts "large-scale, diverse, mixed-quality" against "small-scale, high-quality, low-diversity" as a package deal—there is no experiment that varies diversity while holding quality constant (or vice versa) in either training phase.

The consequence. A practitioner cannot confidently apply the principle to datasets that differ from the paper's specific instantiation. Suppose an organization has a moderately diverse, moderately high-quality reasoning dataset. Which phase should it go into? The paper's principle says "diversity for pretraining, quality for SFT" but provides no threshold for what counts as "diverse enough" or "high-quality enough." If the dataset is 45% math with 2K-token average answers, does it behave more like 𝒟_{LDQ} or 𝒟_{SHQ}? The abstract dimensions ("diversity," "quality") are underspecified relative to the concrete datasets that operationalize them, creating a generalization gap between the paper's terminology and its actual empirical coverage.

What evidence exists in the paper. Table 1 demonstrates ℳ_{LDQ} (64.09%) > ℳ_{SHQ} (54.98%) at the base model stage. Table 5 demonstrates SFT[𝒟_{SHQ}] (44.99%) > SFT[𝒟_{LDQ}] (31.54%) at the SFT stage. These comparisons jointly establish that the specific 𝒟_{LDQ} dataset is better than the specific 𝒟_{SHQ} dataset during pretraining, and vice versa during SFT. But 𝒟_{LMQ} (Table 1: 64.07%) shows that adding 𝒟_{SHQ}'s high-quality data to 𝒟_{LDQ}'s diversity provides no additional pretraining benefit—yet 𝒟_{LMQ} does not isolate quality independent of diversity because it also increases total token count. Similarly, the 𝒟_{ALF} experiment (Table 6) varies answer length within the 𝒟_{LDQ} distribution but does so only for SFT, not pretraining. No experiment cleanly isolates the diversity dimension from the quality dimension by, for example, subsampling 𝒟_{LDQ} to match 𝒟_{SHQ}'s domain composition while retaining diversity of sources, or by degrading 𝒟_{SHQ}'s quality while retaining its narrow domain focus.

Mitigation status. Partially addressed through the 𝒟_{ALF} and 𝒟_{LMQ} datasets, but these provide only indirect evidence. 𝒟_{ALF} (Table 6) shows that within the 𝒟_{LDQ} distribution, filtering for answer length (a quality proxy) dramatically improves SFT effectiveness, supporting the "quality matters in SFT" half of the principle. 𝒟_{LMQ} (Table 4) shows that adding quality to diversity in pretraining has latent benefits after SFT, supporting the idea that pretraining quality is not irrelevant—it's just less immediately visible. But neither experiment varies diversity at fixed quality or quality at fixed diversity, so the abstract claim of an "asymmetric principle" operating on independent dimensions remains an interpretation of the data rather than a directly tested hypothesis.

---

The RLVR Analysis Is Incomplete — Only Two of Four Pretraining Variants Are Evaluated

The assumption or constraint. The paper's headline finding—a 19% average gain on expert-level benchmarks from reasoning-rich pretraining (Section 1, Table 3)—is based on comparing only two of the four pretrained models after RLVR: ℳ_{LMQ} (the strongest variant, combining diversity and quality) and ℳ_{base} (the control with no reasoning pretraining). The intermediate variants ℳ_{LDQ} (diverse, mixed-quality) and ℳ_{SHQ} (narrow, high-quality) are not evaluated after RLVR. This means the paper cannot characterize how different pretraining data strategies interact with reinforcement learning—whether the diversity advantage (ℳ_{LDQ} > ℳ_{SHQ}) persists, amplifies, or reverses under RL optimization, and whether the latent effect (ℳ_{LMQ} > ℳ_{LDQ}) compounds further or saturates.

The consequence. Two of the paper's central claims cannot be fully validated through the complete training pipeline. The claim that the pretraining advantage "compounds" (Section 1, Section 4) is supported by the widening gap from base model (+8.35 pts, Table 1) to SFT (+9.30 pts, Table 2) to RLVR (+18.74 pts, Table 3), but the RLVR number comes from comparing only the extreme variants. If ℳ_{LDQ} + SFT + RL achieved, say, 52% (closing much of the gap to ℳ_{LMQ} at 56.66%), the compounding narrative would be nuanced: diversity alone might be nearly as effective as diversity-plus-quality after RL. If ℳ_{SHQ} + SFT + RL achieved only 42% (barely improving over its SFT performance), the claim that "any reasoning pretraining helps" would need qualification—narrow high-quality pretraining might not support effective RL exploration. Without these data points, the paper's strongest quantitative result (the RLVR comparison) characterizes only the best-case vs. worst-case scenario, not the shape of the pretraining data utility curve after reinforcement learning.

What evidence exists in the paper. Table 3 reports only ℳ_{base} + SFT + RL (37.92%) and ℳ_{LMQ} + SFT + RL (56.66%). No explanation is given for why ℳ_{LDQ} and ℳ_{SHQ} were excluded from the RLVR stage. The SFT-stage results (Table 4, Table 8) show a clear ordering: ℳ_{LMQ} + SFT (50.95%) > ℳ_{LDQ} + SFT (46.70%) > ℳ_{SHQ} + SFT (37.33%) > ℳ_{base} + SFT (26.62%). It is plausible but unverified that RLVR preserves this ordering (which would mean the ℳ_{LDQ} vs. ℳ_{base} RL gap is smaller than the 18.74 points reported for ℳ_{LMQ}, and the ℳ_{SHQ} vs. ℳ_{base} RL gap smaller still). The paper's claim of a "19% average gain" (from the abstract and Section 1) is specifically the ℳ_{LMQ} vs. ℳ_{base} comparison and should not be interpreted as the average gain from any reasoning pretraining—a distinction the paper does not always make clear.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation or explain why the two intermediate variants were excluded. Given that the RLVR phase uses one epoch of GRPO on a fixed dataset with fixed hyperparameters, the computational cost of running two additional variants would have been incremental relative to the pretraining cost already invested. The omission weakens the paper's ability to make fine-grained claims about which kind of reasoning pretraining is most valuable when the full training pipeline (including RL) is the metric of interest.

---

The Fixed 80/20 Blend Ratio Leaves the Dose-Response Relationship Uncharacterized

The assumption or constraint. All reasoning-pretrained models use an identical blending ratio: 80% 𝒟_{base} (general corpus) and 20% 𝒟_{res} (reasoning data). This is a single design point on what is presumably a continuous dose-response curve. The paper provides no evidence about whether 5% reasoning data would capture most of the benefit, whether 50% would be harmful (overspecialization), or whether different types of reasoning data have different optimal proportions. The choice of 20% appears to be a reasonable but arbitrary heuristic—large enough to have measurable effects, small enough to avoid turning the model into a reasoning specialist—and is not justified through ablation or prior literature.

The consequence. A practitioner implementing the paper's recommendations faces an immediate unanswered question: "how much diverse reasoning data should I include in pretraining?" The asymmetric principle tells them what kind of data to use but not how much. If the benefit of diverse reasoning data saturates at 5% of the pretraining mix, adding 20% wastes 15% of the pretraining budget that could have been allocated to other data sources or to expanding the base corpus. Conversely, if the benefit continues to scale beyond 20%, the paper's results represent a lower bound and practitioners should invest even more heavily in reasoning data curation. The 20% figure also confounds with the SFT budget: since all models receive the same 4.8M-sample SFT phase, the total reasoning data exposure varies across pretraining conditions. A model with 5% reasoning data in pretraining might achieve similar final performance to one with 20% if both are followed by the same SFT, which would substantially change the cost-benefit calculus of pretraining data curation.

What evidence exists in the paper. None. No experiment varies the blend ratio. The paper's optimization framing (Equation 2, ℬ = |𝒟_{res}^{PT}| + |𝒟_{res}^{SFT}|) implies that the allocation between phases is the key decision variable, but the pretraining-phase intensity (the proportion of reasoning data within the pretraining mix) is held fixed. The total token count of reasoning data in pretraining differs across ℳ_{LDQ} (336B tokens × 20% = ~67B tokens of reasoning data out of 1T total) and ℳ_{SHQ} (~12B tokens equivalent × 20% = ~2.4B tokens equivalent), but both use the same 20% blend ratio—so the "dose" of reasoning data in pretraining is confounded with the type of reasoning data. This makes it impossible to determine whether ℳ_{LDQ} outperforms ℳ_{SHQ} because diverse data is better, because more reasoning tokens are better, or both.

Mitigation status. Not addressed. The paper does not discuss the blend ratio as a design choice requiring justification or future investigation. Since varying the blend ratio would multiply the already-large experimental matrix (four reasoning datasets × four blend ratios = 16 pretraining runs before SFT and RLVR), this limitation is understandable given computational constraints, but it restricts the practical precision of the paper's guidance.

---

No Statistical Significance Reporting for Any Comparison

The assumption or constraint. The paper reports all results as point estimates (mean accuracies) without standard deviations, confidence intervals, or formal hypothesis tests for any comparison. The evaluation protocol includes multiple runs to reduce sampling variance (16 runs for AIME benchmarks, 4 runs for others) but does not report the variance of these runs or use them to quantify uncertainty. Each base model is pretrained exactly once (no replicates with different random seeds for weight initialization or data ordering), so between-model differences could reflect noise from random training variation rather than true effects of the data composition.

The consequence. Several of the paper's interpretive claims depend on the magnitude of small differences whose statistical reliability is unknown. The central evidence for the latent effect—the near-zero gap between ℳ_{LMQ} (64.07%) and ℳ_{LDQ} (64.09%) at the base model stage versus the +4.25 point gap after SFT—relies on interpreting the −0.02 point base-model difference as "essentially zero." Without a confidence interval, we cannot distinguish between "these models have identical base-model performance" and "the true difference is +1.5 points but this experiment cannot reliably detect it." If the true base-model gap were +1.0 points in favor of ℳ_{LMQ}, the post-SFT amplification to +4.25 points would still be interesting but the "latent effect" framing (zero → positive) would be inaccurate—the effect would be partially visible at the base model stage and amplified by SFT. Similarly, the finding that doubling mixed-quality SFT data yields "negligible improvement" (+0.15 points average, Table 6) requires confidence that +0.15 is within the noise floor, while the −4.92 point math drop must be reliably distinct from zero to support the claim of active harm. Without variance estimates, these judgments are subjective.

What evidence exists in the paper. None. The evaluation methodology section (3.2) describes the number of evaluation runs and the benchmarks but does not mention variance, standard deviation, confidence intervals, or significance testing. The large per-benchmark tables in Appendices A and B (Tables 7, 8) report point estimates to two decimal places without error bounds. No footnote or appendix mentions the variability of the reported numbers.

Mitigation status. Not addressed. This is the paper's most straightforward methodological limitation and is unusual for an empirical study making comparative claims. The computational cost of pretraining replicates (the dominant source of between-model variance) is prohibitive, but within-model evaluation variance could be reported from the 4–16 runs already collected at negligible additional cost. Confidence intervals for the AIME benchmarks (where only 30 problems per evaluation set create high inherent variance) would be particularly informative, as the AIME results are among the paper's most dramatic findings (e.g., 45.21% vs. 12.29% on AIME-24 after RLVR, Table 3) and also the most vulnerable to small-sample instability. The paper does not acknowledge the absence of variance reporting as a limitation, nor does it provide the raw evaluation-run data that would allow readers to compute confidence intervals independently.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the conversation around LLM training data strategy from a phase-agnostic quality optimization problem to a phase-dependent allocation problem where the marginal utility of data characteristics inverts between pretraining and SFT. Prior to this work, the field had independently developed domain-specific heuristics—pretraining practitioners emphasized data filtering at the trillion-token scale, while post-training researchers converged on the LIMA-inspired principle that small amounts of high-quality data suffice for alignment (Zhou et al., 2023). These were treated as separate conversations about separate stages, with no framework for understanding them as manifestations of a single underlying dynamic: that the same data characteristic can be beneficial in one phase and harmful in another.

The paper's asymmetric allocation principle—diversity and scale dominate in pretraining (+9.09 point gain for ℳ_{LDQ} over ℳ_{SHQ} at the base model stage, Table 1), while quality dominates in SFT (+13.45 point gain for 𝒟_{SHQ} over 𝒟_{LDQ} during SFT, Table 5)—provides the first empirical unification of these previously disconnected observations. It is not a paradigm shift in the sense of introducing a new architecture or training algorithm, but it is a conceptual reframing with direct practical consequences for how organizations design training pipelines. The unification resolves the apparent contradiction between "pretraining needs massive diverse data" and "post-training needs small curated data" by showing that both are correct, but each describes a different phase where different optimization pressures apply.

More specifically, the paper changes the landscape in four ways:

First, it establishes that pretraining data decisions are strategically decisive rather than merely preparatory. The catch-up refutation (Table 4: ℳ_{base} + SFT[2×] achieves 34.01% vs. ℳ_{SHQ} + SFT at 37.33% and ℳ_{LMQ} + SFT at 50.95%) demonstrates that SFT cannot compensate for a weak pretraining foundation. This overturns the plausible—and widely held—assumption that generalist base models could be specialized for reasoning through aggressive post-training. The implication is that organizations building reasoning-capable models must make reasoning data curation a first-class concern from the earliest stages of model development, not a downstream afterthought. This shifts pretraining from a relatively generic phase (produce a general-purpose base model that post-training will shape) to the primary determinant of the final performance ceiling.

Second, it introduces the latent effect as a diagnostic concept that changes how pretraining ablations should be evaluated. The near-zero gap between ℳ_{LMQ} (64.07%) and ℳ_{LDQ} (64.09%) at the base model stage, followed by a +4.25 point emergence of advantage after SFT (Table 4), establishes that base model benchmarks systematically underestimate the value of certain pretraining data choices. This is a methodological caution with broad implications: the standard practice of running downstream evaluations immediately after pretraining to guide data mixture decisions may be systematically misleading for data whose primary contribution is to create favorable conditions for alignment rather than to improve immediate capabilities. If the field had evaluated ℳ_{LMQ} only at the base model stage, the natural conclusion would have been that adding high-quality data to diverse pretraining provides no benefit—a conclusion the SFT results refute. This finding redirects attention toward developing evaluation protocols that surface latent capabilities, and it casts doubt on the completeness of any pretraining data ablation that stops at the base model stage.

Third, it identifies naive SFT scaling as actively harmful, not merely inefficient. The finding that doubling mixed-quality SFT data causes a −4.92 point drop in math accuracy while adding 0.4% more high-quality data yields consistent improvements (Table 6) transforms the conversation around SFT data scaling. Prior work (Zhao et al., 2024; Prabhakar et al., 2025) had established that quality matters for SFT, but this was framed positively—"small amounts of high-quality data work well"—rather than negatively—"large amounts of low-quality data destroy capability." The paper's framing of low-quality SFT data as counterproductive rather than neutral is stronger and carries a different practical imperative: it means that scaling SFT data without stringent quality filtering is not a cost-inefficient but harmless strategy; it is a strategy that actively erases pretraining investments. For organizations considering whether to include large-scale but noisy instruction-formatted data in their SFT mix, the paper provides a clear answer: don't, unless you have verified that the quality threshold is met.

Fourth, it demonstrates that data redundancy between pretraining and SFT is beneficial, not harmful. The finding that ℳ_{SHQ} + SFT[𝒟_{SHQ}] significantly outperforms ℳ_{base} + SFT[𝒟_{SHQ}] (Appendix C, Figure 2) refutes the concern—grounded in the continual fine-tuning literature (Luo et al., 2025b)—that using the same data in both stages would cause overfitting or brittleness. Instead, the opposite occurs: strategic redundancy reinforces foundational skills, with pretraining serving as slow integration of reasoning patterns and SFT acting as reinforcement on an already-prepared substrate. This changes the default assumption about data reuse between stages from "avoid if possible" to "consider leveraging intentionally," particularly for capabilities that benefit from consolidation across different optimization regimes.

The paper also makes certain research directions less attractive. The demonstration that SFT cannot compensate for weak pretraining (Table 4) reduces the expected return on investment in post-training-centric approaches to reasoning—such as increasingly sophisticated SFT data curation pipelines applied to generalist base models—relative to approaches that front-load reasoning data. Similarly, the finding that beam-search-like optimization of reasoning data quality at the SFT stage (filtering for long reasoning traces, Table 9) yields large gains while pretraining benefits from broad exposure suggests that efforts to develop universal "data quality" metrics applicable across all training phases are likely to produce misleading results. A metric that predicts SFT utility may be uncorrelated with or even inversely related to pretraining utility, and vice versa.

---

Follow-Up Research This Work Enables

Characterizing the dose-response relationship between reasoning data proportion in pretraining and downstream capability. The paper uses a single 80/20 blend ratio for all reasoning-pretrained models, leaving the functional form of the benefit curve completely uncharacterized. A direct follow-up would pretrain a series of models with the same 𝒟_{LDQ} reasoning dataset at blend ratios of 5%, 10%, 20%, 35%, and 50% (all to the same total token count, with the base corpus adjusted to maintain 1T total tokens), then evaluate each after identical SFT and RLVR stages. The key measurements would be: (1) whether the benefit saturates early (e.g., 5% captures >80% of the 20% gain, making the practical recommendation "include a modest amount of diverse reasoning data"), (2) whether there is an optimal proportion beyond which reasoning performance degrades due to displacement of general knowledge from the base corpus, and (3) whether the latent effect observed for ℳ_{LMQ} (adding quality to diversity) emerges only above some threshold diversity level. This experiment would transform the paper's binary finding ("20% diverse reasoning data helps") into a continuous dose-response curve that directly informs practical budget allocation.

Testing whether the asymmetric principle generalizes across model scales and architectures. The paper's entire empirical foundation rests on a single 8B hybrid Mamba-transformer model. The most important stress-test of its conclusions is whether the phase-dependent utility of diversity and quality persists at larger scales (70B, 405B) and in pure transformer architectures. A 70B-scale replication would train two models: one with 𝒟_{LDQ}-style diverse pretraining data and one with 𝒟_{SHQ}-style high-quality pretraining data (both at the same blend ratio, matched for total tokens), then cross them with high-quality and diverse SFT data and evaluate after RLVR. The critical question is whether the asymmetry flattens at larger scales—do larger models, with their greater capacity to absorb noisy data productively, benefit less from the diversity/quality distinction? Or do they benefit more, because the richer representations they can form make the pretraining data characteristics even more consequential? The 8B finding that diversity provides +9.09 points in pretraining might be a lower bound (if larger models extract more value from broad exposure) or an upper bound (if larger models can compensate for narrow pretraining through in-context learning during SFT). This experiment would also partially address the generalization concern about the Mamba-transformer hybrid architecture, though a pure dense transformer comparison would be needed to fully isolate architectural effects.

Developing lightweight predictors of phase-dependent data utility to reduce the cost of applying the framework. The paper's experimental design requires full-scale pretraining to evaluate data allocation strategies—a cost that makes the framework impractical for most organizations. A natural follow-up is to develop cheaper proxies that predict how valuable a given dataset will be in each training phase. One concrete approach: train small proxy models (e.g., 100M–300M parameters) on the same data mixtures as the paper's 8B models, then measure whether the relative ordering of data strategies at small scale predicts the 8B ordering. If a 300M model pretrained with 20% 𝒟_{LDQ} similarly outperforms one with 20% 𝒟_{SHQ} at pretraining, and shows the same SFT quality-dominance reversal, then the 300M scale becomes a cheap testbed for data allocation decisions. Additional proxies could include: (1) measuring the pretraining loss curvature or Hessian spectrum of small models trained on different data mixtures to predict latent effects, (2) using representation similarity metrics (CKA, centered kernel alignment) between models trained with different data mixtures at small scale to predict whether the differences will amplify or wash out during SFT, (3) developing "data cards" that characterize a reasoning dataset along the dimensions the paper identifies as consequential (domain diversity entropy, average reasoning trace length, teacher model quality distribution) and training a predictor from the paper's results to estimate phase-specific utility for new datasets. The goal would be to reduce the cost of applying the paper's framework from "pretrain multiple 8B models" to "run a set of diagnostic experiments at 1/30th the scale and apply a learned mapping."

Investigating the mechanism of the latent effect through controlled representational analysis. The paper observes that adding high-quality data (𝒟_{SHQ}) to diverse pretraining (𝒟_{LDQ}) produces no immediate base-model improvement but a +4.25 point gain after SFT (Table 4), and names this a "latent effect" without mechanistic explanation. A mechanistic follow-up would take the ℳ_{LMQ} and ℳ_{LDQ} models (which differ only in the inclusion of high-quality pretraining data) and compare their internal representations before and after SFT using tools from mechanistic interpretability. Specific analyses: (1) Probe whether reasoning-relevant circuits (e.g., attention heads that attend to numerical entities, MLP neurons that activate for mathematical operators) are more structured or sparsely encoded in ℳ_{LMQ} even at the base model stage, even though this structure does not manifest in improved few-shot accuracy. (2) Measure whether ℳ_{LMQ} shows less representational interference between reasoning and non-reasoning inputs—i.e., whether the model has learned to separate reasoning circuits from general language circuits, making SFT less disruptive to either. (3) Track how SFT updates change the representations in both models: does SFT primarily amplify existing reasoning circuits in ℳ_{LMQ} while needing to restructure them in ℳ_{LDQ}, explaining the differential gain? (4) Ablate the high-quality data by domain (math, code, science) to determine whether the latent effect is domain-specific or domain-general—if adding high-quality math data improves science reasoning after SFT, the mechanism is likely representational abstraction rather than domain-specific knowledge injection. This work would transform the latent effect from an empirical observation into an understood phenomenon with predictive power.

Exploring whether the asymmetric principle extends to multimodal and multilingual reasoning training. The paper's reasoning data is exclusively text-based and English-dominant. A natural extension would test whether the phase-dependent value of diversity and quality holds for other modalities and languages. For multimodal reasoning (e.g., vision-language models trained on interleaved image-text reasoning data), the "diversity" axis becomes more complex—it could mean diversity of visual domains (natural images, diagrams, charts, screenshots), diversity of reasoning types (spatial reasoning, causal reasoning from visual evidence, mathematical reasoning from figures), or diversity of task formats. A follow-up would pretrain vision-language models with reasoning data injection analogous to the paper's design, crossing diverse/mixed-quality visual reasoning data against narrow/high-quality data at the pretraining and instruction-tuning stages, and measure whether the asymmetric principle holds in the multimodal regime. For multilingual reasoning, the key question is whether reasoning data in one language provides latent benefits for reasoning in another language after SFT—a test of whether the representations built during pretraining are genuinely abstract (language-independent logical patterns) or language-specific (English reasoning templates). This would involve pretraining with diverse multilingual reasoning data versus English-only high-quality data, then fine-tuning in multiple languages and measuring cross-lingual transfer of reasoning capability.

Testing the interaction between pretraining reasoning data and reinforcement learning algorithms. The paper's RLVR results (Table 3) compare only the two extreme pretraining variants (ℳ_{LMQ} and ℳ_{base}) and use only GRPO with fixed hyperparameters. A deeper investigation would cross the full set of pretraining data strategies (ℳ_{base}, ℳ_{SHQ}, ℳ_{LDQ}, ℳ_{LMQ}) with multiple RL algorithms (GRPO, PPO, rejection sampling fine-tuning, iterative DPO) and multiple RL hyperparameter settings (KL penalty coefficients, numbers of rollouts per prompt, learning rates) to characterize how pretraining data quality interacts with the RL optimization landscape. The paper's finding that the pretraining advantage compounds dramatically through RL (+18.74 point gap vs. +9.30 at SFT) suggests that reasoning-rich pretraining makes RL more effective, but the mechanism is unknown. Does better pretraining primarily improve exploration (the model generates higher-quality rollouts, providing better training signal), or credit assignment (the model's value function is better calibrated, making advantage estimation more accurate), or policy improvement (the model's parameter space is structured such that RL updates more reliably improve reasoning)? Each suggests different RL algorithm design choices: if exploration is the bottleneck, increasing the number of rollouts per prompt would disproportionately help weaker pretrained models; if credit assignment is the bottleneck, better reward shaping or critic training would help; if policy improvement is the bottleneck, the pretraining data strategy is the primary lever and RL algorithm choice is secondary. Measuring RL learning curves (accuracy per training step) for each pretraining variant, rather than just terminal accuracy, would reveal whether the compounding effect reflects faster convergence, higher asymptote, or both.

---

Practical Applications and Downstream Use Cases

Cost-efficient pretraining data curation for reasoning-focused LLMs. For organizations building LLMs intended for reasoning-heavy applications (mathematical problem-solving, scientific Q&A, code generation, logical analysis), the paper's asymmetric principle provides an actionable blueprint for allocating data curation budget. The finding that diverse, mixed-quality reasoning data provides a +9.09 point base model gain over narrow high-quality data during pretraining (Table 1: ℳ_{LDQ} at 64.09% vs. ℳ_{SHQ} at 54.98%), while the opposite holds during SFT (Table 5: +13.45 point advantage for high-quality SFT data over diverse mixed-quality data), implies a two-phase investment strategy. During pretraining, prioritize scale and domain breadth: incorporate large reasoning corpora (e.g., 336B tokens as in 𝒟_{LDQ}) even if individual example quality is heterogeneous. The cost of curating high-quality pretraining data (expensive teacher model distillation, human verification, careful filtering) may be better deferred to the SFT phase, where quality's marginal utility is highest. Conversely, SFT budget should be concentrated on a smaller amount of meticulously curated, long-chain-of-thought examples—the paper shows that even a 0.4% increase in dataset size concentrated on high-quality reasoning traces yields measurable gains (Table 6, 𝒟'_{ALF} at 43.04% vs. 𝒟_{ALF} at 42.66%), while doubling mixed-quality data causes active regression in math (−4.92 points). The concrete financial implication: a team with a fixed data curation budget should allocate the majority of their pretraining curation spend to acquiring and lightly filtering large, diverse reasoning corpora, and reserve their expensive quality-assurance processes (human review, teacher model generation, long-CoT verification) for the SFT dataset, where each dollar spent on quality improvement yields outsized returns.

Staged model release strategies with reasoning as a differentiating axis. The paper's finding that reasoning-rich pretraining establishes latent capabilities unlocked by SFT (Table 4: the ℳ_{LMQ} vs. ℳ_{LDQ} gap of +4.25 points emerges only after SFT) has implications for organizations that release models in stages (base model → instruct model → reasoning model). If base model benchmarks don't reflect the latent value of reasoning pretraining data, a company evaluating pretraining data strategies based on base-model metrics would systematically undervalue diverse reasoning data—mistakenly concluding that the ℳ_{LDQ} and ℳ_{LMQ} variants are equivalent, and potentially selecting the cheaper-to-curate option. The consequence would be a base model that looks competitive on standard benchmarks but underperforms its potential after SFT, leaving capability on the table that cannot be recovered without retraining. The practical recommendation is to evaluate pretraining data strategies using at least a lightweight SFT probe—fine-tune candidate base models on a small high-quality reasoning dataset and compare post-SFT performance before committing to the full-scale pretraining run. This adds cost to the evaluation pipeline but prevents decisions that look correct at the base model stage but prove suboptimal after alignment.

Mitigating capability regression during SFT data scaling for production models. For teams maintaining production LLMs that undergo periodic SFT updates (e.g., quarterly fine-tuning on new instruction data, domain-specific adaptation), the paper's finding that naive SFT scaling can actively harm reasoning (Table 6: −4.92 point math drop when doubling mixed-quality data) provides a concrete warning and mitigation strategy. A production team planning to expand their SFT dataset to cover new domains or task formats should not simply concatenate the new data with the existing SFT corpus. Instead, they should: (1) evaluate whether the new data meets the quality threshold established by their existing high-quality SFT data (using answer length and teacher model provenance as rough proxies, per the 𝒟_{ALF} ablation in Table 9), (2) isolate the new data in a controlled A/B SFT experiment against a small reasoning-pretrained model (even a 1B-parameter proxy) to verify it does not cause reasoning regression before deploying at scale, and (3) consider maintaining separate SFT recipes for reasoning capability (high-quality, long-CoT data) versus breadth capabilities (diverse task formats, instruction following), since the paper shows these objectives trade off against each other (Table 5: 𝒟_{LDQ} SFT improves instruction following to 54.61% but reduces math to 32.28%, while 𝒟_{SHQ} SFT does the opposite). The cost of not following this protocol is the silent erosion of carefully built reasoning capabilities with each SFT update—a regression that may go unnoticed if reasoning benchmarks aren't prominently monitored.

Guiding open-source reasoning model development with principled data allocation. The open-source LLM community has increasingly focused on reproducing proprietary reasoning model capabilities (e.g., OpenThoughts, DeepSeek-R1 reproductions, OpenMathReasoning). These efforts typically involve assembling reasoning datasets from multiple sources (public math corpora, synthetic long-CoT generations, distilled outputs from frontier models) and face the allocation problem the paper formalizes: given a fixed total budget of collected reasoning data, how should it be split between continued pretraining/mid-training and SFT? The paper's results directly inform this decision. The community should allocate large-scale, diverse but potentially noisier reasoning corpora (e.g., web-scraped math and code data, synthetic data from smaller teacher models) to the continued pretraining or mid-training phase, where diversity's benefit dominates (+9.09 point pretraining gain from diversity in Table 1). The most expensive, highest-quality data (distillations from the strongest available teacher models, human-verified long-CoT traces) should be reserved for the final SFT phase, where quality's marginal utility is highest (+13.45 point SFT gain from quality in Table 5). The paper also suggests that using the same high-quality data in both phases is not wasteful but reinforcing (Appendix C, Figure 2), so the highest-quality distillation data should appear in both the mid-training and SFT mixes—but the SFT phase should be dominated by quality, while the mid-training phase can tolerate a broader mix. This concrete allocation recipe, derived from a systematic study rather than intuition, could improve the efficiency of the open-source reasoning model development pipeline by directing scarce high-quality data curation effort to the phase where it yields maximal returns.

---

When to Prefer This Method

The paper does not propose a specific "method" or algorithm to be preferred over named alternatives. Rather, it provides an empirical framework and principle for data allocation decisions. The relevant decision rule is not "use this method vs. that method" but rather "adopt this data strategy vs. a phase-agnostic strategy." The paper's evidence supports the following decision guidance:

Adopt phase-dependent, asymmetric data allocation (diversity-heavy pretraining, quality-heavy SFT) when:

• You are training an LLM from scratch or continuing pretraining from an existing checkpoint, and the downstream application involves reasoning-intensive tasks (mathematical problem-solving, scientific reasoning, code generation). The paper demonstrates compounding gains culminating in a +18.74 percentage point lead on expert-level benchmarks after RLVR (Table 3) for the strongest pretraining configuration versus no reasoning pretraining, with the gap widening at each subsequent training stage rather than closing.

• You have access to large-scale reasoning corpora of heterogeneous quality (e.g., web-scraped math/code data, synthetic data from diverse teacher models, public reasoning datasets) and a smaller amount of carefully curated, long-chain-of-thought reasoning data. The asymmetric principle provides clear allocation guidance: the large, diverse corpora should be prioritized for pretraining, where diversity provides an +9.09 point advantage over narrow high-quality data (Table 1), while the curated data should be reserved for SFT, where quality provides a +13.45 point advantage over diverse mixed-quality data (Table 5).

• You are willing to invest in pretraining data curation with the understanding that some benefits may be latent—invisible at the base model stage but unlocked by subsequent alignment. The paper's latent effect finding (Table 4: ℳ_{LMQ} gains +4.25 points over ℳ_{LDQ} only after SFT) implies that pretraining data decisions should be evaluated using post-SFT metrics rather than base model benchmarks, which requires a different evaluation protocol from standard practice.

Default to a phase-agnostic, quality-maximizing strategy (high-quality data at all stages) only when:

• Your downstream application does not involve structured reasoning (e.g., the primary use case is open-ended dialogue, content summarization, or creative writing, where the reasoning depth-vs-breadth tradeoff observed in the paper does not apply). The paper's finding that general-purpose reasoning benchmarks (gprPT avg, Table 1) show minimal sensitivity to reasoning pretraining data (all models within 1.2 points) suggests that the asymmetric principle is specific to reasoning-intensive capabilities, and quality-maximizing strategies (as recommended by LIMA and related work) may be sufficient for non-reasoning applications.

• The computational cost of running diverse reasoning data through pretraining (which involves training on hundreds of billions of tokens of reasoning corpora) exceeds your budget, and your SFT-only baseline already meets performance requirements. The paper's catch-up refutation shows that SFT cannot compensate for absent reasoning pretraining (Table 4), but if your performance targets are modest, the SFT-only approach may be adequate despite being suboptimal in absolute terms. The paper does not characterize whether intermediate approaches—such as mid-training on diverse reasoning data after standard pretraining, rather than injecting it from the start—capture most of the benefit at lower cost, which would be a relevant consideration for budget-constrained deployments.