Thinking Augmented Pre-training

ArXiv: 2509.20186

🎯 Pitch

Simply adding auto-generated chain-of-thought to pre-training data makes LMs up to three times more data-efficient. The key insight is that many high-quality tokens, like a final numerical answer, are essentially impossible to learn from context alone unless the underlying reasoning is spelled out first.

1. Executive Summary

This paper introduces Thinking augmented Pre-Training (TPT), a data engineering method that augments pre-training text with automatically generated reasoning trajectories to improve the learnability of high-quality but difficult tokens. Experiments across pre-training from scratch, mid-training, and supervised fine-tuning configurations—using models from 1.5B to 8B parameters trained on up to 100B tokens from MegaMath-Web-Pro-Max and FineWeb-Edu—demonstrate that TPT improves data efficiency by a factor of 3× (achieving comparable base model performance with one-third the training tokens) and lifts challenging reasoning benchmarks by over 10% for a 3B model after post-training. The method operates by having an off-the-shelf LLM generate step-by-step thinking trajectories (analogous to chain-of-thought reasoning applied to arbitrary documents rather than just problem-solving prompts) that decompose complex tokens into simpler intermediate steps, establishing that allocating additional training compute to difficult tokens via synthetic reasoning augmentation substantially outperforms vanilla next-token prediction—but only when the base training data contains learnable content that the model cannot efficiently absorb through direct exposure alone.

2. Context and Motivation

The Core Problem: Valuable Tokens Are Difficult to Learn in a Single Prediction Step

The fundamental problem this paper addresses is deceptively simple yet has profound implications for how we train large language models: some of the most valuable tokens in a training corpus are the hardest for a model to learn through standard next-token prediction. The paper opens with a concrete example (Figure 1b): the numeric answer "890" in the statement "The largest positive integer n for which n³ + 100 is divisible by n + 10 is 890." To understand why this token appears in the training data, an LLM must implicitly reconstruct the reasoning chain involving the Remainder Theorem, polynomial division, divisor properties, and algebraic manipulation — all from a single token that is the output of a multi-step human reasoning process. When model capacity is limited (relative to the complexity of the underlying rationale), the model faces an uncomfortable choice: either memorize the token (which does not generalize to other instances of the same reasoning pattern) or fail to learn it effectively. The paper introduces the term "learnability" to capture this tension — certain correct and valuable tokens have low learnability under standard pre-training objectives because the compression required to predict them from context is too extreme for a given model capacity.

This is not an edge case. The paper argues (Section 1) that as pre-training corpora grow to encompass the entire web, the proportion of such "difficult but valuable" tokens increases. Web-crawled data contains everything from simple declarative statements to dense mathematical derivations, complex code implementations, and nuanced scientific arguments. The most information-rich content — the content that would most improve a model's reasoning capabilities — is precisely the content where single-token prediction is least effective as a training signal. The paper observes that modern data engineering pipelines invest enormous effort in filtering, deduplication, and quality-based selection, but none of these operations address the fundamental mismatch between the learning objective (predict the next token) and the cognitive depth embedded in many tokens.

Why This Gap Matters: The Data Wall and the Compute Scaling Mismatch

The significance of this problem crystallizes when you consider two converging trends in LLM development that the paper highlights in its introduction:

Trend 1: Data exhaustion. The paper states directly that "the pool of human-authored, organically generated data on the web is finite and has been largely exhausted by existing frontier models" (Section 1). Recent open-source models are trained on over 10 trillion tokens (Dubey et al., 2024; Yang et al., 2025), which represents a substantial fraction of all high-quality text ever written. While compute for training continues to grow — and is projected to continue this trajectory — the supply of new training data from human sources has effectively plateaued. This creates a bottleneck that cannot be solved by simply crawling more web pages; the frontier of LLM development is bumping against a data wall.

Trend 2: Training compute continues to scale. The paper references the scaling laws (Kaplan et al., 2020) which established that both model size and training tokens must scale together for optimal performance. If data is finite but compute budgets keep growing, the only options are to train for multiple epochs (which yields diminishing returns and risks overfitting) or to find ways to extract more value from the same underlying data. The paper positions TPT squarely in the latter category.

The practical stakes are described through multiple deployment scenarios in the paper:

• Pre-training from scratch: If you have a fixed compute budget and a finite dataset of raw documents, TPT effectively increases the "useful information per raw byte" extracted during training. Table 1 shows that a TPT-8B model trained on 100B tokens achieves GSM8k accuracy of 50.1% vs. 19.2% for the vanilla model — a 2.6× improvement — using the same underlying documents but augmented with thinking trajectories.

• Data-constrained regimes (Section 3.2): When the total raw document tokens are artificially limited to 10B (simulating a scenario where high-quality data is genuinely scarce), the vanilla model plateaus after the first epoch while TPT continues improving through its single epoch. This is the regime where the paper claims "LLM scaling under constrained data [is] a critical concern" (Section 3.2), citing Muennighoff et al. (2023) on data-constrained language modeling.

• Mid-training on existing checkpoints (Section 3.3): Organizations with deployed models can further improve reasoning capabilities without training from scratch. The paper demonstrates that even strong models like Qwen2.5-Math-7B benefit substantially from TPT mid-training (e.g., AIME24 improves from 50.5% to 57.5% after TPT mid-training and SFT, Table 3).

The theoretical significance extends beyond practical data efficiency. The paper draws an explicit analogy to test-time scaling (Jaech et al., 2024) — the phenomenon where allowing models to generate longer chains of reasoning at inference improves performance. The key insight is that this principle can be applied during training: just as difficult problems benefit from more inference compute (longer chain-of-thought), difficult tokens in training data benefit from more training compute allocated through thinking trajectories. This creates a conceptual bridge between the test-time compute literature and data engineering for pre-training that did not previously exist in the literature.

Prior Approaches and Their Limitations

The paper contextualizes its contribution against several existing research threads, each of which addresses parts of the data efficiency problem but falls short in specific ways that TPT aims to resolve.

Data selection and token-level prioritization. Prior work on data efficiency has largely operated through a filtering lens: identify which training examples or tokens are most valuable and train only on those. The paper cites Lin et al. (2024) who propose training exclusively on "valuable tokens that are learnable but are not yet learned," and Mindermann et al. (2022) who develop a prioritization framework based on similar principles. The gap in these approaches is that they treat tokens as atomic units — they select which tokens to train on but do not modify how those tokens are presented to the model. A token representing the output of a complex reasoning chain remains just as difficult to learn regardless of whether it's selected for training. The paper's position is that selection alone cannot solve the learnability problem; you need to change the training signal itself by decomposing difficult tokens into intermediate steps.

Synthetic data generation for pre-training. Several prior works have generated synthetic data to augment or replace web-crawled pre-training corpora. The paper specifically discusses:

• Phi series models (Gunasekar et al., 2023): Heavily relied on "textbook-like" synthetic data generated by GPT-3.5 and GPT-4. This approach demonstrated that synthetic data can produce strong models, but it requires generating entirely new training documents from scratch — a fundamentally different approach from TPT, which augments existing documents rather than replacing them. The paper positions TPT as orthogonal and complementary: you could generate synthetic documents and augment them with thinking trajectories.

• Text rewriting approaches (Maini et al., 2024; Nguyen et al., 2025; Allen-Zhu & Li, 2024): These methods transform raw text through paraphrasing, restructuring, or other surface-level modifications to improve data quality. The paper explicitly states that TPT is "orthogonal to rewriting based approaches" and operates on a different axis — rather than improving the linguistic quality or readability of the text, TPT adds a layer of interpretive reasoning that explains the cognitive steps underlying the original content.

Hidden thought injection for training. The most directly comparable prior work falls into a category the paper describes as "mining hidden thoughts" to enhance model training:

• Reasoning CPT (Ishibashi et al., 2025): Proposes generating hidden thoughts to enhance continual pre-training, demonstrating improvements on MMLU. The paper acknowledges this as closely related but identifies several limitations: Reasoning CPT only scales to approximately 150M training tokens (vs. TPT's 100B), operates exclusively in a continual training setup without post-training evaluation, and its evaluation is limited to base models on MMLU — it doesn't assess whether the benefits transfer to instruction-tuned models on reasoning-heavy benchmarks.

• BoLT (Ruan et al., 2025): Similarly generates latent thoughts for training but focuses specifically on mathematical tasks using a 1B parameter model and 8B training tokens. BoLT introduces an EM (expectation-maximization) algorithm to iteratively bootstrap thought generation — generating thoughts, training on them, then using the improved model to generate better thoughts. The paper argues that this iterative approach adds significant complexity and computational cost without necessarily improving results. TPT's key methodological claim is that a single pass of thinking trajectory generation (no iterative bootstrapping) suffices for substantial gains.

• ToW (Xu et al., 2025): Injects thoughts at the word level to explain latent reasoning during continual pre-training. This is a finer-grained approach compared to TPT's document-level augmentation.

The paper's critique of these prior approaches centers on three dimensions: scale (prior work operates at 150M–8B tokens vs. TPT's 100B), domain scope (prior work evaluates on narrow benchmarks, primarily MMLU or math-specific tasks, vs. TPT's evaluation across math, code, and general reasoning), and methodological complexity (prior work requires custom fine-tuning of thought-generating models, EM-like iterative procedures, or prompt engineering for specific document types, vs. TPT's single-prompt, single-pass approach).

Reinforcement learning for pre-training (RPT). The paper discusses RPT (Dong et al., 2025) as operating "in a similar spirit" — applying reinforcement learning to the next-token prediction objective during pre-training so the model explores and learns from its own generation mistakes. The critical difference is computational efficiency: RPT requires online rollouts during training (the model generates, receives feedback, and updates in a loop), which the paper describes as "substantially more compute." TPT, by contrast, generates thinking trajectories offline before training begins, and then trains on the augmented dataset using standard next-token prediction loss. This decoupling of trajectory generation from model training makes TPT far more scalable — the data generation is a one-time cost independent of the training pipeline.

Chain-of-Thought and reasoning elicitation. The paper connects to the broader chain-of-thought (CoT) literature (Wei et al., 2022) and subsequent extensions like tree-of-thought (Yao et al., 2023), but draws a key distinction: these methods apply reasoning at inference time to improve task performance, while TPT applies reasoning during training to improve data learnability. The thinking trajectories generated for TPT are not designed to solve a specific problem the model will later face at inference — they are designed to make the original training text more comprehensible and learnable. The connection to OpenAI o1 and DeepSeek-R1 (Jaech et al., 2024; Guo et al., 2025) is more direct: these models are fine-tuned with reinforcement learning to generate long thinking trajectories at inference time. TPT essentially brings this same reasoning-augmentation principle to the pre-training data itself, using open-source reasoning models (DeepSeek-R1-Distill-Qwen-7B) as the thought generator.

How This Paper Positions Itself

The paper situates TPT not as a replacement for existing data engineering practices but as a complementary, orthogonal enhancement that operates at a different stage of the pipeline. The introduction frames the contribution as:

"Orthogonal to the development of enhanced data curation pipelines, a critical but underexplored dimension is the maximization of utility from existing data." (Section 1)

This framing is deliberate. Modern data engineering pipelines (the paper cites FineWeb's pipeline as a representative example) perform operations like parsing, deduplication, heuristic filtering, model-based quality filtering, and domain balancing. TPT slots in after these standard operations — you take whatever high-quality corpus your pipeline produces, then augment it with thinking trajectories. The paper emphasizes this modularity: "TPT is highly scalable as it requires no human annotation and imposes no constraints on document structure" (Section 1, immediately following the method description).

The paper also positions itself at a specific point on the simplicity-sophistication spectrum. Compared to RPT's online RL, BoLT's EM algorithm, or Reasoning CPT's custom fine-tuning of thought-generating models, TPT is deliberately minimal: one prompt template, one forward pass through an off-the-shelf reasoning model, concatenation with the original text, and standard next-token prediction training. Section 5's ablation studies directly test whether more sophisticated alternatives (a custom fine-tuned back-thinking model, prompts with random focus points) improve results, and find they provide only marginal gains. This empirical finding supports the paper's methodological claim that simplicity is not just convenient — it is close to optimal, at least given current thinking-generation models.

Perhaps most importantly, the paper positions its contribution at the intersection of two previously disconnected research communities: data engineering (focused on what training data to use and how to process it) and reasoning/scaling (focused on how models perform inference-time computation). The thinking pattern analysis in Section 4 is designed to bridge these communities by showing that TPT naturally up-samples high-quality data and dynamically allocates training compute proportional to reasoning intensity — effectively providing a training-time analog of test-time compute scaling. This conceptual alignment is the paper's deepest intellectual contribution: it suggests that the principles governing effective inference (allocate more compute to harder problems, decompose complex reasoning into steps) also govern effective training (allocate more training tokens to harder content, decompose complex tokens into intermediate representations).

The three experimental configurations (pre-training under abundant data, pre-training under constrained data, and mid-training from existing checkpoints) are designed to demonstrate that this principle holds across different resource regimes and deployment scenarios. This breadth of validation — rather than novelty of any individual experiment — is the paper's primary rhetorical strategy for establishing the generality of the approach.

3. Technical Approach

3.1 Reader Orientation

The system being built is a data preprocessing pipeline that transforms standard pre-training text documents into augmented training samples containing both the original text and automatically generated reasoning explanations, without modifying the model architecture or training objective. The core problem it solves is that certain valuable tokens in pre-training data—like numeric answers to math problems or conclusions in scientific arguments—are extremely difficult for models to learn through standard next-token prediction because they compress multi-step human reasoning into a single prediction target; the solution reshapes the training data distribution by inserting intermediate reasoning steps that decompose these difficult tokens into sequences of simpler, more learnable sub-predictions, effectively allocating more training compute to the content that benefits most from it.

3.2 Big-Picture Architecture (Diagram in Words)

The TPT system has four major components connected in a sequential pipeline:

1. Raw Document Corpus — the pre-training dataset (MegaMath-Web-Pro-Max and FineWeb-Edu) containing web-crawled text documents covering mathematics, code, and general knowledge content. This is the input that enters the pipeline.
2. Thinking Trajectory Generator — an off-the-shelf instruction-tuned LLM (DeepSeek-R1-Distill-Qwen-7B for mid-training, Qwen3-8B for pre-training from scratch) that takes each document as input and produces a chain-of-thought style analysis explaining the document's content, reasoning step-by-step through complex aspects. This component is queried once per document, offline, before any model training begins.
3. Data Concatenator — a simple formatting step that appends the generated thinking trajectory to the original document, creating the augmented training sample $x = [d; t]$ where $d$ is the original document text and $t$ is the thinking trajectory. This component also handles truncation (documents truncated to 2k tokens, thinking trajectories capped at 8k tokens) and packing into 8k-token training sequences.
4. Standard Next-Token Prediction Training — the model being trained (1.5B to 8B parameters) is trained on these concatenated samples using exactly the same causal language modeling objective as standard pre-training, minimizing cross-entropy loss over all tokens in the augmented sequence with no architectural modifications, no special loss weighting, and no auxiliary objectives.

Information flows strictly left-to-right: raw documents → thinking generator → concatenation → model training. The key design choice is that trajectory generation is completely decoupled from model training—the thinking trajectories are generated once as a preprocessing step, then reused throughout training, making the approach scalable to 100B-token training runs without requiring online generation or reinforcement learning loops.

3.3 Roadmap for the Deep Dive

• The core training objective and data format — I start with exactly what the model is trained on and what loss it minimizes, since understanding the training sample structure is prerequisite to everything else.
• Thinking trajectory generation mechanism — the prompt template, generation hyperparameters, model selection, and what constraints are applied during generation, since the quality and characteristics of these trajectories determine whether the approach works.
• Training configurations and hyperparameters — the three distinct training regimes (pre-training under abundant data, pre-training under constrained data, mid-training), their different resource constraints, and how TPT's data processing interacts with each, since the experimental design depends on these configurations.
• Dynamic compute allocation through thinking length — the mechanism by which harder content naturally receives longer thinking trajectories and therefore more training tokens, since this is the conceptual core of the paper's efficiency claims.
• Supervised fine-tuning integration — how the augmented pre-trained or mid-trained models are subsequently fine-tuned on instruction datasets, since the downstream reasoning benchmarks that demonstrate TPT's value are evaluated after SFT.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical data engineering paper whose core idea is that augmenting pre-training text with automatically generated step-by-step reasoning trajectories improves the learnability of difficult tokens by decomposing them into simpler intermediate steps, thereby increasing the effective data efficiency of LLM training without modifying the model architecture or the next-token prediction objective.

Training Objective and Data Format

The model is trained on concatenated sequences of original documents and their generated thinking trajectories using the standard causal language modeling loss. The augmented training sample is constructed as:

$x = [d; t]$

where $d$ is the original document text (truncated to a maximum of 2,000 tokens) and $t$ is the generated thinking trajectory (capped at a maximum of 8,000 tokens). The colon notation indicates simple string concatenation: the thinking trajectory is appended directly after the document text with no special separator tokens beyond what the thinking generation model naturally produces.

What it computes: the operation physically takes two strings—the original web-crawled document and the LLM-generated reasoning analysis—and joins them into a single contiguous text sequence that becomes one training sample. No alignment, filtering, or selection is performed at this stage; every document that receives a thinking trajectory becomes an augmented sample.

Why this form: the concatenation preserves the original document in its entirety, meaning the model still sees all the raw text it would see in vanilla pre-training, plus the additional reasoning content. This is critical because it means TPT adds information without removing any existing training signal. The alternative—replacing the original document with only the thinking trajectory—would risk losing factual content, stylistic diversity, and domain coverage present in the original text. The concatenation approach also means no special token types or architectural modifications are needed; the model treats thinking trajectories identically to any other text during training.

The model minimizes the standard autoregressive language modeling objective:

$\min \mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \log p(x_i \mid x_{<i})$

where $N$ is the total number of tokens in the augmented sample $x$, $x_i$ is the $i$-th token in the sequence, and $p(x_i \mid x_{<i})$ is the model's predicted probability for that token given all previous tokens in the sample.

What it computes: the average negative log-likelihood of each token given its preceding context, summed over every token position in the concatenated document-plus-thinking sequence. The model processes tokens sequentially from the start of the original document through the end of the thinking trajectory, computing a prediction error at each position and averaging across all positions. There is no segmentation between document tokens and thinking tokens in the loss computation—every token is treated identically.

Why this form: this is precisely the standard pre-training objective used for all autoregressive language models (GPT, LLaMA, etc.). The paper deliberately does not introduce any specialized loss terms, token weighting schemes, or auxiliary objectives. This design choice has two justifications: first, it demonstrates that the benefits come purely from the data augmentation rather than from any algorithmic innovation in the training procedure, making the approach maximally compatible with existing training infrastructure; second, standard next-token prediction over the augmented sequence automatically implements the intended mechanism—the model must learn to predict both the original tokens and the reasoning tokens, and since reasoning tokens are easier to predict (they are generated by a weaker model and follow predictable chain-of-thought patterns), they provide a denser learning signal that regularizes the model's representations of the surrounding difficult content.

A subtle but important property: because the loss is computed over all tokens in the augmented sample, documents that generate longer thinking trajectories contribute proportionally more to the total loss. A document with an 8,000-token thinking trajectory contributes roughly 5× more to the training objective than a document with a 1,000-token trajectory (accounting for the original document's 2,000 tokens). This creates an implicit weighting mechanism where content that triggers deeper reasoning analysis receives more training compute, as analyzed in Section 4.

The training data itself is drawn from two sources, mixed with explicit sample weights to balance domain coverage:

• MegaMath-Web-Pro-Max (Zhou et al., 2025; Wang et al., 2025): approximately 70B tokens of math-intensive web content, filtered from the larger MegaMath corpus. This provides the mathematical reasoning content that TPT most dramatically improves.
• FineWeb-Edu (Penedo et al., 2024): a high-quality educational subset of the FineWeb dataset. This is deduplicated at the document level using exact matching before augmentation.

Why this mixture: the paper applies a sample weight of 0.125 to FineWeb-Edu when mixing the two datasets. This means that each FineWeb-Edu document contributes only 12.5% as much to the training batch composition as each MegaMath document. The justification is domain balancing: without this weighting, FineWeb-Edu (which is a much larger corpus) would dominate the training distribution, and the model would see insufficient mathematical content to develop strong reasoning capabilities. The specific weight of 0.125 was chosen so that math content constitutes a substantial fraction of each training batch, ensuring that the thinking trajectories generated for mathematical documents actually influence the model's learned representations. The paper does not provide an ablation of this weighting choice.

Documents from both sources are packed into training samples of exactly 8,192 tokens (8k), with document boundaries respected to avoid cross-document attention contamination during training. The packing is standard: when one document ends before filling the 8k context, the next document begins immediately, with a special separator token between documents (the exact separator is not specified but is standard in LLaMA-style training). This packing means that a single training sample may contain parts of multiple augmented documents or a single very long augmented document.

Thinking Trajectory Generation

The thinking trajectories are generated by prompting an off-the-shelf instruction-tuned LLM to analyze each document and produce a detailed reasoning explanation. The prompt template is fixed across all documents and all experiments—the paper explicitly states that no per-document prompt engineering or domain-specific prompting is performed, which is central to the scalability claim.

Prompt template. The generation prompt, quoted verbatim from Section 2, is:

{{CONTEXT}}
## End of the context
Simulate an expert's in-depth thought process as they analyze the above
context, focusing on complex and informative aspects.
Skip trivial
details.
Use Feynman technique whenever possible to ensure a deep
understanding.

The {{CONTEXT}} placeholder is replaced with the document text before generation. The prompt has several notable design features:

• No task specification: unlike chain-of-thought prompts that ask the model to solve a specific problem, this prompt asks the model to "analyze" and "simulate an expert's thought process" without specifying any output format, answer, or goal. This generality means the prompt works identically for mathematical derivations, code explanations, scientific articles, historical narratives, or any other web content.
• Feynman technique instruction: this is a learning strategy where one explains a concept in simple terms as if teaching it to someone else, identifying gaps in understanding when the explanation breaks down. By instructing the model to use this technique, the prompt encourages explanations that decompose complex ideas into simpler constituent parts—precisely the decomposition that makes difficult tokens more learnable.
• Explicit instruction to skip trivial details: this prevents the model from generating lengthy analyses of obvious or simple content (e.g., "This sentence states the author's name") and focuses the generation budget on genuinely complex aspects of the document.
• No output structure constraints: the model is not required to produce its thinking within specific tags or follow a particular format. However, since the thinking generation model (DeepSeek-R1-Distill-Qwen-7B) was trained to output reasoning within <think> and response tags, its generated trajectories naturally follow this structure, with the reasoning appearing between the thinking tags and the final response typically summarizing the analysis.

Generation hyperparameters. The paper specifies generation settings in Appendix A.1:

• Input truncation: documents are truncated to a maximum of 2,000 tokens before being passed to the thinking generation model. This prevents extremely long documents from exceeding the generation model's context window or producing impractically long trajectories. The choice of 2,000 tokens is a tradeoff: longer documents would provide more context for richer thinking trajectories, but would increase generation cost and could cause the thinking model to focus on less important parts of the document.
• Maximum output length: the maximum number of generated thinking tokens is set to 8,000 tokens. This is substantially longer than typical chain-of-thought outputs for problem-solving (which might be 200–500 tokens) and reflects the fact that thinking trajectories are explaining existing content rather than solving a specific problem—they can freely explore tangents, alternative explanations, and deeper implications.
• Sampling parameters: temperature of 0.6 and top-p of 0.9 are used during generation. Temperature 0.6 provides moderate stochasticity—high enough to produce diverse trajectories across different documents (and potentially different runs for the same document, though this is not explored) but low enough to maintain coherence and avoid hallucination. Top-p 0.9 is a standard nucleus sampling parameter that truncates the token distribution to the smallest set whose cumulative probability exceeds 0.9, preventing sampling from the long tail of unlikely tokens.
• Termination condition: the paper specifies that generation stops at the end of the thinking tag ("e.g., response"), and the final summary response is not generated. The justification is practical: "the final response is often the summary of the thinking process without introducing new information." Generating the response would add training tokens without contributing novel analytical content, effectively wasting training compute on redundant summarization.

Choice of thinking generation model. Two different models are used depending on the training configuration:

• For mid-training experiments: DeepSeek-R1-Distill-Qwen-7B is used as the thinking generator. The paper provides a specific rationale: "Despite not being the strongest model available, it has an open-source post-training recipe provided by OpenR1, which facilitates fair comparisons with other baselines." This is a reproducibility consideration—using a model with a fully open training pipeline means that other researchers can exactly replicate the thinking generation process, and the OpenR1 models (which are fine-tuned on the same dataset used for TPT's SFT phase) serve as natural comparison points.
• For pre-training from scratch: Qwen3-8B (Yang et al., 2025) is used instead. The paper does not provide an explicit justification for this switch, but it is likely motivated by capacity: Qwen3-8B is a stronger general-purpose model than the 7B distillation, potentially producing higher-quality thinking trajectories that are more critical when training from scratch (where the model has no pre-existing knowledge to build on, making data quality even more important).

Why an off-the-shelf model rather than a custom-trained generator: the paper's ablation study (Section 5, Table 4) directly tests whether customizing the thinking generator improves results. They fine-tune DeepSeek-R1-Distill-Qwen-7B on an SFT dataset to produce "back-thinking"—generating the thinking content given the final response and original question as input, essentially training the model to work backwards from answers to reasoning. This custom-trained generator produces only marginal improvements over the default off-the-shelf model (e.g., AIME24: 14.7% vs. 11.7% for the default at 40B mid-training tokens), while adding "extra implementation complexity due to the need for custom fine-tuning." This result supports the paper's simplicity claim: the default model's thinking trajectories are already sufficiently high-quality that customizing the generator yields diminishing returns.

Generation cost and scale. Appendix A.1 reports that "the entire data generation pipeline takes roughly 20k A100 GPU hours to yield enough data for 100B training tokens." To put this in context: 20k A100 GPU hours is approximately 2.3 GPU-years, or about 2-3 days on a cluster of 300-400 A100 GPUs. For an organization training 8B parameter models (which takes approximately 1 week on 32 MI300 GPUs per the paper's training details), the thinking generation cost is significant but not prohibitive—it represents roughly 10-20% of the total training compute budget. Importantly, this cost is amortized: the thinking trajectories are generated once and can be reused for multiple training runs, hyperparameter sweeps, or different model sizes.

Training Configurations and Hyperparameters

The paper evaluates TPT across three distinct training configurations, each with different resource constraints and objectives. The training hyperparameters for all configurations are specified in Appendix Table 5.

Configuration 1: Pre-training under abundant data (Section 3.1). Two 8B parameter models are trained from scratch following the LLaMA-3-8B architecture, both with a total training budget of 100B tokens. The only difference is the training data: one model trains on the original (vanilla) documents, the other on thinking-augmented documents. Key hyperparameters:

• Batch size: 4M tokens (512 samples of 8k tokens)
• Learning rate: 3e-4 with constant schedule (no decay)
• Optimizer: AdamW with β₁ = 0.9, β₂ = 0.95
• Weight decay: 0.1
• Gradient clipping: 1.0
• Warmup: 100 steps
• Training duration: 25,000 steps
• Hardware: 32 GPUs (MI300)

Why constant learning rate: standard pre-training typically uses cosine decay, but the paper uses a constant schedule here. The justification is not explicitly stated, but the likely reason is that with less than one epoch of training (the dataset is larger than 100B tokens, so each document is seen at most once), the model never encounters repeated data that would benefit from learning rate annealing to refine representations. The constant rate maintains full optimization velocity throughout the entire training run.

Important note on document exposure: because thinking trajectories add tokens to each document, the vanilla model is exposed to approximately 3× more raw documents than the TPT model during the 100B-token training run. Both models process 100B total tokens, but the TPT model's tokens are split between original document text and thinking trajectories, while the vanilla model's tokens are entirely original documents. This means the TPT model sees fewer unique documents but more tokens per document. The paper argues this is a feature, not a bug: it's precisely the mechanism by which TPT allocates more training compute to valuable content—the model sacrifices breadth (number of unique documents seen) for depth (more tokens analyzing each document).

Configuration 2: Pre-training under constrained data (Section 3.2). To simulate a regime where high-quality data is scarce (motivated by the observation that "frontier LLM training is approaching the exhaustion of high-quality web data"), the total number of unique raw document tokens is limited to 10B via random sampling from the full corpus. Two 8B models are trained with a 40B-token training budget. Under this constraint:

• The vanilla model sees the 10B-token dataset 4 times (4 epochs) because it processes 40B tokens of original documents.
• The TPT model sees the data approximately once because the thinking augmentation increases the effective token count per document, causing the 40B-token budget to be exhausted before a second epoch would begin.
• All other hyperparameters follow the abundant-data configuration.

Why this matters: the constrained-data setting directly tests whether TPT's gains come from simply seeing more tokens (which the vanilla model achieves through multiple epochs) or from the quality of the augmented tokens. If TPT outperforms despite seeing each document fewer times, it demonstrates that thinking trajectories provide a better training signal than repeated exposure to raw text. The results in Figure 3 confirm this: the vanilla model plateaus after the first epoch while TPT continues improving, indicating that multiple epochs of raw data yield diminishing returns while a single pass through augmented data continues to provide new learning signal.

Configuration 3: Thinking augmented mid-training (Section 3.3). Rather than training from scratch, existing open-source checkpoint models undergo additional training on 100B tokens of thinking-augmented data before supervised fine-tuning. Three model scales and families are tested:

• Qwen2.5-Math-1.5B (initialized for mid-training)
• LLaMA-3.2-3B (initialized for mid-training)
• Qwen2.5-Math-7B (initialized for mid-training)

Mid-training hyperparameters differ from pre-training in several ways reflecting the different starting point:

• Batch size: 8M tokens for all model sizes (larger than pre-training's 4M, leveraging the fact that mid-training starts from already-converged models that can tolerate larger updates)
• Learning rates: 6e-5 (1.5B), 4e-5 (3B), 3e-5 (7B)—substantially lower than pre-training from scratch, reflecting that the model weights are already in a good region and aggressive updates would disrupt existing capabilities
• Learning rate schedule: cosine decay (vs. constant for pre-training), which gradually reduces the learning rate to zero over the 12,500 training steps, allowing fine-grained refinement as the model approaches convergence on the augmented data
• Weight decay: 0.1 (same as pre-training)
• Gradient clipping: 0.3 (lower than pre-training's 1.0, providing more conservative gradient norms to prevent destabilizing existing representations)
• Training duration: 12,500 steps (half of pre-training's 25k steps, reflecting that 100B tokens at 8M batch size requires fewer steps than 100B at 4M)

Model initialization choices: the mid-training experiments deliberately use different base model families to test generality. Qwen2.5-Math variants are pre-trained with substantial mathematical content and already have strong reasoning capabilities; LLaMA-3.2-3B is a general-purpose model with less math-specific pre-training. The paper hypothesizes (and results confirm) that TPT provides larger relative gains for LLaMA models because their pre-training corpora contain "less reasoning-intensive data," meaning there is more headroom for improvement through thinking augmentation.

Why these three configurations collectively: the paper uses this triad to establish that TPT's benefits are not specific to any particular training regime. The abundant-data configuration shows that TPT helps even when data is not the bottleneck (compute is). The constrained-data configuration shows that TPT helps even more when data is scarce (the most practically relevant regime). The mid-training configuration shows that TPT enhances already-strong models without requiring training from scratch (the most cost-effective deployment path). Together, they argue for TPT as a general-purpose data engineering technique rather than a solution to any single narrow problem.

Dynamic Training Compute Allocation Through Thinking Length

While the thinking trajectory generation process uses a fixed prompt and fixed maximum output length, the actual length of generated trajectories varies substantially across documents depending on the content's complexity, domain, and reasoning intensity. This variability is not an accident—it is the key mechanism by which TPT achieves dynamic training compute allocation without any explicit difficulty estimation or per-document hyperparameter tuning.

The paper provides quantitative evidence for this mechanism in Section 4's thinking pattern analysis. Using a balanced sample of 20,000 documents stratified across metadata groups from the essential-web-v1.0 dataset (AI et al., 2025), the paper measures average thinking trajectory length across three dimensions:

By domain (Figure 4, left panel): Mathematics and Physics documents generate the longest thinking trajectories (approximately 1,900 and 1,800 tokens on average, respectively), while domains like Social Service and Economics generate substantially shorter trajectories (approximately 1,100–1,200 tokens). The top-10 longest-thinking domains are dominated by STEM fields. This aligns with intuition: mathematical content requires multi-step logical derivations that naturally produce longer explanations, while descriptive content in social sciences can be analyzed more concisely.

By reasoning intensity (Figure 4, middle panel): The essential-web-v1.0 dataset provides a "reasoning intensity" metadata tag with levels "Advanced," "Intermediate," "Basic," and "None." The paper finds that documents tagged "Advanced" produce thinking trajectories approximately 50% longer than those tagged "None" (roughly 1,400 vs. 1,000 tokens on average). This confirms that the thinking generation model naturally allocates more output tokens to content that requires deeper analysis, without any explicit instruction to do so.

By target audience (Figure 4, right panel): Somewhat counterintuitively, "Expert"-level documents produce shorter thinking trajectories than "Undergraduate"-level documents (roughly 1,150 vs. 1,250 tokens). The paper hypothesizes that "expert-level documents often contain more specialized concepts, but do not necessarily require a greater number of reasoning steps for comprehension." This suggests that thinking length correlates with the explanatory gap—how much intermediate reasoning is needed to bridge from general knowledge to the document's content—rather than with the document's absolute difficulty.

What this means for training compute allocation: because the next-token prediction loss is computed over all tokens in the augmented sample (document + thinking trajectory), a document with a longer thinking trajectory contributes more tokens to the loss function and therefore receives more gradient updates during training. Specifically, a Mathematics document with a 1,900-token thinking trajectory contributes roughly 3,900 total tokens to training (2,000 document tokens + 1,900 thinking tokens), while an Economics document with a 1,100-token trajectory contributes roughly 3,100 tokens. The ratio of training compute allocated to math vs. economics content is approximately 1.26:1—a 26% increase for the domain where reasoning is most valuable.

This effect compounds across the entire corpus: content that benefits most from detailed analysis (mathematics, physics, advanced reasoning) naturally receives a larger share of the total training compute budget because its thinking trajectories are longer. The paper frames this as a training-time analog of test-time compute scaling, where harder problems receive more inference compute (longer chain-of-thought). The key distinction is that test-time scaling allocates compute per-query at inference, while TPT allocates compute per-document at training time. The mechanism is the same: complex content triggers longer reasoning, which consumes more compute.

Why this is "natural up-sampling": the paper contrasts this mechanism with explicit data engineering heuristics. Traditional approaches to up-sampling high-quality data require metadata tags, quality classifiers, or hand-crafted rules to identify which documents deserve more training emphasis. TPT achieves the same effect implicitly through the thinking generation model's own behavior—if a document triggers longer thinking, it contains more learnable complexity, and therefore deserves more training compute. The thinking generation model acts as an automatic complexity detector, distributing training tokens in proportion to a document's reasoning demands without any manual intervention.

This dynamic allocation is not calibrated through any explicit optimization; the paper does not tune the thinking generation prompt to maximize length correlation with downstream performance or to achieve any particular allocation ratio. The allocation emerges purely from the interaction between the prompt ("focus on complex and informative aspects, skip trivial details") and the thinking generation model's learned behaviors. This is both a strength (simplicity, no hyperparameters to tune) and a limitation (no guarantee that the allocation is optimal, no way to adjust it without changing the prompt or model).

Supervised Fine-Tuning Integration

After pre-training or mid-training on thinking-augmented data, all models undergo supervised fine-tuning (SFT) to align them with instruction-following behavior and enable evaluation on reasoning benchmarks. The SFT procedure is identical for all TPT and baseline models, ensuring that performance differences can be attributed to the pre-training/mid-training data rather than the fine-tuning process.

SFT dataset. The Mixture-of-Thoughts dataset (HuggingFace, 2025) is used, containing 350,000 examples covering math, coding, and science domains. This dataset is distilled from DeepSeek-R1 (Guo et al., 2025)—the examples contain chain-of-thought reasoning traces generated by DeepSeek-R1 in response to prompts, formatted as instruction-following dialog. The paper notes that this dataset "has been reported to replicate the performance of DeepSeek-R1-Distill-Qwen-7B, which is trained on a private dataset of 800k examples," but acknowledges that reproduction at smaller scales (1.5B) remains inferior, "likely attributed to the smaller dataset size."

Why this dataset: the choice enables direct comparison with OpenR1 models (HuggingFace, 2025), which are the primary baselines in the mid-training experiments. OpenR1 models are fine-tuned on the exact same Mixture-of-Thoughts dataset, starting from the same base checkpoints. This means any performance difference between TPT models and OpenR1 models can be attributed exclusively to the thinking-augmented mid-training phase—both models use the same initialization, the same SFT data, and the same SFT hyperparameters.

SFT hyperparameters (Appendix Table 5):

• Batch size: 1M tokens for all model sizes
• Learning rates: 6e-5 (1.5B), 4e-5 (3B), 3e-5 (7B)
• Learning rate schedule: cosine decay
• Training epochs: 5
• Weight decay: 0 (disabled, unlike pre-training/mid-training where it was 0.1)
• Gradient clipping: 0.2 (lower than mid-training's 0.3, reflecting the more delicate nature of fine-tuning on a small dataset)
• Warmup: 400 steps
• Maximum sequence length: 32,768 tokens (32k)
• Data packing: enabled, with example boundaries respected to prevent cross-example attention
• Loss computation: only over assistant response tokens (instruction tokens are not included in the loss)

Why disable weight decay during SFT: weight decay acts as a regularizer that prevents weights from growing too large, which is important during long pre-training runs where overfitting is a risk. During SFT on only 350k examples, the primary risk is underfitting (not learning the task patterns well enough) rather than overfitting, so regularization is counterproductive. The paper's results in Figure 6 show that performance continues improving through 5 epochs with no overfitting, supporting this choice.

Context length extension for SFT. The base models are pre-trained or mid-trained with an 8k context length, but SFT uses 32k context to accommodate long chain-of-thought reasoning traces during both training and evaluation. To support this extension, the RoPE (Rotary Position Embedding) base frequency is multiplied by 16 (Appendix A.1). This is a standard technique: RoPE embeddings encode position information through sinusoidal functions with a base frequency; multiplying the base frequency effectively rescales the position encoding to accommodate longer sequences without retraining the position embeddings from scratch. The specific multiplier of 16 extends the effective context from 8k to 128k (theoretically), though the paper only uses 32k in practice.

Evaluation protocol for instruction-tuned models (Appendix A.2). After SFT, models are evaluated on challenging reasoning benchmarks with specific generation settings:

• Maximum thinking tokens: 32,768 (32k)—this is the model's own chain-of-thought output during evaluation, not the pre-training thinking trajectories
• Sampling temperature: 0.6, top-p: 0.95
• Multiple samples for variance reduction: 64 samples for AIME24/25, 16 for LiveCodeBench, 8 for GPQA-Diamond, 4 for MATH-500
• Pass@1 computed by averaging correctness across all samples for each problem

Why the SFT phase is critical to the experimental narrative: the paper's most dramatic results appear after SFT (Tables 2 and 3), not at the base model stage (Table 1). This is because thinking-augmented pre-training/mid-training builds a foundation of reasoning capabilities that SFT then "unlocks" by teaching the model to apply those capabilities in an instruction-following format. The base model might understand polynomial division and the Remainder Theorem from its thinking-augmented training, but it doesn't know to apply that understanding when asked "What is the largest positive integer n for which n³+100 is divisible by n+10?" unless it has been fine-tuned to follow such instructions. The paper's architecture implicitly assumes a two-stage deployment: pre-train/mid-train with TPT to build reasoning foundations, then SFT to make those foundations accessible through natural language interaction.

Ablation on SFT data size (Figure 6). The paper sweeps SFT epochs from 0.5 to 5 and finds that performance generally improves monotonically, with no overfitting observed. LiveCodeBench appears not to have fully converged even at 5 epochs, suggesting additional SFT data or epochs could yield further gains. More importantly for TPT's claims: the models that underwent thinking-augmented mid-training start from substantially higher performance at epoch 0.5 and maintain this advantage throughout SFT. The gap between TPT and vanilla models widens during SFT for some tasks (e.g., AIME24 goes from near-zero for vanilla to 15+ for TPT), indicating that the SFT process amplifies the differences created during pre-training/mid-training rather than washing them out.

Why SFT alone is insufficient (Figure 5, "0B" data points): the paper includes an important control experiment where models undergo SFT directly on the base checkpoint without any thinking-augmented mid-training. These "0B mid-training" baselines perform substantially worse, particularly on challenging reasoning tasks like AIME24 (near 0% for LLaMA-3B without mid-training vs. ~15% with 100B tokens of TPT mid-training). This demonstrates that SFT on 350k examples cannot teach reasoning capabilities from scratch—it can only elicit capabilities that were already developed during pre-training or mid-training. TPT's role is to develop those capabilities during the data-intensive phase so that even a modest SFT dataset can effectively activate them.

4. Key Insights and Innovations

Innovation 1: Reframing Data Engineering as a Learnability Problem Rather Than a Quality Problem

The paper's most fundamental conceptual move is shifting the framing of data engineering from selecting high-quality content to improving the learnability of already-selected content. Prior to this work, the dominant paradigm for maximizing training data utility—exemplified by data selection frameworks like Lin et al. (2024) and Mindermann et al. (2022)—asked: which tokens are worth training on? The answer involved classifiers, quality filters, perplexity thresholds, and prioritization schemes that identified valuable tokens and discarded or down-weighted the rest. This framing implicitly assumes that tokens are atomic, self-contained learning units: if a token contains valuable information, training on it directly suffices to absorb that information.

TPT challenges this assumption by diagnosing a fundamentally different bottleneck. The problem with difficult tokens—the numeric answer "890" in a math proof, the conclusion of a complex scientific argument—is not that they contain low-quality or irrelevant information. They contain precisely the information you most want the model to learn. The problem is that the compression ratio demanded by next-token prediction is too extreme: a single token must be predicted from context that does not contain the intermediate reasoning steps connecting that context to the token. The model can memorize the token (fragile, doesn't generalize) or fail to learn it (wasted training compute). This is not a quality problem—it's a learnability problem arising from the mismatch between the training objective (predict the next token given preceding text) and the cognitive depth encoded in individual tokens.

This reframing matters because it redirects research attention from filtering to transforming training data. The paper is explicit that TPT is "orthogonal to rewriting based approaches" (Section 6) and complementary to existing data curation pipelines. You can apply all of FineWeb's filtering, deduplication, and quality scoring, and then additionally apply TPT to the surviving documents. The two operations address different failure modes: curation removes content that is intrinsically unhelpful (spam, noise, repetition), while TPT makes intrinsically helpful but unlearnable content more accessible to the model. The distinction is analogous to the difference between selecting which textbooks to assign (curation) and adding explanatory margin notes to the assigned textbooks (TPT).

The significance of this reframing extends beyond the specific method. If the field accepts that token learnability—not just token quality—is a first-class concern in pre-training data engineering, it opens a new axis for optimization. Future work might explore: what other transformations beyond thinking trajectories increase learnability? Can learnability be quantified and predicted before training? Is there a learnability scaling law analogous to data scaling laws? The paper does not answer these questions, but by naming the problem and demonstrating one effective solution, it establishes the intellectual scaffolding for asking them.

Evidence anchor: Figure 1b provides the canonical illustration—the token "890" is correct and valuable, yet predicting it from the preceding question text requires implicitly reconstructing the Remainder Theorem, polynomial division, and divisor analysis in a single forward pass. The thinking-augmented version of the same document decomposes this into 1,054 tokens of step-by-step reasoning, converting a single nearly-impossible prediction into a sequence of tractable ones.

Innovation 2: Training-Time Compute Scaling as a Natural Consequence of Reasoning-Based Augmentation

The paper draws an explicit analogy between TPT and test-time compute scaling (Jaech et al., 2024; Guo et al., 2025), but the deeper insight is that this analogy is not merely metaphorical—it is mechanistic. Test-time scaling works because harder problems receive longer chain-of-thought reasoning at inference, which consumes more compute and improves accuracy. TPT works because harder content naturally triggers longer thinking trajectories during data generation, which consumes more tokens during training and improves learnability. The mechanism connecting difficulty to compute allocation is identical in both cases: an LLM, prompted to reason step-by-step, produces output length proportional to the complexity of the input.

What makes this insight non-obvious is that the field had treated training compute and inference compute as governed by fundamentally different principles. Training compute is typically allocated uniformly (every token in the corpus gets equal weight in the loss) or through explicit reweighting heuristics (domain balancing, up-sampling of high-quality sources). Inference compute is allocated dynamically per-query (longer chain-of-thought for harder problems). TPT demonstrates that dynamic, difficulty-proportional allocation of training compute can be achieved implicitly through the behavior of a separate generation model, without any explicit difficulty estimator, quality classifier, or per-document hyperparameter tuning.

The thinking pattern analysis in Section 4 provides the empirical backbone for this claim: mathematics documents generate thinking trajectories roughly 73% longer than social science documents (1,900 vs. 1,100 tokens); "Advanced Reasoning" documents generate roughly 50% more tokens than "No Reasoning" documents (1,400 vs. 1,000). These differences are not designed—they emerge from the interaction between the fixed prompt ("focus on complex and informative aspects, skip trivial details") and the thinking generation model's learned behaviors. The generation model acts as an automatic complexity detector and training compute allocator, distributing tokens across the corpus in proportion to each document's reasoning demands.

The practical significance is that this allocation mechanism requires zero manual engineering. Traditional approaches to up-sampling high-quality data require metadata tags (domain labels, difficulty ratings), quality classifiers trained on human annotations, or hand-crafted heuristics for which documents deserve more training emphasis. TPT achieves comparable or better allocation through a single forward pass of an off-the-shelf instruction-tuned model. This is not just computationally cheaper—it is more robust because it adapts automatically to any corpus structure, and more nuanced because it allocates compute at the document level based on specific content rather than at the domain level based on coarse categories.

A more subtle contribution: this mechanism provides an explanation for why TPT improves data efficiency by specifically 3× (Figure 1a). If thinking trajectories are on average 2–3× the length of the original documents they augment (the paper does not report the exact corpus-wide average, but the thinking pattern analysis suggests trajectories range from roughly 1,000–1,900 tokens compared to up-to-2,000-token documents), and if the augmented tokens are genuinely more learnable than repeated epochs of raw tokens, then 100B tokens of augmented data should provide roughly the learning signal of 300B tokens of raw data. The 3× efficiency factor is not a magic number—it is a consequence of the expansion ratio between document length and thinking trajectory length, combined with the improved learnability of reasoning-structured text.

Evidence anchor: Figure 4 directly quantifies the allocation mechanism across domain, reasoning intensity, and target audience, showing clear positive correlations between content complexity and thinking trajectory length. Figure 1a then shows that TPT reaches the same performance at ~33B tokens that vanilla training reaches at ~100B tokens—a 3× efficiency gain matching the rough expansion ratio.

Innovation 3: Demonstrating That a Single Pass of Reasoning Augmentation Suffices—The "No Bootstrapping Required" Finding

This is fundamentally a negative result with positive implications: the paper shows that iteratively refining thinking trajectories (through EM-like bootstrapping as in BoLT, or through custom fine-tuning of the generation model) yields minimal additional benefit over a single pass with an off-the-shelf model. The ablation study in Table 4 tests two complex alternatives—a custom fine-tuned "back-thinking" model trained to generate reasoning given final answers, and a prompt variant that adds a random focus point—and finds only marginal improvements over the default approach (e.g., AIME24: 14.7% for back-thinking vs. 11.7% default at 40B mid-training tokens).

What makes this finding significant is that it challenges a natural intuition: surely better thinking trajectories produce better training data, and surely the way to get better trajectories is to improve the generator. The paper's evidence suggests this intuition may be wrong, at least given current models. The default DeepSeek-R1-Distill-Qwen-7B, prompted with a generic template, already generates trajectories that are close-to-optimal for downstream training. Adding complexity—custom fine-tuning, prompt engineering, iterative refinement—introduces implementation overhead and potential distribution shift without commensurate gains.

Even more striking is the finding that using a smaller generation model can outperform a larger one (Table 4, bottom section): DeepSeek-R1-Distill-Qwen-1.5B generates trajectories that produce better downstream performance than the 7B version (e.g., AIME24: 17.7% vs. 11.7%). The paper speculates that "the smaller model may generate trajectories that are better suited for downstream model learning," citing similar observations from OpenThoughts (Guha et al., 2025). This is counterintuitive—a weaker model generating better training data—but it aligns with a broader pattern in knowledge distillation: the most informative teacher is not always the strongest one, because overly sophisticated explanations may be incompressible by the student model. A smaller generation model produces simpler, more incremental reasoning steps that are closer to what the training model can assimilate.

This finding has substantial practical implications for anyone implementing TPT-like approaches. It suggests that the optimal thinking generator is not the most capable reasoning model available, but rather a model matched to the capacity of the model being trained. For a 3B-parameter student, a 1.5B-parameter teacher may be optimal. For larger students, the relationship may invert. The paper does not systematically explore this matching, flagging it as an area where "the relationship between the model being fine-tuned and the model used for thinking generation warrants further investigation." This identifies a new design dimension—teacher-student capacity matching for reasoning augmentation—that did not exist in the literature before TPT.

The "no bootstrapping required" finding also strengthens TPT's scalability claim relative to methods like BoLT's EM algorithm or RPT's online RL. If a single pass of thinking generation achieves near-optimal results, the data generation cost is a one-time fixed expense rather than scaling with training steps. This is critical for training runs at the 100B-token scale and beyond, where iterative refinement would multiply generation costs by the number of iterations.

Evidence anchor: Table 4 shows the three ablation variants (back-thinking, random focus, smaller model) against the default, with the smaller model actually outperforming the default on AIME24 (17.7% vs. 11.7%) and MATH-500 (75.5% vs. 72.0%). The back-thinking and random focus variants show marginal gains at best (+1.8 percentage points on MATH-500 for back-thinking, +2.5 on HEval for random focus).

Innovation 4: Establishing That Pre-Training Data Augmentation Compensates for SFT Data Scarcity

This insight emerges from the interaction between the mid-training experiments (Section 3.3) and the SFT data scaling analysis (Figure 6), and it addresses a practical bottleneck the paper does not explicitly name: post-training datasets for reasoning are small and expensive to produce, but pre-training data is abundant and cheap to augment. The paper's results demonstrate that investing compute in thinking-augmented pre-training or mid-training can substantially reduce the SFT data required to achieve strong reasoning performance.

The evidence for this claim is distributed across multiple experiments. In Table 3, TPT models fine-tuned on only 350k SFT examples (the Mixture-of-Thoughts dataset) approach or exceed the performance of DeepSeek-R1-Distill models trained on a private 800k-example dataset—more than 2× the SFT data. The TPT-Qwen2.5-7B achieves 92.5% on MATH-500 and 57.5% on AIME24 vs. 93.5% and 53.2% for the DeepSeek distillation baseline, despite using less than half the SFT data. This is not a direct comparison (the DeepSeek models may differ in other ways), but the magnitude of the gap relative to the SFT data disparity is suggestive.

More directly, Figure 5 shows that direct SFT without thinking-augmented mid-training produces near-zero performance on AIME24 for the LLaMA-3B model, while the same model with 100B tokens of TPT mid-training achieves ~15% after identical SFT. This is not a small improvement—it is the difference between a model that cannot solve any AIME problems and one that can solve roughly one in six. The SFT data is identical in both cases; the only difference is the pre-SFT training. This means TPT effectively pre-installs reasoning capabilities during the data-intensive phase that SFT can later elicit with minimal examples.

The conceptual contribution is identifying a division of labor between pre-training and post-training for reasoning. The conventional wisdom—implicit in the design of models like LLaMA-3-Instruct and Qwen2.5-Instruct—is that pre-training builds broad knowledge and language understanding, while post-training (SFT + RLHF) teaches specific behaviors like instruction-following and reasoning. TPT's results suggest a more nuanced picture: some capabilities currently allocated to post-training can be shifted to pre-training if the pre-training data is appropriately augmented. Reasoning, in particular, may be more efficiently learned from diverse, automatically-augmented pre-training documents than from a small set of carefully curated SFT examples.

This has direct implications for the economics of reasoning model development. High-quality SFT data with verified chain-of-thought reasoning traces is expensive—it requires human annotation or distillation from frontier models, and the supply is limited. Pre-training data is abundant and augmenting it with thinking trajectories is a one-time compute cost. If TPT can shift reasoning capability acquisition from the expensive SFT phase to the cheap pre-training phase, it reduces the overall cost of building strong reasoning models. This is particularly important for smaller organizations that lack access to large proprietary SFT datasets.

Evidence anchor: Figure 5, "0B" data points show direct SFT performance without mid-training (near-zero AIME24 for LLaMA-3B). The 100B data points show the same model with TPT mid-training. The gap—from ~0% to ~15% on AIME24—represents reasoning capabilities acquired entirely during mid-training and elicited by the same SFT process. Table 3 extends this across three model sizes and two model families, with TPT consistently outperforming baselines fine-tuned on the exact same SFT data.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary pre-training and mid-training corpus consists of two sources: MegaMath-Web-Pro-Max (Zhou et al., 2025; Wang et al., 2025), a filtered subset of approximately 70B tokens of math-intensive web content, and FineWeb-Edu (Penedo et al., 2024), a high-quality educational subset of the FineWeb dataset, deduplicated at the document level via exact matching. These are mixed with a sample weight of 0.125 applied to FineWeb-Edu to balance domain distribution (Appendix A.1). For supervised fine-tuning, the Mixture-of-Thoughts dataset (HuggingFace, 2025) is used, containing 350,000 examples covering math, coding, and science domains, distilled from DeepSeek-R1 (Guo et al., 2025).

• Base model(s). Three distinct training configurations use different model families and scales. For pre-training from scratch (Sections 3.1 and 3.2), two 8B-parameter models following the LLaMA-3-8B architecture are trained with random initialization. For mid-training (Section 3.3), existing checkpoints are used: Qwen2.5-Math-1.5B, LLaMA-3.2-3B, and Qwen2.5-Math-7B. The paper states the Qwen models are from Yang et al. (2024) and LLaMA from Dubey et al. (2024). These specific models are chosen to span two major open-source families and a 4.7× range of parameter counts, testing generality. The LLaMA-3.2-3B variant is deliberately included because its pre-training corpora contain "less reasoning-intensive data" (Section 3.3), providing a stronger test of whether TPT can compensate for weaker initial reasoning capabilities.

• Metrics. For base model evaluation, the primary metric is accuracy (exact match after answer extraction via regular expressions) on five datasets: GSM8k (5-shot with CoT), MATH (4-shot with CoT), BoolQ (0-shot), MMLU (2-shot with CoT), and MMLUPro (2-shot with CoT). The paper acknowledges that failure of answer extraction yields a zero score, which can push multiple-choice task scores below random guessing at early training stages (Section 3.1). For instruction-tuned models, the primary metric is Pass@1 — the fraction of problems where the model's first generated answer is correct, averaged across multiple samples per problem to reduce variance on small datasets. Specifically, 64 samples are generated for AIME24 and AIME25, 16 for LiveCodeBench, 8 for GPQA-Diamond, and 4 for MATH-500 (Appendix A.2). The aggregated score in Figure 2 averages accuracy across the five base-model evaluation datasets with equal weight.

• Baselines. The paper employs several comparison points. Vanilla pre-training: the same model architecture and training budget but on original (non-augmented) documents, used as the primary baseline throughout Sections 3.1 and 3.2. LLaMA-3.1-8B (trained on 15T tokens) and LLaMA-2-7B (trained on 2T tokens) serve as external reference points for pre-training data efficiency. OpenR1 models (HuggingFace, 2025) — fine-tuned on the identical Mixture-of-Thoughts dataset starting from the same base checkpoints — are the primary baselines for mid-training experiments (Section 3.3) because they share the same SFT data and initialization, isolating the effect of TPT mid-training. DeepSeek-R1-Distill models (Guo et al., 2025) are listed for reference but explicitly noted as "not directly comparable" because they were trained on a private 800k-example SFT dataset versus the public 350k-example dataset used by TPT and OpenR1. Direct SFT (no mid-training) serves as a control in Figure 5 (the "0B" data points) to measure how much reasoning capability the SFT phase alone can develop. Vanilla mid-training (Table 6): continual training on the raw text data for 40B tokens without thinking augmentation, followed by the same SFT — this tests whether additional standard pre-training provides any benefit versus TPT's augmented data.

• Generation budget / compute accounting. The universal unit of training compute is total training tokens processed — the sum of all tokens (original documents plus thinking trajectories, if present) that the model's loss is computed over. Both vanilla and TPT models use exactly the same total token budget within each experimental configuration (100B tokens for abundant-data pre-training and mid-training; 40B tokens for constrained-data pre-training). This makes the comparison fair in terms of FLOPs: both models perform the same number of forward and backward passes over the same number of tokens. The paper explicitly notes an important asymmetry: at equal token budgets, the vanilla model sees approximately 3× more unique documents than the TPT model because thinking trajectories consume tokens that would otherwise be spent on new documents (Section 3.1). The thinking trajectory generation cost — roughly 20k A100 GPU hours for 100B training tokens (Appendix A.1) — is a one-time preprocessing expense that is not included in the training compute budget. The paper does not amortize this generation cost in any efficiency calculation. For evaluation, generation length is capped at 2k tokens for base models and 32k tokens for instruction-tuned models (Appendix A.2), with the generation budget controlled through these caps rather than through explicit token counting.

• Cross-validation / statistical protocol. The paper reports no cross-validation across data splits. The pre-training experiments use a single train/test split: models are trained on the MegaMath-Web-Pro-Max and FineWeb-Edu corpora and evaluated on the standard test sets of GSM8k, MATH, BoolQ, MMLU, and MMLUPro. The mid-training experiments use fixed model initializations and fixed SFT data, with evaluation on the standard benchmarks listed in Table 3. To reduce variance from sampling during evaluation, multiple samples are generated per problem and averaged (Pass@1), as described above. The paper does not report confidence intervals, standard errors, or statistical significance tests for any result. For the constrained-data experiment (Section 3.2), the 10B-token subset is drawn via "random sampling" from the full corpus, but the paper does not specify whether multiple random seeds were tested or whether the results are sensitive to the particular sample drawn. The five difficulty bins used in the oracle difficulty analysis are constructed by splitting the test set into quintiles based on pass@1 rate, but this analysis appears only in the prior sections context and is not part of the TPT paper itself — TPT does not perform per-difficulty evaluation. This absence of statistical rigor is a meaningful limitation, particularly given the relatively small evaluation sets: AIME24 and AIME25 each contain 30 problems, meaning a single additional correct answer shifts scores by ~3.3 percentage points.

Main Quantitative Results

Pre-Training Under Abundant Data: 8B Models on 100B Tokens

The headline result appears in Figure 2 (right panel) and Table 1: TPT achieves a 17.7 percentage point improvement in average base model performance over vanilla pre-training at the same 100B-token budget (43.9% vs. 26.2% averaged across five tasks). More remarkably, the TPT-8B model trained on 100B tokens approaches the performance of LLaMA-3.1-8B, which was trained on 150× more data (15T tokens): TPT-8B achieves 43.9% average vs. LLaMA-3.1-8B's 46.8%. The gains are concentrated in mathematical reasoning: GSM8k improves from 19.2% to 50.1% (a 2.6× improvement), and MATH more than doubles from 9.1% to 21.8%. Gains on non-mathematical tasks are more modest but still substantial: BoolQ improves from 66.5% to 75.0% (+8.5 points), MMLU from 26.2% to 46.7% (+20.5 points), and MMLUPro from 10.3% to 26.2% (+15.9 points).

Figure 2 (left panel) shows the training loss curves, where the TPT model's loss is "substantially lower" than the vanilla model's loss. However, the paper correctly notes that these loss values are "not directly comparable due to differences in data distributions" — thinking-augmented text may be inherently easier to predict (lower perplexity) because it follows more predictable chain-of-thought patterns, but lower loss on augmented data does not guarantee better downstream performance. The right panel of Figure 2 addresses this by tracking the aggregated evaluation score across training: TPT initially performs similarly to the vanilla model but "rapidly surpasses it after 20B tokens, and this performance gap continues to widen" through the full 100B-token budget. This divergence pattern — early parity followed by accelerating TPT advantage — is consistent with the paper's learnability hypothesis: early in training, both models are absorbing simple patterns that are learnable from raw text; as training progresses, the vanilla model begins to hit diminishing returns on difficult tokens while TPT continues extracting value from the thinking trajectories.

Post-SFT results in Table 2 amplify these findings. After fine-tuning on the 2B-token Mixture-of-Thoughts dataset, the gap between TPT and vanilla models widens dramatically on challenging reasoning benchmarks: AIME24 jumps from 1.0% to 35.2%, MATH-500 from 33.8% to 82.4%, LiveCodeBench from 1.9% to 23.4%, GPQA-Diamond from 27.7% to 45.2%, and MMLUPro from 29.0% to 59.8%. The TPT-8B→SFT model outperforms LLaMA-3.1-8B-Instruct on every benchmark despite the latter being trained on 150× more pre-training data plus its own instruction-tuning pipeline. This is the paper's strongest single result and supports the claim that TPT's benefits are not merely incremental — they enable a qualitatively different level of reasoning capability from the same pre-training compute budget.

The 3× data efficiency claim originates from Figure 1a (reproduced in the paper's abstract graphic). The TPT model reaches the same average score at roughly 33B tokens that the vanilla model achieves at roughly 100B tokens. However, this figure is shown only in the summary graphic and not analyzed in detail in the text. The paper does not report the exact interpolation method or the specific accuracy value used to define "same level of base model performance" for the 3× calculation. This is a notable omission: the 3× figure is the paper's headline efficiency claim, but the reader must estimate it visually from a small plot.

Pre-Training Under Constrained Data: 8B Models with 10B Raw Document Tokens

When raw document tokens are limited to 10B (simulating data scarcity) and the training budget is 40B tokens, the vanilla model sees the 10B-token dataset 4 times (4 epochs) while the TPT model sees it approximately once (Table 7). Figure 3 tracks performance across training: both models start at similar levels, but the vanilla model's performance "plateaus or improves slowly as unique tokens are exhausted, whereas TPT continues to improve steadily." By the end of training, TPT achieves 32.6% average vs. 16.6% for vanilla — a 16-point gap, nearly doubling performance. The divergence is most pronounced on mathematical reasoning: GSM8k reaches 30.5% for TPT vs. 6.7% for vanilla (4.6× improvement), MATH reaches 12.9% vs. 4.8% (2.7× improvement).

Critically, this experiment demonstrates that TPT's gains cannot be replicated by simply training for more epochs on the same raw data. The vanilla model processes each document 4 times but plateaus, while TPT processes each document once (with augmentation) and continues improving. This directly supports the learnability hypothesis: repeated exposure to the same difficult tokens in their raw form provides minimal additional learning signal, while a single exposure to decomposed, reasoning-augmented versions provides substantially more.

The paper does not report an epoch sweep beyond 4 for the vanilla model, leaving open the question of whether further epochs would eventually provide some benefit (unlikely given the plateau shape, but not proven). The constrained-data experiment also does not test whether mixing the 10B raw tokens with additional non-augmented data from other sources (simulating a scenario where the main bottleneck is domain-specific data scarcity rather than total data scarcity) would change the outcome.

Thinking Augmented Mid-Training: Three Model Scales Across Two Families

Table 3 presents the most comprehensive results, evaluating mid-trained models after SFT on 10 benchmarks spanning math (MATH-500, AIME24, AIME25, GSM8k, HMMT), code (HumanEval, LiveCodeBench v4_v5), and general reasoning (GPQA-Diamond, MMLUPro, JEEBench). The headline pattern is that TPT models substantially outperform OpenR1 baselines across all model sizes and nearly all benchmarks, with the largest relative gains on the most challenging reasoning tasks and the largest absolute gains for the LLaMA family.

For the Qwen2.5-1.5B based models, TPT improves AIME24 from 20.8% to 28.5% (+7.7 points), LiveCodeBench from 11.6% to 17.2% (+5.6 points), and JEEBench from 38.2% to 50.3% (+12.1 points). MATH-500 improves from 79.6% to 82.3% (+2.7 points) — a ceiling effect, as the model is already near saturation on this benchmark at 1.5B scale. GSM8k shows minimal improvement (79.2% to 80.1%), likely also a ceiling effect.

For the LLaMA-3.2-3B based models, the gains are substantially larger, consistent with the paper's hypothesis that LLaMA's pre-training contains less reasoning-intensive data and therefore has more headroom for improvement. AIME24 jumps from 5.8% to 18.6% (a 3.2× increase), MATH-500 from 59.8% to 75.5% (+15.7 points), LiveCodeBench from 13.9% to 20.0% (+6.1 points), GPQA-Diamond from 32.8% to 41.7% (+8.9 points), MMLUPro from 45.8% to 55.5% (+9.7 points), and JEEBench from 26.6% to 42.4% (+15.8 points). The only benchmark where the OpenR1 baseline retains a narrow advantage is HMMT (6.3% vs. 11.7%), a challenging math competition benchmark.

For the Qwen2.5-7B based models, TPT pushes performance to levels that approach or exceed the DeepSeek-R1-Distill-Qwen-7B reference despite using less than half the SFT data (350k vs. 800k examples). TPT achieves 92.5% vs. 93.5% on MATH-500, 57.5% vs. 53.2% on AIME24 (an improvement), 39.4% vs. 35.5% on AIME25, 39.3% vs. 36.2% on LiveCodeBench, 54.7% vs. 49.0% on GPQA-Diamond, and 73.6% vs. 49.9% on JEEBench. The JEEBench result is particularly striking — a 23.7-point advantage over the DeepSeek distillation baseline and a 4.7-point improvement over the OpenR1 baseline (69.1%). GPT-4o is included as an external reference and scores 74.6% on MATH-500, 9.3% on AIME24, 32.9% on LiveCodeBench, and 72.6% on MMLUPro — all below or comparable to the TPT-Qwen2.5-7B model.

Table 6 provides a critical control experiment: vanilla mid-training — continuing to train the LLaMA-3.2-3B model on the same text data for 40B tokens without thinking augmentation, then applying the same SFT. This control shows "minimal impact on most benchmarks, with mathematical and general reasoning scores remaining largely stable," and a "notable decline in coding performance." For example, HumanEval drops from 45.7% (direct SFT) to 36.0% (vanilla mid-training → SFT), and LiveCodeBench drops from 13.9% to 5.7%. This result is essential because it demonstrates that the benefits observed in Table 3 are specifically due to the thinking augmentation, not simply due to additional training on relevant data. In fact, additional vanilla training may be actively harmful for some capabilities — a finding that strengthens the paper's claim that standard next-token prediction on raw text has limited ability to improve reasoning beyond a certain point.

Figure 5 shows the effect of scaling the mid-training token budget from 0 to 100B tokens for both Qwen2.5-1.5B and LLaMA-3.2-3B models. Performance improves monotonically across all four benchmarks shown (AIME24, MATH-500, GPQA-Diamond, LiveCodeBench). The slopes are steeper earlier in training and show signs of continued improvement at 100B tokens, particularly for AIME24 and LiveCodeBench. The "sustained upward trend suggests that scaling beyond 100B tokens would likely yield further improvements." This is important because it indicates TPT has not saturated — the 100B-token budget was chosen for practical reasons, not because it represents an upper bound on effectiveness.

Supervised Fine-Tuning Analysis

Figure 6 sweeps SFT epochs from 0.5 to 5 for the LLaMA-3.2-3B model, comparing models with and without thinking-augmented mid-training. Several patterns emerge:

• TPT mid-training provides a substantially higher starting point. At 0.5 epochs, the TPT model already achieves meaningful performance on AIME24 (the vanilla model is at near-zero), and this advantage is maintained at every subsequent epoch. This supports the claim that TPT builds reasoning foundations during mid-training that SFT then elicits.

• Performance improves with more SFT data. Across all benchmarks, scores increase monotonically through 5 epochs with no overfitting observed. LiveCodeBench appears not to have converged, "suggesting the potential for further gains with extended training."

• The gap between TPT and vanilla can widen during SFT. On AIME24, the TPT model goes from near-zero at 0.5 epochs to ~15% at 5 epochs, while the vanilla model remains near-zero throughout. This means the SFT process amplifies the differences created during mid-training rather than washing them out.

• MATH-500 shows a different pattern: both models improve similarly during SFT, with the TPT advantage established at epoch 0.5 being maintained but not substantially widened. This is likely because MATH-500 is substantially easier than AIME24 (the baseline without mid-training already achieves ~60%), and the additional improvements from more epochs are incremental for both variants.

Ablation Studies and Robustness Checks

Thinking trajectory generation strategy (Table 4): Three alternatives to the default generation method are tested on LLaMA-3B with 40B tokens of mid-training, evaluated after SFT on 8 benchmarks. Custom back-thinking model: fine-tuning DeepSeek-R1-Distill-Qwen-7B on SFT data to generate thinking given the final response and original question, then using this model to generate trajectories given pre-training documents as input. Results show marginal improvements over default on MATH-500 (73.8% vs. 72.0%), AIME24 (14.7% vs. 11.7%), MMLUPro (53.9% vs. 52.3%), and JEEBench (41.7% vs. 36.0%), but a decline on HumanEval (57.3% vs. 64.6%). Prompt with random focus point: modifying the prompt to instruct the model to focus on a randomly selected position within the document. This yields minimal differences across all benchmarks (e.g., AIME24: 12.4% vs. 11.7% default, MATH-500: 72.2% vs. 72.0%). The paper concludes that these alternatives "introduce extra implementation complexity due to the need for custom fine-tuning or dynamic prompt modification" without commensurate gains.

Scaling the thinking generation model size (Table 4, bottom row): This is the most surprising and consequential ablation. Using DeepSeek-R1-Distill-Qwen-1.5B (smaller than the default 7B model) for thinking generation improves performance across most benchmarks compared to the default: AIME24 increases from 11.7% to 17.7% (+6.0 points), AIME25 from 15.0% to 16.5% (+1.5 points), LiveCodeBench from 18.7% to 21.4% (+2.7 points), and JEEBench from 36.0% to 41.3% (+5.3 points). MATH-500 improves from 72.0% to 75.5% (+3.5 points). Only GPQA-Diamond shows a slight decline (36.6% vs. 37.7%). The paper speculates that "the smaller model may generate trajectories that are better suited for downstream model learning," aligning this finding with observations from OpenThoughts (Guha et al., 2025). This is a materially important result: it suggests that for TPT to work well, the thinking generator should be matched to the student model's capacity, not maximized for its own reasoning strength. The paper does not systematically explore this matching — no experiments vary the student model size while holding the generator constant, or vice versa — identifying this as an area that "warrants further investigation."

Impact of mid-training token budget (Figure 5): This is not a traditional ablation (nothing is removed) but rather a scaling analysis that demonstrates the approach's behavior as a key hyperparameter varies. For both Qwen2.5-1.5B and LLaMA-3.2-3B, performance improves monotonically from 0B to 100B mid-training tokens across AIME24, MATH-500, GPQA-Diamond, and LiveCodeBench. The curves show no sign of plateauing at 100B tokens, indicating that the paper's chosen budget is not at the point of diminishing returns. The "0B" data point (direct SFT without mid-training) is particularly informative: for LLaMA-3B on AIME24, performance is near-zero without mid-training, rises gradually through 40B tokens, and reaches ~15% at 100B tokens. This demonstrates that SFT alone cannot teach AIME-level reasoning to a model that lacks the foundational capabilities, and that these capabilities are acquired gradually during mid-training.

Impact of SFT data size (Figure 6): Sweeping SFT epochs from 0.5 to 5 demonstrates several robustness properties. First, TPT models maintain their advantage over vanilla counterparts at every SFT data scale tested, meaning the benefit is not an artifact of a particular SFT budget. Second, performance improves monotonically with more SFT data with no overfitting, indicating that the models are underfitting rather than overfitting the SFT data at 5 epochs. Third, the gap between TPT and vanilla models does not close with more SFT — it either stays constant (MATH-500, GPQA-Diamond) or widens (AIME24), indicating that SFT cannot compensate for the absence of thinking-augmented pre-training.

Vanilla mid-training control (Table 6): Training the LLaMA-3.2-3B model on the same text data for 40B tokens without thinking augmentation, followed by the same SFT, produces "minimal impact on most benchmarks." Specifically, AIME24 is 5.0% with vanilla mid-training vs. 5.8% with direct SFT (no mid-training) — essentially no change. HumanEval drops from 45.7% to 36.0%, and LiveCodeBench drops from 13.9% to 5.7%. This negative result is arguably the most important ablation in the paper because it isolates the effect of thinking augmentation from the effect of simply training longer on relevant data. The finding that additional vanilla training can degrade performance on some tasks (particularly code) while TPT on the same underlying documents improves performance is strong evidence that the thinking trajectories, not the additional training tokens per se, are responsible for the observed gains.

Reference model comparisons (Tables 1, 2, 3, 7): Several external reference points provide context for the absolute performance levels achieved. LLaMA-2-7B (2T tokens) achieves 28.8% average on base model benchmarks, which TPT-8B surpasses at 43.9% with only 100B tokens (Table 1). LLaMA-3.1-8B (15T tokens) achieves 46.8%, which TPT-8B approaches within 2.9 percentage points. After SFT, TPT-8B outperforms LLaMA-3.1-8B-Instruct on all five challenging benchmarks in Table 2. GPT-4o scores (Table 3) are comparable to TPT-7B on several benchmarks but substantially lower on AIME24 (9.3% vs. 57.5%) and JEEBench (44.8% vs. 73.6%). These comparisons are not controlled experiments (the models differ in architecture, training data composition, and SFT procedures), but they establish that TPT enables models trained on modest compute budgets to reach performance levels competitive with or exceeding models trained on far larger budgets.

Critical Assessment

The central claim of this paper — that augmenting pre-training data with automatically generated thinking trajectories substantially improves data efficiency and downstream reasoning performance — is well-supported by the experiments, but the specific magnitude of the claimed improvements and their generality warrant careful qualification.

Does TPT actually improve data efficiency by 3×? The 3× figure originates from Figure 1a, where the TPT model reaches the same average score at roughly 33B tokens that vanilla pre-training reaches at roughly 100B tokens. This is a fair visual reading of the plot, but the paper provides no interpolation methodology, no specific accuracy threshold for the "same level of performance" determination, and no confidence intervals around the estimated crossover point. The 3× figure should be understood as an approximate, order-of-magnitude claim rather than a precise measurement. More importantly, this efficiency gain is demonstrated for 8B-parameter models trained on the specific data mixture of MegaMath-Web-Pro-Max and FineWeb-Edu. Whether the same efficiency multiplier holds for different model scales, different data mixtures, or different domains is untested. The paper acknowledges this limitation implicitly by only claiming the 3× figure for the specific configuration tested, but readers should not assume the multiplier is universal.

Furthermore, the 3× efficiency calculation excludes the cost of generating the thinking trajectories. The paper reports that trajectory generation takes roughly 20k A100 GPU hours for 100B training tokens (Appendix A.1). In a deployment scenario where thinking trajectories are generated once and reused across multiple training runs, this amortization makes the omission reasonable. But for a single training run — which is the scenario described in the paper — the total cost includes both generation and training. A fully honest efficiency calculation would compare: (cost of generating trajectories + cost of training on 33B augmented tokens) vs. (cost of training on 100B vanilla tokens). The paper does not perform this calculation. Based on the reported numbers — 20k A100 hours for generation plus roughly 1 week on 32 GPUs for training (from Appendix A.1) — the generation cost is substantial and would meaningfully reduce the net efficiency gain. The 3× figure should therefore be interpreted as a training-time efficiency gain conditional on pre-existing thinking trajectories, not as an end-to-end efficiency gain.

Does TPT generalize beyond mathematical reasoning? The largest absolute improvements consistently appear on mathematical reasoning benchmarks: GSM8k (+30.9 points), MATH (+12.7 points), AIME24 (+34.2 points after SFT). Non-mathematical benchmarks show smaller but still meaningful gains: BoolQ (+8.5 points), MMLU (+20.5 points), MMLUPro (+15.9 points), GPQA-Diamond (+17.5 points after SFT). The pattern is clear: TPT helps most on tasks that require multi-step logical deduction from foundational principles — exactly the type of content where decomposing complex tokens into reasoning steps should be most beneficial. Tasks requiring factual recall or shallow pattern matching benefit less. This is not a weakness of the method per se, but it does bound the claims: TPT is primarily a reasoning capability amplifier, not a general-purpose data efficiency multiplier. The paper's abstract frames the contribution broadly ("improve the data efficiency of large language model training"), but the experimental evidence supports a narrower interpretation focused on reasoning-intensive tasks.

The domain composition of the training data may partially explain this pattern. The paper uses MegaMath-Web-Pro-Max (math-intensive) and FineWeb-Edu (educational content) with a sample weight of 0.125 on FineWeb-Edu — meaning math content is heavily up-sampled. The thinking trajectories for mathematical documents will naturally involve step-by-step derivations, which are the most natural fit for the thinking augmentation approach. For other domains (historical narratives, descriptive text, dialogue), thinking trajectories may add less value both because the original tokens are more learnable and because the "reasoning" about such content is less structured. The paper does not test TPT on a domain-balanced corpus, leaving open the question of whether the gains are driven primarily by the math-heavy data composition.

Does TPT work for models beyond the tested scale range? The experiments span 1.5B to 8B parameters, with consistent gains across this range. However, the paper does not test models at the scale where data scarcity is most acute — the 70B–405B range where training data requirements are in the tens of trillions of tokens. At these scales, several factors could change: the base models may already have strong reasoning capabilities that leave less room for TPT improvement; the cost of generating thinking trajectories at sufficient scale may become prohibitive; and the ratio of thinking trajectory length to document length may shift as the thinking generator (itself a large model) may produce disproportionately long or short analyses for the content. The paper's claim that TPT "is highly scalable" is based on the offline, decoupled nature of trajectory generation rather than on empirical demonstration at scale. This is a reasonable argument but not an empirical finding.

How robust is TPT to the choice of thinking generation model? The ablation in Table 4 provides partial evidence: switching from a 7B to a 1.5B generator actually improves results, suggesting the method is not fragile to generator capacity. However, the paper tests only two generator models, both from the same family (DeepSeek-R1-Distill-Qwen). Whether TPT works with generators from different model families, with different reasoning styles, or with different training procedures is unknown. The finding that a smaller generator can outperform a larger one is intriguing but unexplained — the paper speculates about "trajectories that are better suited for downstream model learning" without defining what "better suited" means or how to predict it. This is a significant gap if practitioners want to choose an optimal generator without running expensive ablation experiments.

Are the baselines fair and comprehensive? For mid-training, the primary baseline (OpenR1 models fine-tuned on the same SFT data) is well-chosen: it controls for SFT data quantity and base model initialization, isolating the effect of TPT mid-training. For pre-training from scratch, the comparison against LLaMA-3.1-8B (15T tokens) is useful for context but not controlled — the models differ in training data, architecture details, and training procedures beyond just the thinking augmentation. A more informative baseline would be an 8B model trained on 300B tokens of vanilla data (3× the TPT budget), which would directly test whether the claimed 3× efficiency gain persists when the vanilla model is given proportionally more compute. The paper does not run this experiment.

Does TPT's performance depend on the SFT phase? The most dramatic improvements appear after SFT (Tables 2 and 3), not at the base model stage (Table 1). The base model improvements are substantial but not transformative: TPT-8B achieves 43.9% average vs. 26.2% for vanilla — a large relative improvement but still far below LLaMA-3.1-8B's 46.8% (Table 1). After SFT, TPT-8B outperforms LLaMA-3.1-8B-Instruct on all benchmarks in Table 2. This pattern suggests that TPT's primary mechanism is building a better foundation for subsequent fine-tuning rather than improving the base model's zero-shot or few-shot performance per se. The paper does not explicitly make this distinction, but it is important for practitioners: TPT may be most valuable not as a standalone pre-training improvement but as part of a pipeline where thinking-augmented pre-training is followed by targeted SFT. The paper does not test whether vanilla pre-training with a larger SFT budget could match TPT's post-SFT performance, which would help determine whether TPT is truly more efficient or simply shifts capability acquisition from SFT to pre-training.

Missing experiments that would strengthen the paper:

• Scaling the vanilla model to 300B tokens (3× the TPT budget) to directly test whether TPT's claimed 3× efficiency gain holds in a FLOPs-matched comparison. Currently, the comparison is at equal token budgets, not equal total compute budgets.
• Testing TPT on a domain-balanced corpus without heavy math up-sampling to determine whether the gains are primarily driven by mathematical reasoning or generalize broadly.
• Systematic variation of the thinking generator model across multiple architectures, scales, and training procedures to characterize the teacher-student matching phenomenon identified in the ablation.
• Statistical confidence measures for the reported scores, particularly on small benchmarks like AIME24 (30 problems) where single-question swings can change scores by several percentage points.
• Ablation of the sample weight (0.125 on FineWeb-Edu) to determine how sensitive results are to the domain balance of the training data.
• Direct comparison with BoLT and Reasoning CPT using the paper's own experimental setup (same base model, same data, same training budget) to empirically validate the claim that TPT outperforms or matches these prior methods while being simpler.

Bottom line on central claims: The claim that TPT substantially improves reasoning performance across multiple training configurations is robustly supported. The claim of 3× data efficiency is directionally correct but lacks the precision and comprehensive accounting to be treated as a verified constant. The claim that TPT's simplicity (single prompt, single pass, no iterative refinement) is sufficient for strong gains is well-supported by the ablation showing minimal benefits from more complex alternatives. The claim of broad applicability beyond the tested model scales and domains remains an extrapolation from limited empirical evidence and should be treated as such.

6. Limitations and Trade-offs

The Thinking Trajectory Generation Cost Is Unaccounted for in the Headline Efficiency Claims

The paper's central claim is that TPT improves data efficiency by a factor of 3× (Figure 1a, Section 1: "TPT enhances the data efficiency of LLM pre-training by a factor of 3"), and the experimental results demonstrate that a TPT model reaches the same downstream performance at approximately 33B training tokens that a vanilla model reaches at approximately 100B tokens. However, this comparison treats the cost of generating the thinking trajectories as zero — the 3× figure compares only the tokens processed during model training, not the total compute required to produce the augmented dataset.

The paper reports in Appendix A.1 that "the entire data generation pipeline takes roughly 20k A100 GPU hours to yield enough data for 100B training tokens." Training an 8B model on 100B tokens takes approximately one week on 32 MI300 GPUs (Appendix A.1), which — assuming MI300 GPUs are roughly comparable to A100s in throughput for this workload — translates to approximately 5,400 GPU-hours. The generation cost (20k A100 hours) is therefore approximately 3.7× larger than the training cost for the 100B-token run.

This has direct implications for the practical efficiency claims. When TPT achieves the same performance at 33B tokens that vanilla achieves at 100B tokens, the training cost savings are roughly 3× (67B fewer tokens to process). But the generation cost must be amortized across training runs. If the thinking trajectories are generated once and used for a single training run, the total cost is generation (20k hours) + training on 33B tokens (~1.8k hours) ≈ 21.8k GPU-hours, compared to vanilla training on 100B tokens (~5.4k hours). The end-to-end cost is roughly 4× higher for TPT in this scenario — the exact opposite of the claimed efficiency improvement. The paper implicitly assumes the generation cost is amortized across many training runs (hyperparameter sweeps, different model sizes, or iterative development), which is reasonable in research settings but may not hold for one-off production training runs.

The paper explicitly flags this issue in Section 3.2 when discussing the broader difficulty estimation problem — "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" — but this acknowledgment appears in the context of difficulty estimation (which TPT does not require), not in the context of TPT's own generation cost. The paper never provides an end-to-end compute accounting that includes trajectory generation, and the abstract and introduction present the 3× efficiency claim without qualification.

Mitigation status: The paper does not address this tradeoff directly. The generation cost is reported transparently in the appendix but is not factored into any efficiency calculation or headline claim. The paper suggests no method for reducing generation cost (e.g., using smaller generators, caching trajectories across documents with similar content, or generating trajectories only for a subset of the corpus). The finding that a smaller generator (1.5B vs. 7B) can produce better trajectories (Table 4) offers a partial path to cost reduction — smaller models are cheaper to run — but the paper does not quantify the generation cost difference between the two generator sizes or propose cost-aware generator selection as a design principle.

The 3× Data Efficiency Claim Is Demonstrated Only for a Single Model Size and a Math-Heavy Data Mixture

The paper's headline efficiency claim — that TPT reduces required training tokens by a factor of 3 to achieve the same level of base model performance — is supported by Figure 1a, which shows two training curves (vanilla and TPT) for 8B-parameter models trained on a specific data mixture. The paper extrapolates broadly from this single experiment, stating in the abstract that TPT "enhances the data efficiency of LLM pre-training by a factor of 3" and describing the method as a "universal methodology."

However, the experiment that generates this claim uses a very specific configuration: 8B-parameter models following the LLaMA-3 architecture, trained on a mixture of MegaMath-Web-Pro-Max (math-intensive web content) and FineWeb-Edu (educational content) with a sample weight of 0.125 on FineWeb-Edu — meaning the training data is heavily skewed toward mathematical content. The thinking pattern analysis in Section 4 demonstrates that mathematical content generates the longest thinking trajectories (Figure 4, left panel: Mathematics averages ~1,900 thinking tokens vs. ~1,100 for Social Service), and the paper's own results show that TPT's largest absolute gains are on mathematical reasoning benchmarks (GSM8k: +30.9 points, MATH: +12.7 points, AIME24: +34.2 points after SFT), with smaller gains on general knowledge tasks (BoolQ: +8.5 points, MMLU: +20.5 points). This pattern raises the possibility that the 3× efficiency factor is specific to math-heavy training data and math-focused evaluation — the very regime where thinking trajectories are longest and most directly relevant to downstream tasks.

The paper does not report an equivalent scaling curve (like Figure 1a) for any other model size, any other data mixture, or any non-math-focused evaluation. The mid-training experiments (Section 3.3) test multiple model sizes (1.5B, 3B, 7B) and two model families, but these experiments use a fixed 100B-token budget and evaluate after SFT — they do not measure the training-token crossover point where TPT matches vanilla performance, so they cannot confirm or refute the 3× multiplier. The constrained-data experiment (Section 3.2) uses 8B models and 40B training tokens but does not provide a direct efficiency multiplier.

Mitigation status: The paper does not acknowledge this limitation as such. The abstract and introduction present the 3× figure as a general property of the method. Section 3.3's experiments across model sizes provide some evidence of generality for the existence of TPT's benefit (it helps across scales), but not for the magnitude of the efficiency gain. The paper would be strengthened by either (a) demonstrating the 3× multiplier in at least one additional configuration (different model size, different data mixture, different domain emphasis) or (b) qualifying the claim to specify the conditions under which it was measured.

Hard Problems That Require Capabilities Beyond the Thinking Generator's Reach Receive Minimal Benefit

The paper's method relies on an off-the-shelf LLM (DeepSeek-R1-Distill-Qwen-7B or Qwen3-8B) to generate the thinking trajectories that make difficult tokens more learnable. This creates an inherent capability ceiling: the thinking generator can only decompose reasoning that it can itself perform. If a training document contains content that neither the generator nor the student model can reason about — a mathematical proof at the Olympiad level, a complex multi-step scientific argument, code that implements a novel algorithm — the generated thinking trajectory will be incomplete, incorrect, or superficial, providing limited additional learning signal.

The paper provides indirect evidence for this ceiling in the pre-training results. Table 1 shows that TPT-8B achieves 50.1% on GSM8k and 21.8% on MATH after 100B training tokens — substantial improvements over vanilla (19.2% and 9.1%) but still far from ceiling performance. The post-SFT results in Table 2 show more dramatic gains (AIME24: 35.2%, MATH-500: 82.4%), but AIME is specifically designed to be challenging for frontier models, and the 35.2% score, while impressive relative to the vanilla baseline (1.0%), still means the model fails to solve nearly two-thirds of AIME problems. The paper does not break down performance by problem difficulty within benchmarks, so it is impossible to determine whether the unsolved problems are precisely those where the thinking generator's own capabilities were insufficient to produce useful trajectories.

The mid-training results in Table 3 provide additional perspective. TPT-Qwen2.5-7B achieves 57.5% on AIME24 and 39.4% on AIME25 — strong results that exceed the DeepSeek-R1-Distill-Qwen-7B baseline on some benchmarks. But the thinking generator for these experiments is DeepSeek-R1-Distill-Qwen-7B (or a smaller variant). This means the generator is producing thinking trajectories about math content that it can itself solve at roughly a 50-55% level on AIME (based on the DeepSeek distillation baseline's score). For the hardest AIME problems — those that the 7B distillation model cannot solve — the generated trajectories likely contain flawed or incomplete reasoning. Training on these trajectories may still provide some benefit (the model learns general patterns of mathematical exposition), but it cannot teach problem-solving strategies that the generator itself does not possess.

This limitation is conceptually analogous to the "verifier over-optimization" limitation — just as test-time search is bounded by verifier quality, TPT is bounded by thinking generator quality. The paper does not explore this boundary: no experiment tests whether TPT's benefits saturate or reverse when the training data contains problems substantially harder than the generator can solve, and no experiment tests whether using a substantially stronger generator (e.g., DeepSeek-R1-671B or a frontier API model) would push performance further on the hardest benchmarks.

Mitigation status: The paper does not explicitly address this limitation. The finding that a smaller generator (1.5B) can outperform a larger one (7B) on some benchmarks (Table 4) is tangentially relevant — it suggests that generator capability is not monotonically related to downstream performance — but this finding is left unexplained and does not address the capability ceiling question directly. The paper identifies "exploring more powerful thinking generation models" as future work (Section 7), which implicitly acknowledges the limitation without analyzing its current impact.

The Method Has Only Been Validated on a Single Model Architecture Family and a Narrow Set of Data Sources

All experiments in the paper use transformer-based autoregressive language models following the LLaMA architecture (for pre-training from scratch) or Qwen2.5/LLaMA-3 architectures (for mid-training). The training data across all experiments comes from exactly two sources: MegaMath-Web-Pro-Max and FineWeb-Edu. While the mid-training experiments span two model families (Qwen and LLaMA) and three model sizes (1.5B, 3B, 7B), this still represents a narrow slice of the design space for large language model training.

Several aspects of the paper's setup may interact with TPT's effectiveness in ways that do not generalize:

• Data composition: As discussed above, the training data is heavily skewed toward mathematical and educational content. FineWeb-Edu is specifically an educational subset, and MegaMath is explicitly math-focused. The paper does not test TPT on a general web crawl (e.g., C4, Dolma, or the full FineWeb), on code-heavy corpora, on multilingual data, or on domain-specific corpora (legal, medical, financial). The thinking pattern analysis in Section 4 suggests that thinking trajectory length — and therefore the dynamic compute allocation mechanism — varies substantially by domain. On a corpus dominated by short-form social media content, news articles, or product reviews, thinking trajectories may be uniformly short, and TPT's benefits may be proportionally smaller.

• Model architecture: All tested models are standard dense transformers. The paper does not test TPT with mixture-of-experts architectures, state-space models, or other architectural variants. While there is no obvious reason TPT would fail for these architectures, the interaction between thinking augmentation and architectural inductive biases is unexplored.

• Training scale: The largest experiment uses 8B parameters and 100B tokens. Modern frontier models are trained at scales of 70B–405B parameters and 10T–15T tokens. Whether the 3× efficiency multiplier persists at these scales — where the base models already have stronger inherent reasoning capabilities and may benefit less from decomposed training signals — is unknown. The paper's mid-training results on already-strong models (Qwen2.5-Math-7B) provide some evidence that TPT helps even when the base model is capable, but these experiments use only 100B tokens of mid-training, not a full pre-training run from scratch.

• Thinking generator model: All experiments use DeepSeek-R1-Distill-Qwen variants (7B or 1.5B) or Qwen3-8B as the thinking generator. These are all models from the Qwen family with similar pre-training and fine-tuning procedures. The paper provides no evidence that TPT works with generators from other model families (e.g., LLaMA-based reasoning models, proprietary API models) or with generators trained with different reasoning elicitation methods (e.g., RL-based reasoning training vs. distillation-based).

Mitigation status: The paper does not claim to have tested on a broader range of architectures, data sources, or scales, and the generalization question is acknowledged only implicitly through the scope of the reported experiments. The mid-training experiments across two model families are the strongest evidence for cross-family generalization, but they still use the same data sources and the same generator model family. The paper's abstract describes TPT as a "universal methodology" — a claim that the current experimental scope does not fully support.

The Paper Provides No Statistical Confidence Measures for Any Reported Result

Every quantitative result in the paper is reported as a point estimate without confidence intervals, standard errors, or statistical significance tests. This is consequential for several reasons:

• Small benchmark sizes: Several evaluation benchmarks have very few test examples. AIME24 and AIME25 each contain only 30 problems. At this sample size, a single additional correct answer changes the reported score by 3.3 percentage points. The paper reports AIME24 scores of 35.2% (TPT-8B→SFT, Table 2), 57.5% (TPT-Qwen2.5-7B, Table 3), and 28.5% (TPT-Qwen2.5-1.5B, Table 3) — all of which could shift by several points due to sampling variance alone. The paper attempts to mitigate this by averaging over 64 samples per problem (Appendix A.2), which reduces variance from generation stochasticity but does not address variance from the limited problem set — if the 30 AIME24 problems are not fully representative of the difficulty distribution, the reported Pass@1 is a noisy estimate of true performance.

• No cross-validation for strategy selection: Unlike the prior-sections reference paper which uses two-fold cross-validation to select compute-optimal strategies, TPT does not perform any strategy selection that would require held-out validation. However, the paper reports results from a single training run for each configuration — there is no assessment of training variance (how much would results change with a different random seed for initialization, data ordering, or dropout?) or evaluation variance. The constrained-data experiment (Section 3.2) uses a single random sample of 10B tokens from the full corpus; results could vary substantially with a different random draw, but this sensitivity is not measured.

• Comparing across models without error bars: Tables 1, 2, and 3 present scores for TPT models alongside baseline models (LLaMA-3.1-8B, DeepSeek-R1-Distill, GPT-4o). These comparisons are made at face value — e.g., "TPT-8B→SFT outperforms LLaMA-3.1-8B-Instruct on every benchmark" (Section 3.1) — without any indication of whether the margins are statistically meaningful. A 1.5 percentage point difference on MATH-500 (82.4% vs. 80.9% if LLaMA-3.1-8B-Instruct scored that) could easily be noise, while a 34.2 point difference on AIME24 (35.2% vs. ~1% for vanilla) is clearly signal. The paper treats all differences uniformly as evidence of improvement, without distinguishing statistically robust findings from within-noise comparisons.

• No ablation significance: The ablation study (Table 4) compares three alternative thinking generation strategies against the default on 8 benchmarks using a single training run each. The paper concludes that "alternative thinking generation strategies yield slight improvements over our default method" and that the back-thinking model and random focus point "demonstrate only marginal gains." Without variance estimates, it is impossible to determine whether these "marginal gains" are distinguishable from zero — the +1.8 point improvement on MATH-500 for the back-thinking model could be a real effect or noise.

Mitigation status: The paper makes no attempt to address this limitation — no confidence intervals are reported, no significance tests are performed, and no justification is offered for the omission. This is a meaningful gap in experimental rigor that affects the interpretability of the reported results, particularly for the smaller benchmarks and the ablation comparisons. The paper's claims about relative improvements (e.g., "a 3× increase" for AIME24 from 5.8% to 18.6%) are directionally credible given the magnitude of the differences, but the precise scores and the ordering of methods with small performance gaps should be treated as approximate.

The Revision Model's Correct-to-Incorrect Reversion Problem Has No Analogue in TPT, but a Related Degeneracy May Exist: Over-Optimization on Thinking-Trajectory Patterns

While this limitation is more speculative than the others and the paper provides no direct evidence for it, it represents a structural risk that a practitioner deploying TPT should consider. The thinking trajectories are generated by a specific model (DeepSeek-R1-Distill-Qwen-7B) with a specific reasoning style, including characteristic phrases ("Okay, so I need to figure out...", "Hmm, let's start by...", "Wait, let me check that again..."), patterns of self-correction, and particular ways of structuring mathematical derivations. The student model is trained on hundreds of billions of tokens of this stylized reasoning text, which constitutes a substantial fraction of its total training data (since thinking trajectories are often 2–3× the length of the original documents).

The risk is that the student model learns to mimic the surface patterns of the thinking generator's reasoning style without developing robust underlying reasoning capabilities. This is analogous to the "correct-to-incorrect reversion" problem — the model learns the format of revisions without learning when not to revise. TPT-trained models might learn to produce fluent, plausible-sounding chain-of-thought reasoning (because they've been trained on millions of examples of such reasoning) while remaining substantively wrong on problems that require novel reasoning not present in the training trajectories. The thinking trajectories serve as both a blessing (they make difficult content learnable) and a potential curse (they teach a specific reasoning aesthetic that may not generalize).

The paper provides no direct test for this failure mode. The strong post-SFT results on challenging benchmarks (AIME24, AIME25, LiveCodeBench) provide some evidence against it — if the models were merely mimicking reasoning style, they would not achieve 35–57% on AIME, which requires genuine mathematical problem-solving. However, the thinking generator (DeepSeek-R1-Distill-Qwen-7B) itself achieves ~50–55% on AIME, and the student model's performance is in the same range. This raises the possibility that the student is effectively learning to replicate the generator's capabilities (including its limitations) rather than developing genuinely independent reasoning.

Mitigation status: The paper does not address this risk. The finding that a smaller generator (1.5B) can produce better training data than a larger one (7B) — Table 4 — could be interpreted as evidence that overly sophisticated or stylized trajectories are counterproductive, which would partially support the concern. But without experiments that systematically vary the generator's reasoning style (e.g., by using generators from different model families or with different prompting strategies) and measure whether the student's reasoning patterns track the generator's, this remains a theoretical concern rather than an empirically demonstrated limitation. The paper identifies "exploring more powerful thinking generation models" as future work (Section 7) but does not frame this as a potential robustness concern.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing that shifts the data engineering conversation from selecting which tokens to train on to transforming how difficult tokens are presented to the model. Prior to TPT, the dominant paradigm for maximizing pre-training data utility—exemplified by data selection frameworks like Lin et al. (2024) and Mindermann et al. (2022)—treated tokens as atomic units whose value could be assessed and filtered. TPT's core insight is that this framing misses a critical bottleneck: the most valuable tokens in a corpus (correct answers to complex reasoning problems, conclusions of multi-step derivations, key insights in scientific arguments) are often the least learnable under standard next-token prediction because the compression required to predict them from preceding context is too extreme for a given model capacity. The paper names this the learnability problem and demonstrates that decomposing difficult tokens into intermediate reasoning steps—via automatically generated thinking trajectories—can improve data efficiency by approximately 3× (Figure 1a) without changing the model architecture, training objective, or underlying document corpus.

This reframing is not a full paradigm shift—it builds on existing data engineering pipelines rather than replacing them—but it opens a genuinely new axis for optimization. The paper explicitly positions TPT as "orthogonal to rewriting based approaches" (Section 6) and complementary to standard curation techniques like filtering, deduplication, and domain balancing. You can apply FineWeb's entire quality pipeline and then add thinking augmentation. The two operations address fundamentally different failure modes: curation removes content that is intrinsically unhelpful (spam, noise, repetition), while TPT makes intrinsically helpful but unlearnable content more accessible to the training objective. This division of labor—curation for quality, augmentation for learnability—provides a cleaner conceptual framework than prior approaches that conflated these two concerns under the single umbrella of "data quality."

The paper also introduces the notion of training-time compute scaling as a natural consequence of reasoning-based augmentation. The analogy to test-time compute scaling (Jaech et al., 2024) is not merely rhetorical—it is mechanistic. Test-time scaling works because harder problems trigger longer chain-of-thought reasoning at inference, consuming more compute and improving accuracy. TPT works because harder content naturally triggers longer thinking trajectories during data generation (Figure 4: Mathematics documents average ~1,900 thinking tokens vs. ~1,100 for Social Service; "Advanced Reasoning" documents generate ~50% more tokens than "No Reasoning" documents), consuming more training tokens and improving learnability. The thinking generation model acts as an automatic complexity detector and training compute allocator, distributing tokens across the corpus in proportion to each document's reasoning demands—without any explicit difficulty estimation, quality classifier, or per-document hyperparameter tuning. This mechanism connects two previously disconnected research communities (data engineering and inference-time scaling) by showing that the same principle—allocate more compute to harder content—governs both effective training and effective inference.

A more subtle contribution is the paper's reconciliation of conflicting intuitions about synthetic data for pre-training. Prior work on synthetic data generation for LLM training falls into two broad camps: one that generates entirely new training documents from scratch (e.g., Phi series models; Gunasekar et al., 2023) and one that minimally transforms existing documents through paraphrasing or rewriting (Maini et al., 2024; Nguyen et al., 2025). These approaches make opposite bets about what matters most: the first camp bets that content authorship (who/what generates the text) is primary, while the second bets that surface form (how existing content is expressed) is primary. TPT resolves this tension by demonstrating that neither authorship nor surface form is the key variable. What matters is whether the training text decomposes complex predictions into learnable intermediate steps. The thinking trajectories are synthetic (authored by an LLM) but they are anchored to existing documents (not generated from scratch) and they modify the cognitive structure of the training signal (not just the surface form). This suggests a more nuanced taxonomy for synthetic data methods in pre-training: the axis that matters is not "synthetic vs. organic" or "rewritten vs. original" but rather "does the data make difficult tokens more learnable?"

The paper's empirical results—particularly the finding that a 100B-token TPT-trained 8B model approaches the performance of LLaMA-3.1-8B trained on 150× more data (Table 1)—challenge the implicit assumption in the scaling laws literature that data quantity and data quality are separate knobs. TPT demonstrates that they interact: the effective quantity of training data can be multiplied by improving its learnability, and this multiplier can be substantial (3× in the paper's configuration). This has implications for how organizations should think about their data strategy. Rather than the prevailing paradigm of "acquire more data at any cost," the results suggest a regime where it is more cost-effective to augment existing data to improve its learnability than to acquire or generate new data. This is particularly relevant given the data wall that the paper identifies—"the pool of human-authored, organically generated data on the web is finite and has been largely exhausted by existing frontier models" (Section 1)—since it offers a path to continued scaling that depends on compute for data transformation rather than on access to new data sources.

The paper also makes several research directions less attractive by negative implication. The ablation study showing that a smaller thinking generator (1.5B parameters) can outperform a larger one (7B parameters) for downstream training (Table 4: AIME24 improves from 11.7% to 17.7%) suggests that naively scaling the thinking generator model is not the right optimization target. This finding redirects attention away from "build the strongest possible reasoning model to generate training data" and toward "understand what properties of generated trajectories make them effective for student learning." Similarly, the finding that custom fine-tuning of the thinking generator (back-thinking model) and prompt engineering (random focus points) yield only marginal improvements over the simple default approach (Table 4) suggests that complexifying the generation process is unlikely to be the most productive research direction—the low-hanging fruit lies in better understanding teacher-student compatibility and trajectory selection, not in better trajectory generation per se.

Follow-Up Research This Work Enables

Characterizing the teacher-student matching function for thinking augmentation. The paper's most surprising empirical finding is that a 1.5B-parameter thinking generator produces trajectories that yield better downstream performance than a 7B-parameter generator (Table 4: AIME24 17.7% vs. 11.7%). The paper speculates that "the smaller model may generate trajectories that are better suited for downstream model learning" but provides no mechanism. A strong follow-up would systematically vary both the thinking generator capacity (e.g., 0.5B, 1.5B, 3B, 7B, 14B, 32B) and the student model capacity (e.g., 1.5B, 3B, 7B) while holding the training data, SFT procedure, and evaluation benchmarks constant. The output would be a matrix showing which generator-student pairs are optimal, revealing whether there is a consistent "generator-to-student ratio" that maximizes downstream performance. The hypothesis to test is whether the optimal generator is approximately matched to the student in capacity (so that trajectories are neither too simple to provide learning signal nor too complex to be compressed) or whether a consistent "smaller is better" pattern holds. This experiment would require roughly 24 training runs (6 generators × 4 student sizes, or a sparser grid) at the 40B-token mid-training scale used in Table 4, making it feasible with the paper's reported compute budget. A negative result—finding no systematic relationship between generator capacity and student performance—would be equally informative, suggesting that trajectory quality depends on properties of the generator that are not captured by parameter count.

Does TPT's benefit come from the thinking content or from the increased token allocation to difficult documents? A clean ablation would disentangle two mechanisms that are confounded in TPT: (1) the cognitive decomposition provided by thinking trajectories (breaking complex tokens into simpler intermediate steps) and (2) the compute allocation effect (difficult documents receive more training tokens because thinking trajectories are longer). To isolate mechanism (1), you would train a model on thinking-augmented data but truncate all thinking trajectories to a uniform length (e.g., 1,000 tokens regardless of document difficulty), removing the dynamic compute allocation while preserving the decomposition. To isolate mechanism (2), you would train a model on the original documents but replicate each document a variable number of times proportional to the thinking trajectory length that document would have generated—providing more training tokens to difficult content without any decomposition. Comparing these variants against full TPT and vanilla baselines would reveal how much of the 3× efficiency gain is attributable to each mechanism. The paper's thinking pattern analysis (Figure 4) provides the trajectory length distributions needed to design the replication schedule for mechanism (2). This experiment matters because if the compute allocation effect dominates, simpler methods (just repeating difficult documents) could achieve comparable gains without the generation cost; if the decomposition effect dominates, it implies that making content learnable requires genuinely transforming the training signal, not just allocating more compute to it.

Testing TPT on a domain-balanced corpus without math up-sampling. The paper's training data uses a sample weight of 0.125 on FineWeb-Edu relative to MegaMath-Web-Pro-Max, meaning math content is heavily up-sampled. The thinking pattern analysis (Figure 4) shows that mathematical content generates the longest thinking trajectories, and the largest performance gains are on mathematical reasoning benchmarks (GSM8k: +30.9 points, MATH: +12.7 points). A critical stress-test would replicate the abundant-data pre-training experiment (Section 3.1) using a domain-balanced corpus—perhaps a random sample from the full FineWeb without the math-focused MegaMath component, or a standard benchmark corpus like C4 or Dolma—with no domain-specific up-sampling. The key question: does the 3× efficiency multiplier persist when the training data distribution matches the natural distribution of web content rather than being skewed toward the domain where thinking augmentation is most natural? If the multiplier drops to, say, 1.5× on a balanced corpus, it would substantially narrow the range of deployments where TPT's generation cost is justified. If the multiplier holds, it would demonstrate that thinking augmentation is a genuinely general-purpose technique rather than a math-specific one. The benchmark suite should emphasize general knowledge and language understanding tasks (MMLU, MMLU-Pro, HellaSwag, WinoGrande) rather than math, to avoid favorable evaluation skew.

Scaling TPT to the 70B-parameter regime to test whether the efficiency multiplier persists at scale. The paper's largest experiment uses 8B parameters and 100B tokens. Frontier models operate at 70B–405B parameters and 10T–15T tokens. There are plausible reasons the 3× multiplier could shrink at larger scales: larger models may already have stronger inherent reasoning capabilities, reducing the marginal benefit of decomposed training signals; the thinking generator (itself a 7B model) may produce trajectories that are too simplistic to aid a 70B student; and the generation cost for 10T+ tokens of augmented data may become prohibitive. A scaling study that trains 1.5B, 3B, 7B, 13B, and 70B models (all at the same Chinchilla-optimal token-to-parameter ratio) with and without TPT would reveal whether the efficiency multiplier is constant across scale or decays. This experiment would be expensive—training a 70B model on hundreds of billions of tokens requires substantial compute—but it directly addresses the most important unanswered question for practitioners: does TPT matter at the scales where training costs dominate total expenditure? A negative result (the multiplier shrinks to near 1.0× at 70B) would not invalidate TPT for smaller-scale deployments but would significantly limit its relevance to frontier model development.

Understanding whether TPT-trained models learn genuine reasoning or mimic the generator's reasoning style. This is a stress-test designed to detect a potential failure mode: the student model may learn to produce fluent, plausible-sounding chain-of-thought reasoning that mimics the training trajectories' surface patterns without developing robust underlying reasoning capabilities. The experiment would compare TPT-trained models against baseline models on two types of evaluation: (1) standard benchmarks where the test distribution matches the training distribution (e.g., MATH, GSM8k) and (2) out-of-distribution reasoning probes designed to require genuine understanding rather than pattern matching. For example, systematically perturbed math problems—where the problem statement is modified in ways that change the correct answer but preserve surface structure—would test whether the model is solving problems or matching patterns. If TPT models show a larger performance drop than vanilla models on perturbed problems (despite outperforming on standard benchmarks), it would indicate that some of the gains come from learning the generator's reasoning aesthetic rather than the underlying reasoning principles. The paper provides no test of this kind, and it represents the most important negative result that could refine our understanding of what TPT actually teaches.

Combining TPT with test-time compute scaling. The paper draws an explicit analogy between training-time compute allocation (through variable thinking trajectory lengths) and test-time compute scaling (longer chain-of-thought at inference), but never tests whether these two mechanisms are additive, redundant, or synergistic. A natural experiment would take a TPT-trained model and evaluate it under varying test-time compute budgets—comparing performance improvements from additional inference-time chain-of-thought against a vanilla-trained model given the same test-time compute. The hypothesis is that TPT pre-training makes the model more capable of benefiting from test-time compute, because it has learned to reason effectively from the thinking trajectories in its training data. If TPT models show steeper test-time scaling curves than vanilla models (i.e., they improve more per additional inference token), it would mean TPT and test-time compute are complementary—TPT builds reasoning foundations during training that test-time compute can more effectively exploit. If the curves are parallel (equal slope but shifted upward), it means the mechanisms are independent. If the curves converge (smaller slope for TPT), it means TPT and test-time compute are partially redundant. This experiment directly connects the paper's conceptual framework to the active test-time compute literature and would help practitioners decide whether to invest in TPT, test-time scaling, or both.

Practical Applications and Downstream Use Cases

Cost-efficient pre-training of domain-specialized reasoning models. Organizations building models for mathematical reasoning, code generation, or scientific Q&A—where the problem distribution skews toward content requiring multi-step logical deduction—can use TPT to achieve competitive performance with substantially smaller training budgets. The paper's results provide a concrete benchmark: a TPT-8B model trained on 100B tokens achieves GSM8k accuracy of 50.1% and MATH accuracy of 21.8% (Table 1), approaching LLaMA-3.1-8B's 47.0% and 14.1% despite the latter being trained on 15T tokens (150× more data). After SFT on a modest 350k-example dataset, TPT-8B achieves 35.2% on AIME24 and 82.4% on MATH-500 (Table 2), outperforming LLaMA-3.1-8B-Instruct on every benchmark. For an organization with access to a cluster of a few hundred GPUs, TPT makes it feasible to train a competitive reasoning model in weeks rather than the months required for trillion-token training runs. The key practical requirement is access to (or the ability to generate) thinking trajectories for the target domain's training data—a one-time compute cost of approximately 20k A100 GPU hours per 100B training tokens (Appendix A.1), which amortizes across multiple training runs.

Mid-training on existing checkpoints to add reasoning capabilities without full retraining. Organizations with deployed models (e.g., Qwen2.5-7B or LLaMA-3.2-3B) can apply TPT as a mid-training phase to substantially improve mathematical and general reasoning performance without training from scratch. Table 3 provides the quantitative case: TPT mid-training on LLaMA-3.2-3B for 100B tokens followed by SFT on public data improves AIME24 from 5.8% to 18.6% (a 3.2× increase), MATH-500 from 59.8% to 75.5%, and JEEBench from 26.6% to 42.4%. For Qwen2.5-7B, TPT mid-training pushes AIME24 from 50.5% to 57.5%, achieving performance competitive with or exceeding DeepSeek-R1-Distill-Qwen-7B (trained on more than 2× the SFT data) on several benchmarks. The mid-training approach is particularly attractive because it avoids the cost of pre-training from scratch—the 100B-token mid-training run takes approximately one week on 32 GPUs (Appendix A.1)—while providing substantial capability improvements. The only prerequisite is a thinking-augmented version of the mid-training corpus, which can be generated once and reused across multiple model sizes and families.

Data generation for self-improvement pipelines in data-constrained domains. For domains where high-quality training data is scarce (scientific literature in niche fields, proprietary technical documentation, low-resource languages), TPT provides a method to amplify the utility of limited data without requiring human annotation or synthetic data generation from scratch. The constrained-data experiment (Section 3.2) demonstrates the principle: with only 10B tokens of raw documents, TPT achieves 32.6% average base model performance vs. 16.6% for vanilla training with 4 epochs (Table 7)—nearly double the performance from the same underlying documents. The practical workflow would be: collect whatever domain-specific documents are available, generate thinking trajectories for them using an off-the-shelf reasoning model, and train on the augmented corpus. This is substantially simpler than existing approaches that require training domain-specific quality classifiers, designing custom data generation prompts, or implementing iterative data bootstrapping (as in BoLT's EM algorithm). The paper's finding in Table 4 that a 1.5B-parameter generator can outperform a 7B one further reduces the barrier—practitioners can use a relatively small, cheap-to-run model for trajectory generation and still achieve strong downstream results.

Amplifying the benefits of limited SFT budgets. The paper demonstrates that thinking-augmented pre-training or mid-training substantially improves models' ability to benefit from small SFT datasets. Figure 5 shows that direct SFT on the 350k-example Mixture-of-Thoughts dataset without mid-training yields near-zero AIME24 performance for LLaMA-3B, while the same model with 100B tokens of TPT mid-training reaches ~15% after SFT. Table 3 extends this across model sizes: TPT models fine-tuned on only 350k examples approach or exceed the performance of models trained on 800k-example private datasets (e.g., TPT-Qwen2.5-7B achieves 57.5% on AIME24 vs. 53.2% for DeepSeek-R1-Distill-Qwen-7B). For practitioners with access to limited SFT data—which is expensive to produce, requiring human annotation or distillation from frontier models—TPT offers a way to "pre-install" reasoning capabilities during the data-intensive pre-training or mid-training phase, so that even modest SFT datasets can effectively elicit strong downstream performance. This is particularly valuable for smaller organizations that cannot afford to generate or license large proprietary SFT datasets.