To Code, or Not To Code? Exploring Impact of Code in Pre-training

🎯 Pitch

Replacing just 10% of web-scraped code with high-quality synthetic code boosts natural language reasoning by 9% and code performance by 45%, revealing that code quality—not quantity—supercharges an LLM's general abilities.

1. Executive Summary

This paper systematically investigates the impact of including code data in pre-training on downstream performance beyond code generation, conducting extensive controlled experiments with models ranging from 470M to 2.8B parameters trained on SlimPajama text and Stack-based code datasets and evaluated on natural language reasoning, world knowledge, code generation, and LLM-as-a-judge win-rates. The central findings identify several critical mechanisms: initialization with code pre-trained models (training a text model starting from a code-LM checkpoint rather than random weights), code quality and properties (replacing small fractions of web-code with synthetically generated, formally verified Python code), and code in pre-training cooldown (up-weighting high-quality code during the final annealing phase of training) — establishing that code serves as a critical building block for generalization far beyond coding tasks, with the best variant achieving relative increases of 8.2% in natural language reasoning, 4.2% in world knowledge, a 6.6% improvement in generative win-rates, and a 12× boost in code performance compared to text-only pre-training. The paper also establishes that high-quality synthetic code has an outsized impact, with a mere 10% replacement of web-code yielding 9% and 44.9% relative improvements in NL reasoning and code performance respectively, and that these benefits transfer across both code-only pre-training and continual pre-training settings — but only when code quality is sufficiently high and code is present across all phases of training.

2. Context and Motivation

The Core Problem: We Include Code in Pre-training Without Knowing Why It Works

The fundamental question this paper tackles is deceptively simple: if you include code data in an LLM's pre-training mixture, does it improve the model's performance on non-code tasks, and if so, why? This matters because the practice has become nearly universal among state-of-the-art models — yet it rests on little more than practitioner intuition and anecdotal evidence. The paper opens by observing that code data differs significantly from high-quality web text in its structural and textual characteristics, making its inclusion in general-purpose language model training non-obvious. If code is just a specialized domain, adding it might dilute the natural language signal and hurt general performance. The fact that it doesn't — that models like PaLM, Gopher, and Bloom included code despite not being primarily intended for code generation — demands explanation.

The paper frames this gap squarely in the introduction:

"While there has been consensus anecdotally among practitioners that code data plays a vital role in LLMs' performance, there has been only limited work analyzing the precise impact of code on non-code tasks."

This is a gap in understanding, not just in practice. The field has been operating on a heuristic — "add code, performance improves" — without systematic knowledge of which code, how much code, when in training to add it, or what mechanisms drive the observed improvements. The paper positions itself as the first exhaustive study to answer these questions through controlled pre-training experiments at scale.

Why This Problem Matters: Practical Resource Allocation and Scientific Understanding

The significance of this question operates on two levels.

Practical significance: pre-training is enormously expensive. The proportion of code in pre-training mixtures has been steadily increasing — from Llama 2's 4.5% to Llama 3's 17% — without rigorous justification for why quadrupling the code fraction is beneficial. Every percentage point of the pre-training mixture represents a substantial compute investment. If code data drives disproportionate gains in general reasoning, then investing in higher-quality code curation and expanding code proportions may be one of the most cost-effective ways to improve model performance. Conversely, if code only helps code generation and slightly degrades other capabilities, the trend toward ever-increasing code fractions would be misguided. The paper directly investigates this trade-off, finding that code proportions exhibit non-linear effects: reasoning peaks at 25% code, world knowledge monotonically degrades with increasing code, and code performance increases linearly (Figure 4). These findings have immediate implications for how organizations allocate their pre-training data budgets.

Scientific significance: code as a window into generalization. Code possesses properties that natural language text typically lacks: formal syntax, compositional structure, long-range dependencies, and explicit algorithmic reasoning. Understanding how and why exposure to these properties during pre-training transfers to improved performance on natural language reasoning, world knowledge, and open-ended generation would reveal fundamental insights about what representations are most useful for general intelligence. If code improves reasoning because it teaches the model to follow multi-step logical procedures, that suggests a specific mechanism. If it improves world knowledge because it provides structured context for factual information, that suggests a different one. The paper doesn't fully resolve these mechanistic questions — it focuses on establishing the empirical phenomena — but its ablation design lays the groundwork for such investigation.

Real-world deployment stakes. The paper's investigation of initialization from code pre-trained models (Section 3.1) has direct implications for how organizations structure their training pipelines. If starting from a code-LM checkpoint and continuing with text yields better general performance than training from scratch on a balanced mixture, this changes the economics of model development — organizations might train specialized code models as "pre-training backbones" even when their ultimate target is general-purpose language understanding.

Where Prior Approaches Fall Short

The paper identifies several limitations in existing work, which collectively leave the core question unresolved.

1. Prior studies are narrow in scope, typically isolating a single mechanism or task.

Muennighoff et al. (2023a) demonstrated that Python code data improves pre-training performance, but their work focused on a low-resource, data-constrained regime — evaluating perplexity improvements when total pre-training data is limited. This is a fundamentally different setting from the saturated data regime where state-of-the-art models operate. Additionally, their evaluation was limited to perplexity, which provides no signal about downstream task performance on reasoning, knowledge, or generation quality. The paper's multi-task evaluation suite (Table 1) addresses this gap directly.

2. Existing work conflates code's role across training stages.

Zhang et al. (2024) investigated code's impact on LLM reasoning but restricted their study to the supervised fine-tuning (SFT) stage — that is, instruction-tuning rather than pre-training. This is a critical distinction: SFT operates on a model whose representations are already formed during pre-training. Finding that code in SFT helps reasoning tells us about the model's ability to leverage code-like structure during instruction-following, but says nothing about whether code during pre-training fundamentally shapes the model's internal representations. The paper explicitly separates these phases, studying code in initial pre-training, continual pre-training, and cooldown independently (Sections 3.1, 3.4, 3.5).

3. Practitioner reports lack controlled ablations.

The Llama 3 technical report (Dubey et al., 2024) discusses the importance of data mixture ratios for general knowledge, mathematical reasoning, code, and multilingual performance, but does not exhaustively examine code's distinct impact. Similarly, the DeepSeek-Coder-V2 report (Zhu et al., 2024) reports performance on general natural language benchmarks but compares chat and instruct model variants without systematically ablating different phases of pre-training or properties of code data. These reports demonstrate that code-adjacent training works, but not why it works or which aspects of code data matter most.

4. Specialized findings haven't been integrated into a unified picture.

Several prior works have shown specific benefits of code: entity tracking (Kim et al., 2024), commonsense reasoning (Madaan et al., 2022b), and mathematical reasoning (Liang et al., 2022; Madaan et al., 2022a; Gao et al., 2023). These are valuable but piecemeal — each isolates a single capability improvement without providing a framework for understanding code's general impact. The paper's contribution is not identifying any single benefit (many of which were previously known) but rather establishing that code's benefits span a much broader range of capabilities than previously documented, and that these benefits are modulated by specific controllable factors: code proportion, code quality, training stage, and model initialization.

5. No study has systematically varied code properties and quality.

A crucial gap the paper identifies: even when prior work included code in pre-training, it treated code as a monolithic category. The paper's experiments with markup-style languages, code-adjacent data (GitHub commits, Jupyter notebooks, StackExchange threads), and synthetically generated high-quality code (Section 3.4) represent the first systematic investigation of which properties of code data drive generalization improvements. The finding that a small proportion of high-quality synthetic code (formally verified Python programs) yields outsized gains — 9% relative improvement in NL reasoning and 44.9% in code performance despite comprising only 10% of code tokens — fundamentally reframes the problem from "should we include code?" to "what kind of code should we include, and how do we produce more of it?"

How This Paper Positions Itself

The paper explicitly positions itself as the first systematic study of code's impact in pre-training on non-code tasks (Section 4, final paragraph):

"To the best of our knowledge, this work is the first study that presents a thorough investigation of the impact of code in pre-training on non-code tasks. Our experiment spans several axes, with costly ablations at scale including model initialization strategies, different proportions and properties of code data, and model scales."

This claim rests on the comprehensiveness of the experimental design relative to prior work (Figure 1), which simultaneously varies five key dimensions: (1) code proportion in pre-training (0% to 100%), (2) code quality and properties (web-code, markup, code-adjacent, synthetic), (3) model initialization (random, balanced, code-LM), (4) model scale (470M to 2.8B parameters), and (5) training stage at which code is introduced (initial pre-training, continual pre-training, cooldown).

The paper's contribution is empirical rather than methodological — it doesn't propose a new training algorithm or architecture. Instead, it provides the controlled evidence base that the field has been missing, with the explicit goal of converting practitioner intuition into actionable, quantified findings. The total of 64 pre-trained models, each requiring thousands of TPU-chip-hours (4,736 for 470M, 13,824 for 2.8B), represents a significant computational investment that the authors frame as necessary for rigor: "a costly endeavor to perform these ablations in a rigorous way."

The paper also positions itself relative to the broader literature on data composition in pre-training. Citing work on data filtering (Raffel et al., 2020; Rae et al., 2022b; Penedo et al., 2023), de-duplication (Zhang et al., 2022), data pruning (Lozhkov et al., 2024; Marion et al., 2023), and synthetic data (Gunasekar et al., 2023; Li et al., 2023b), the paper notes that these are "broader treatments that are focused on characteristics of training data as a whole" — whereas their work focuses narrowly and deeply on code as a specific data category with unique properties. This specialization enables granular ablations that broader data composition studies cannot accommodate.

A final aspect of the paper's positioning is its dual evaluation philosophy. Recognizing the well-documented tension between academic benchmark performance and open-ended generation quality (Üstün et al., 2024; Ouyang et al., 2022), the paper evaluates both discriminative accuracy on structured benchmarks (NL reasoning, world knowledge, code) and generative quality via LLM-as-a-judge win-rates on the Dolly-200 English dataset. This dual lens is important because it addresses the possibility that code data might improve benchmark scores while degrading the fluid, helpful, and instruction-following qualities that matter for user-facing applications. The paper's finding that code improves both discriminative and generative performance — with cooldown models achieving 52.3% win-rates against no-cooldown baselines and code-initialized models beating text-only by 6.6% in win-loss difference — strengthens the case that code's benefits are genuine and not an artifact of benchmark overfitting.

3. Technical Approach

3.1 Reader Orientation

This paper is a controlled empirical study — it does not build a new system or model, but rather constructs a systematic framework for understanding how code data in pre-training affects downstream performance. The core mechanism being studied is the pre-training data pipeline: a sequence of data selection, mixing, and training stages where code data is introduced in varying proportions, qualities, and phases to isolate its causal impact. The "system" under investigation is the end-to-end process of taking raw text and code corpora, filtering and mixing them according to specific recipes, training transformer models from scratch or from existing checkpoints, and evaluating the resulting models across a broad suite of natural language reasoning, world knowledge, code generation, and open-ended generation tasks to measure the transfer effects of code exposure.

The problem this paper solves is ignorance about a widely-adopted practice: practitioners have been adding code to pre-training mixtures for years without rigorous, controlled evidence about which code, how much code, when to add it, or what specific benefits it provides beyond code generation. The shape of the solution is an exhaustive ablation grid spanning five dimensions — code proportion, code quality/properties, model initialization, model scale, and training stage — evaluated on multiple capability categories, enabling precise quantification of code's marginal contribution to each type of downstream performance and revealing previously undocumented transfer effects (e.g., code improving open-ended generation win-rates by 6.6%).

3.2 Big-Picture Architecture (Diagram in Words)

The experimental framework consists of five major component groups:

Data Sources and Filtering Pipeline — Takes raw web-code (The Stack), web-text (SlimPajama with code removed), and specialized code datasets (markup, code-adjacent, synthetic) and produces tokenized, quality-filtered corpora at specific mixing ratios. This is the input layer that determines what the model sees during training.
Model Initialization Variants — Defines the starting point for training: random initialization (Glorot-normal), balanced pre-training (50% text, 50% code), or code-only pre-training (80% code, 20% markup). These variants test whether code exposure during early training creates representations that benefit later text training.
Multi-Phase Training Scheduler — Orchestrates training across up to three phases: initial pre-training (from scratch), continual pre-training (from a code-LM checkpoint with text data), and cooldown (up-weighting high-quality data with linear learning rate annealing). Each phase has independent control over the data mixture, token budget, and learning rate schedule, enabling isolation of when code matters.
Model Architecture (Fixed) — A decoder-only autoregressive transformer (470M or 2.8B parameters) with parallel attention layers, SwiGLU activation, no biases in dense layers, and a BPE tokenizer (vocabulary size 256,000). AdamW optimizer with cosine learning rate scheduling (warmup 1325 steps), batch size 512, maximum sequence length 8192. This is the compute layer held constant across experiments to isolate data effects.
Multi-Category Evaluation Suite — Assesses the trained model on four axes: (a) natural language reasoning (11 benchmarks covering QA, NLI, sentence completion, coreference resolution), (b) world knowledge (2 QA benchmarks measuring memorization and retrieval), (c) code generation (HumanEval and MBPP pass@1), and (d) generative quality (LLM-as-a-judge win-rates on Dolly-200 English). This is the measurement layer that determines whether code data transfers to non-code capabilities.

Information flow: Raw corpora → quality filtering and deduplication → tokenization → mixing according to experiment-specific ratios → token-budget-limited training across phases → evaluation on held-out benchmarks → comparison across variants to isolate the marginal contribution of each code-related factor.

3.3 Roadmap for the Deep Dive

First, the pre-training data pipeline (Section 2.1 in the paper's numbering; here, Section 3.4.1): how text and code datasets are sourced, filtered, and mixed, because the composition of the data mixture is the primary independent variable in every experiment.
Second, model architecture and training infrastructure (Section 2.3 in the paper; Section 3.4.2 below): the fixed model configuration and compute environment, because understanding the scale and cost of the experiments is essential for interpreting the practical significance of the findings.
Third, the experimental variant taxonomy (Sections 3.1–3.5 spanning multiple mechanisms; Section 3.4.3 below): the systematic enumeration of initialization strategies (random, balanced, code-LM), code proportion ablations (0% to 100%), code quality and property interventions (markup, code-adjacent, synthetic), cooldown configurations, and scale experiments, because these define the "knobs" being turned and what each comparison isolates.
Fourth, the evaluation protocol (Section 2.2; Section 3.4.4 below): the composition of the four evaluation categories, the specific benchmarks and metrics used, the LLM-as-a-judge setup, and the rationale for multi-category assessment, because the paper's contribution hinges on demonstrating that code improves performance across the board — not just on code tasks — and the evaluation design must support that claim.
Fifth, the FLOPs accounting and compute budget (Section 2.3; Section 3.4.5 below): the concrete numbers for TPU-chip-hours per experiment, the total number of models trained (64), and the training hyperparameters, because these establish the credibility of the ablations — this is an enormous computational effort that enables statistically meaningful comparisons.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical ablation study whose core idea is that code data in pre-training improves general downstream performance — not just code generation — through mechanisms that depend on code proportion, code quality, model initialization, and training stage, and that these benefits can be isolated and quantified through controlled experiments where all other factors are held constant.

3.4.1 Pre-Training Data Pipeline: Sources, Filtering, and Mixing

The data pipeline is the most critical component because it defines what the model sees during training — every ablation in the paper is fundamentally a comparison between different data mixtures. The pipeline has three stages: source selection, quality filtering, and proportional mixing.

Text data source: SlimPajama (de-coded). The paper uses SlimPajama (Soboleva et al., 2023), a deduplicated, quality-filtered, multi-corpus open-source dataset based on RedPajama-1.2T. SlimPajama originally spans 627B tokens across CommonCrawl, C4, GitHub, Books, ArXiv, Wikipedia, and StackExchange. To create a text-only source, the paper explicitly filters out all documents from GitHub and StackExchange — the two code and code-adjacent sources. This is a critical design choice: if SlimPajama were used as-is, the text-only baseline would be contaminated with code data, making it impossible to isolate code's marginal contribution. After removing code sources, the text corpus contains 503B tokens.

Code data sources: four categories with different properties. The paper uses multiple code datasets to investigate which properties of code data drive improvements:

Web-based code data (The Stack): The primary code source is The Stack (Kocetkov et al., 2022), a dataset of permissively licensed code scraped from GitHub, originally used to train StarCoder. The paper applies additional quality filters beyond those in the original Stack release: removing documents with more than 1,000 float numbers (a heuristic for auto-generated or minified files), documents containing instances of the string "0x" (indicating hex dumps or binary data), documents that are lists of top-level domains, and documents with "generated by" in the first 400 characters (catching boilerplate or auto-generated code). The paper further restricts to the top 25 programming languages by document count (Table 3 in the appendix). After all filtering, the web-code-only subset is 139B tokens.
Markup-style language data: The paper separately processes markup-style languages — Markdown, HTML, CSS, SCSS, YAML, JSON, reStructuredText, TOML, RMarkdown, and Sass — from The Stack. These are structurally different from programming languages: they are declarative rather than procedural, and they define presentation or data serialization rather than algorithms. The paper treats these as a distinct code category to test whether code's benefits come from formal syntax (which markup shares) or from algorithmic structure (which markup lacks). After filtering, the markup subset is 180B tokens — actually larger than the programming-language subset.
Synthetic code data (high-quality): To ablate code quality, the paper uses a proprietary synthetically generated code dataset consisting of Python programming problems that have been formally verified. "Formally verified" means that each program has been mechanically checked for correctness against a specification, guaranteeing that the code is free of logical errors. The synthetic dataset is only 3.2B tokens — less than 2.3% of the web-code dataset — but it represents the highest-quality code in the study: guaranteed-correct, well-structured Python. This small size relative to web-code is important for interpreting the results: any improvements from synthetic code must come from quality rather than quantity.
Code-adjacent data: To test whether code's benefits come from the surrounding ecosystem of software engineering artifacts (not just code files), the paper includes auxiliary datasets: GitHub commits (5% of code-adjacent tokens), Jupyter notebooks (5%), and StackExchange threads (5%). GitHub commits capture the process of code development (diffs, commit messages), Jupyter notebooks capture interleaved code and natural language explanations, and StackExchange captures Q&A discussions about code. Together, the code-adjacent dataset is 21.4B tokens — substantially larger than the synthetic dataset but representing a qualitatively different type of "code-like" data. The paper replaces 15% of code tokens with code-adjacent data in the relevant ablation (Section 3.4 in the paper).

Mixing ratios and token budgets. The pre-training data mixture is defined by the proportion of code tokens relative to text tokens. In code proportion experiments (Section 3.3 of the paper), six mixtures are tested: 0% code (text-only baseline), 25% code, 50% code, 75% code, 90% code, and 100% code. In initialization experiments (Table 2), specific mixtures are used: (a) text-only pre-training (400B text tokens), (b) balanced pre-training (200B text + 200B code tokens), (c) code-LM initialization (200B code tokens, then 180B text + 20B code in continual pre-training), and (d) balanced-initialized text training (100B text + 100B code in initial pre-training, then 180B text + 20B code in continual pre-training).

Cooldown data mixture. The cooldown phase (final 40B tokens, 10% of the token budget for the models studied) uses a different mixture: high-quality text, math, code, and instruct-style datasets. In cooldown with code, 20% of the cooldown tokens are code. In cooldown without code, code is entirely absent from the cooldown mixture. The cooldown mixture is up-weighted relative to the main pre-training mixture — higher-quality sources receive higher sampling probability — which makes the cooldown phase a test of whether concentrated exposure to high-quality code at the end of training provides benefits beyond the distributed exposure during main pre-training.

Pre-training cooldown datasets (Section 2.1). The paper explicitly states: "We choose a pre-training cooldown mixture comprising high-quality text, math, code, and instruct-style text datasets" but does not enumerate the specific datasets beyond mentioning that instruction-style data includes Dolly v2. The 40B token budget for cooldown is held constant across all cooldown ablations to ensure fair comparison.

3.4.2 Model Architecture and Training Infrastructure

Architecture (fixed across all experiments). All models use a decoder-only autoregressive transformer architecture (Radford et al., 2019) with the following specifications:

Parallel attention layers (Chowdhery et al., 2022; Wang & Komatsuzaki, 2021): instead of computing attention and feed-forward sequentially within each transformer block, these are computed in parallel and then combined. This is a inference-speed optimization that does not affect the learning dynamics being studied.
SwiGLU activation (Shazeer, 2020): a gated linear unit variant that replaces the standard ReLU or GELU activation with Swish(xW₁) ⊗ (xW₂), where ⊗ is element-wise multiplication. SwiGLU has been shown to outperform standard activations in transformer models.
No biases in dense layers: removing bias terms from linear transformations, a common practice in modern LLM architectures that slightly reduces parameter count without affecting expressivity.
BPE tokenizer with vocabulary size 256,000: byte-pair encoding tokenization with a larger-than-typical vocabulary (compared to, e.g., GPT-2's 50,257 or Llama's 32,000). A larger vocabulary can represent more text with fewer tokens, increasing throughput at the cost of embedding table size.
Two model scales: 470M parameters (smaller scale for rapid experimentation) and 2.8B parameters (larger scale for validating that trends transfer). At the 470M scale, the paper explicitly notes that model capabilities are limited, so they "only compare benchmarks for which all models achieve performance above random" — this is a methodological guardrail against measuring noise.

Training hyperparameters (fixed across all experiments unless otherwise noted).

Optimizer: AdamW (Loshchilov & Hutter, 2019) — the decoupled weight decay variant of Adam that separates L2 regularization from the adaptive learning rate computation. The paper does not specify β₁, β₂, or ε, but these are presumably standard values (0.9, 0.999, 1e-8) since no deviation is mentioned.
Batch size: 512 sequences per batch. With a maximum sequence length of 8,192, each batch contains up to 512 × 8,192 = 4,194,304 tokens.
Learning rate scheduler: cosine decay with a warmup of 1,325 steps. Cosine scheduling reduces the learning rate following η_t = η_min + 0.5(η_max − η_min)(1 + cos(π t / T)), ensuring smooth decay to a small terminal value. The warmup period linearly increases the learning rate from 0 to the peak value over 1,325 steps to avoid destructive gradient updates early in training.
Maximum sequence length: 8,192 tokens — the standard context window for modern pre-training runs.
Cooldown learning rate schedule: different from the main training. The paper states: "We change the learning rate schedule from cosine-based to linear annealing with a final learning rate of 1e−6." Linear annealing means the learning rate decreases at a constant rate rather than following the cosine curve: η_t = η_start − (η_start − 1e−6) × (t / T_cooldown). The final learning rate of 1e−6 (1 × 10⁻⁶) ensures the model continues to learn but in very small steps during the final phase.

Infrastructure and computational cost. All models are trained on TPU v5e chips (Jouppi et al., 2017) using the FAX framework (Yoo et al., 2022). The paper reports per-experiment compute costs, which contextualize the scale of the investigation:

470M model, 200B token pre-training: 4,736 TPU-chip-hours
470M model, 40B token cooldown: 1,024 TPU-chip-hours
2.8B model, 200B token pre-training: 13,824 TPU-chip-hours
Total models trained: 64 independent pre-training runs

These numbers represent a substantial engineering investment — the 2.8B model pre-training alone requires approximately 576 TPU-chip-days (13,824 chip-hours ÷ 24 hours). The total compute across all experiments, while not explicitly stated, can be roughly estimated at roughly 500,000–700,000 TPU-chip-hours given 64 models at varying scales. This scale is necessary because each ablation must be independently trained from scratch (or from a specific checkpoint) to avoid confounding: if a single model were used and the data mixture were changed mid-training, it would be impossible to isolate whether observed effects came from the new data or from the interaction between old and new training.

Why these choices? The architectural choices (parallel attention, SwiGLU, no biases, BPE tokenizer) are standard in modern LLM training and are not the focus of the investigation — they are held constant so that performance differences can be attributed to data composition. The two-scale design (470M and 2.8B) enables the paper to test whether findings generalize across an order-of-magnitude difference in model size, addressing the concern that data mixture effects might be scale-dependent. The cooldown with linear annealing reflects emerging best practices (Team et al., 2023; Parmar et al., 2024) where the final phase of training uses higher-quality data and a different schedule to "finish" the model's learning.

3.4.3 Experimental Variant Taxonomy: The Five Dimensions of Ablation

The paper's experimental design can be understood as a taxonomy of five orthogonal dimensions, each of which answers a specific question about code's role in pre-training. The "best variant" path — balanced → text with cooldown including code — represents the combination of optimal settings across these dimensions.

Dimension 1: Model initialization (Section 3.1 in the paper). This dimension asks: does starting from a model already exposed to code (a code-LM checkpoint) improve subsequent text training compared to starting from scratch?

Four initialization strategies are compared, with token budgets specified in Table 2:

Text-only (text-only baseline): Random initialization (Glorot-normal), trained on text-only data for 400B tokens total. This represents the counterfactual: what happens if no code is ever seen? The 400B budget is fixed to match the total token exposure of other variants.
Balanced-only: Random initialization, trained on 200B text + 200B code tokens for 400B tokens total. This tests whether including code in the initial training improves performance without any later specialization.
Balanced → Text: Initialized from a balanced pre-training checkpoint (100B text + 100B code tokens), then continually pre-trained on text data (180B text + 20B code tokens) — note the small percentage of code (10%) in the continual pre-training phase to "avoid a full distribution shift and maintain the benefits of code." Total: 200B initial + 200B continued = 400B tokens. This tests whether code exposure during early training creates representations that persist when the model later focuses on text.
Code → Text: Initialized from a code-only pre-training checkpoint (200B code tokens, with the code dataset being 80% web-code + 20% markup), then continually pre-trained on text (180B text + 20B code tokens). This is the most extreme initialization test: if the model starts from representations optimized for code, can it adapt to text? Total token budget: 200B initial + 200B continued = 400B tokens.

A critical design decision: the paper uses 10% code in the continual pre-training mixture for both balanced → text and code → text variants. The stated rationale is to "avoid a full distribution shift" — if the model were suddenly trained on 100% text after being initialized from a code-heavy checkpoint, catastrophic forgetting might erase the code-derived representations before they could transfer to text tasks. The small code percentage acts as a "reminder" that maintains code-derived representations during text-focused training. This design choice reflects a hypothesis that code's benefits are not just from initial exposure but require sustained presence — a hypothesis partially confirmed by the finding that best results require code in all training phases.

Dimension 2: Code proportion (Section 3.3). This dimension asks: what is the optimal fraction of code in the pre-training mixture for maximizing non-code performance?

Six models are trained from scratch (random initialization) for 200B tokens each, with code proportions of 0%, 25%, 50%, 75%, 90%, and 100%. The remaining proportion in each case is filled with text data from the SlimPajama (de-coded) corpus. Each model is trained independently — they are not checkpoints from a single run — to ensure that the learning dynamics at each code proportion are not confounded by the training trajectory.

The 200B token budget (rather than the full 400B budget used in initialization experiments) reflects the focus here on isolating code proportion effects during the main pre-training phase, without the confounding of continual pre-training or cooldown. The 0% code model serves as the text-only baseline for this dimension.

Dimension 3: Code quality and properties (Section 3.4). This dimension asks: do different types of code data produce different transfer effects to non-code tasks?

The experiments proceed in two stages:

Stage A: Code-only pre-training (Figure 5a). Four models are trained from scratch on 200B tokens of code-only data:

code (baseline): 100% web-based code from The Stack (139B tokens resampled as needed).
code+markup: 80% web-code + 20% markup-style data (CSS, HTML, Markdown, etc.).
code+adjacent: 85% web-code + 15% code-adjacent data (5% GitHub commits, 5% StackExchange, 5% Jupyter notebooks).
code+synth: 90% web-code + 10% synthetically generated, formally verified Python code.

All variants use 200B tokens total. The code-only pre-training setting is deliberately artificial — no text data is included — because the goal is to isolate which code properties create the best foundation for later text training. Effects observed here are upper bounds on what code properties can contribute; in a mixed text-code setting, these effects would be diluted but should still manifest as directional improvements.

Stage B: Continual pre-training (Figure 5b). Based on Stage A findings, the best code property (synthetic data) is incorporated into the continual pre-training variant:

balanced → text: The standard balanced-initialized text training (100B text + 100B code initial, then 180B text + 20B code continued).
balanced+synth → text: Same as balanced → text, but the initial code portion uses 90% web-code + 10% synthetic code, and the same synthetic-inclusive code mixture is used in the continual pre-training phase.

This tests whether synthetic code's benefits in code-only pre-training transfer to the more realistic setting where text dominates the total token budget.

Why synthetically generated, formally verified code? The paper's hypothesis is that high-quality code — code that is guaranteed correct, well-structured, and free of bugs — teaches the model stronger reasoning patterns than noisy web-scraped code. Web-scraped code from GitHub contains incomplete files, buggy implementations, inconsistent style, and auto-generated boilerplate. Synthetic code generated from formal specifications is guaranteed correct by construction and tends to have clearer structure and naming. The 10% replacement is a small enough proportion that it tests whether quality matters more than quantity — the synthetic dataset is only 3.2B tokens vs. 139B tokens of web-code, yet replacing 10% of web-code with synthetic yields the largest improvements.

Why markup data and code-adjacent data? These ablate two distinct hypotheses about what makes code useful:

Markup hypothesis: Code's benefits come from formal syntax and tree-like structure. If true, markup languages (which share these properties but lack algorithms) should provide similar benefits. The paper tests this by replacing 20% of code with markup.
Ecosystem hypothesis: Code's benefits come from the rich metadata of software engineering — commit messages, documentation, Q&A discussions. If true, code-adjacent data should provide benefits even without more actual code. The paper tests this by replacing 15% of code with code-adjacent artifacts.

The results — markup helps NL reasoning but hurts code evaluation, code-adjacent helps NL reasoning but hurts code evaluation, synthetic helps both — disambiguate these hypotheses. Code's benefits to reasoning appear to come from both formal structure (markup helps) and the software engineering ecosystem (code-adjacent helps), but the strongest benefits come from algorithmic correctness (synthetic helps the most and also improves code evaluation).

Dimension 4: Cooldown configuration (Section 3.5). This dimension asks: does including code in the final, high-quality training phase provide additional benefits beyond including it in the main pre-training?

Three models are compared, all based on the balanced → text pre-trained checkpoint (the best variant from Section 3.1):

No cooldown: The model as-is after 400B tokens of initial + continual pre-training.
Cooldown without code: The model is further trained for 40B tokens on a high-quality mixture of text, math, code (absent here), and instruct-style data, with linear learning rate annealing to 1e−6. The code proportion in this cooldown is explicitly 0%.
Cooldown with code: Same cooldown protocol but with 20% code in the cooldown mixture.

The cooldown phase represents 10% of the total token budget (40B tokens out of 440B total). The paper's rationale for testing cooldown separately is that recent work (Parmar et al., 2024; Team et al., 2023) has found that up-weighting high-quality data during the final training phase disproportionately improves model quality. By including or excluding code from this phase, the paper tests whether code's benefits are primarily from the total volume of code seen (which would make cooldown code redundant if the model has already seen 200B+ code tokens) or from code's presence during the final optimization stage when the model's representations are being "finished."

The inclusion of instruct-style data (Dolly v2) in the cooldown mixture is significant because the paper evaluates generative win-rates — cooldown with instruct data directly teaches the model to follow open-ended instructions, which should improve win-rates independent of code. The finding that cooldown with code further improves win-rates beyond cooldown without code (by 4.1%) isolates code's contribution to generative quality from the instruct data's contribution.

Dimension 5: Model scale (Section 3.2). This dimension asks: do the trends observed at 470M parameters persist at larger scales?

The paper replicates three key initialization variants at 2.8B parameters:

Balanced → text: 100B text + 100B code initial, then 180B text + 20B code continued (same recipe as 470M, same total token budget).
Code → text: 200B code initial, then 180B text + 20B code continued (same recipe).
Balanced-only: 200B text + 200B code (same recipe).

The same token budget (400B total before cooldown) is used at both scales, which means the 2.8B model sees fewer tokens per parameter than the 470M model (400B / 2.8B ≈ 143 tokens/parameter vs. 400B / 0.47B ≈ 851 tokens/parameter). This is a known consideration — Chinchilla scaling laws (Hoffmann et al., 2022) suggest optimal training uses ~20 tokens per parameter, so both models are well above optimality, but the 470M model is more "over-trained" relative to its size. The paper does not address this asymmetry, which could mean the 2.8B results are closer to the model's ceiling and more sensitive to data quality differences.

The cooldown experiments are not replicated at 2.8B (likely due to compute constraints — each 2.8B run already costs ~14K TPU-chip-hours), so scale-dependence of cooldown effects remains an open question.

3.4.4 Evaluation Protocol: Multi-Category Assessment

The evaluation suite is designed to answer the paper's central question comprehensively: does code data improve performance on a large variety of downstream tasks beyond code generation? The suite is organized into four categories, each measuring a distinct capability.

Category 1: Natural Language Reasoning (11 benchmarks). This is the largest evaluation category, reflecting the paper's focus on whether code teaches general reasoning skills that transfer to text-based tasks. The constituent benchmarks and what they measure:

Question Answering (QA): BoolQ (yes/no questions based on Wikipedia passages), PiQA (physical commonsense QA about everyday objects), SciQ (science exam questions), SocialIQA (commonsense reasoning about social situations), and QUAC (conversational QA with context). These measure a range of reasoning types: factual retrieval, physical intuition, scientific knowledge, social understanding, and multi-turn reasoning.
Natural Language Inference (NLI): SuperGLUE-CB (CommitmentBank, detecting whether a clause is entailed by context) and SuperGLUE-COPA (choice of plausible alternatives, causal reasoning). These measure logical entailment — can the model determine what follows from given premises?
Sentence Completion: StoryCloze (choosing the correct ending to a short story) and HellaSwag (choosing the most plausible continuation). These measure narrative understanding and commonsense reasoning about everyday scenarios.
Coreference Resolution: Winogrande (resolving pronoun references in ambiguous sentences). This measures the model's ability to track entities across discourse — a capability that code might improve due to variable scoping and reference resolution in programming languages.
General Intelligence: ARC-Easy (grade-school level multiple-choice science questions). This measures broad, shallow reasoning across diverse domains.

All benchmarks in this category are evaluated in 0-shot setting and reported as average accuracy across the 11 tasks. The 0-shot constraint eliminates the possibility that observed improvements come from better in-context learning rather than better underlying representations.

Category 2: World Knowledge (2 benchmarks). Natural Questions Open (open-domain QA based on real Google queries) and TriviaQA (trivia questions). Both are evaluated with 0-shot exact match — the model must generate the correct answer string exactly. These measure factual recall and knowledge memorization. The paper hypothesizes that code might hurt world knowledge (since code tokens replace factual text), and the results partially confirm this: world knowledge degrades monotonically as code proportion increases (Figure 4), though initialization from code models followed by text training can recover and even improve knowledge (balanced → text beats text-only by 4.2%).

Category 3: Code Generation (2 benchmarks). HumanEval-Python (function completion from docstrings, evaluating functional correctness via unit tests) and MBPP (mostly basic Python programs, also function completion). Both use pass@1 — the model gets one attempt and the generated code is checked against test cases. This category serves both as a manipulation check (if adding code data doesn't improve code performance, the ablation is broken) and as a measure of the trade-off between code and non-code performance.

Category 4: Generative Quality (LLM-as-a-judge win-rates). Dolly-200 English, consisting of 200 hand-picked examples from Dolly-15K (Conover et al., 2023). These are open-ended prompts capturing general-purpose non-code use cases like summarization, brainstorming, explanation, and creative writing. For each prompt, two completions (from two different models) are presented to Cohere Command-R+ (the judge LLM), which selects the better one based on a detailed rubric (Appendix B of the paper).

The judge prompt instructs the evaluator to assess four criteria: (1) correct reasoning, (2) directly answering the instruction, (3) factually correct and semantically comprehensible, and (4) grammatically correct and fluent. The judge first writes a concise comparison, then outputs a preference on a new line.

Win-rate is computed as:

$\text{Win-rate(model A vs. model B)} = \frac{\text{\# completions where A preferred}}{\text{\# total comparisons}}$

where each comparison includes both orderings (presenting A first and B first) to control for position bias, and ties are excluded or split.

Why include generative quality? The paper explicitly acknowledges the tension between benchmark performance and generation quality (Üstün et al., 2024; Ouyang et al., 2022) — models optimized for multiple-choice accuracy often produce worse open-ended text. By measuring both, the paper can test whether code data improves real-world usability (as proxied by win-rates) or merely improves the narrow capabilities measured by academic benchmarks. The finding that code improves both strengthens the paper's central claim that code's benefits are genuine and broad.

Evaluation at different scales. At 470M parameters, the paper restricts comparisons to benchmarks where "all models achieve performance above random." This is an important methodological choice: on very difficult benchmarks where the 470M model is near random chance, small differences in accuracy are dominated by noise and cannot be reliably attributed to the experimental condition. The paper does not enumerate which benchmarks pass this threshold at 470M.

3.4.5 FLOPs Accounting and Total Training Budget

The paper explicitly tracks the computational cost of each experiment, both to justify the scale of the investigation and to enable reproducibility assessments.

Per-experiment costs:

470M parameter model, 200B tokens: 4,736 TPU-chip-hours using TPU v5e.
470M parameter model, 40B token cooldown: 1,024 TPU-chip-hours.
2.8B parameter model, 200B tokens: 13,824 TPU-chip-hours.

The total training effort — 64 models — represents a significant fraction of the experiments being conducted at 470M scale (lower cost per run, enabling more ablations) and a few key ablations at 2.8B scale (to verify scale invariance of findings).

Token budget accounting across phases. Table 2 provides a detailed breakdown of token counts per phase for each model variant:

Text-only: 400B text (pre-training) + 32B text / 8B code (optional cooldown)
Balanced-only: 200B text + 200B code (pre-training) + 32B text / 8B code (cooldown)
Balanced → Text: 100B text + 100B code (initial pre-training) + 180B text + 20B code (continual pre-training) + 32B text / 8B code (cooldown)
Code → Text: 0 text + 200B code (initial pre-training) + 180B text + 20B code (continual pre-training) + 32B text / 8B code (cooldown)

Note that in the cooldown phase, the "with code" variant uses 20% code (8B out of 40B tokens), while in the continual pre-training phase, code-initiated variants use 10% code (20B out of 200B tokens). These percentages reflect different design goals: 10% in continual pre-training acts as a "reminder" to prevent catastrophic forgetting, while 20% in cooldown represents up-weighting of high-quality code at the final stage.

Why this token allocation? The paper does not fully justify its token allocation choices. The use of 200B tokens for code-only pre-training but only 100B tokens for balanced pre-training (in the balanced → text variant) reflects a practical constraint: balanced → text totals 400B tokens (100B initial + 200B continued + 100B cooldown?), but the paper needs a comparable code-only initialization that doesn't exceed the total budget. The 200B token code-only initialization is matched against the 200B total text budget, making the comparison between code → text and balanced → text fair in total tokens but not in initial phase tokens.

The paper explicitly acknowledges one limitation: code proportion ablation experiments (Section 3.3) use only 200B tokens total (no continual pre-training, no cooldown), while initialization experiments use 400B tokens. This means the optimal code proportion (25%) established in the 200B-token setting may not generalize to longer training runs — the relationship between code proportion and performance might shift as the model sees more total tokens.

3.4.6 Design Decisions: Why This Approach Over Alternatives

The paper makes several non-obvious methodological choices that deserve explicit justification:

1. Why use SlimPajama with code removed rather than a naturally code-free corpus? Using SlimPajama controls for text quality and distribution: if the paper used a different text corpus for the text-only baseline, any observed differences between text-only and code-inclusive training could be attributed to corpus differences rather than code's influence. By stripping code from SlimPajama, the paper ensures the text portion is identical across all mixtures — only the presence and proportion of code varies.

2. Why use The Stack (web-scraped code) as the baseline code source rather than a curated dataset? The Stack represents the de facto standard for open-source code pre-training data. Using it makes the findings directly comparable to models trained on the same data (StarCoder, SantaCoder) and establishes a realistic baseline — if synthetic code improves over The Stack, the improvement is achievable in practice because The Stack is what practitioners actually use.

3. Why study markup languages separately from programming languages? This is one of the paper's more subtle contributions. Markup languages share syntactic properties with code (formal grammar, nesting, tags/braces) but lack algorithmic content (control flow, state mutation, function composition). By separating markup from code, the paper can test whether code's benefits come from formal syntax (markup would help) or from algorithmic reasoning (markup would not). The result that markup helps NL reasoning suggests formal syntax does transfer, but the stronger result for synthetic code suggests algorithmic correctness transfers even more.

4. Why formally verified synthetic code? Formal verification guarantees that every program in the synthetic dataset is correct — it compiles and produces the right output for all specified inputs. This is a much stronger signal than web-scraped code, which contains an unknown fraction of buggy, incomplete, or incorrect implementations. By showing that verified code produces outsized improvements, the paper implicitly argues for the importance of correctness in training data — a hypothesis with implications beyond code, suggesting that data quality filtering should prioritize logical consistency, not just surface-level features like perplexity or format.

5. Why evaluate at two scales (470M and 2.8B)? Scale interactions are a known concern in LLM research — findings that hold at small scale can reverse at large scale (e.g., the "inverse scaling" phenomenon). By replicating key initialization variants at 2.8B, the paper provides evidence that its findings are not merely artifacts of limited model capacity. The consistent trends across scales strengthen the paper's conclusions, though the caveat remains that 2.8B is still far from frontier model sizes (70B, 405B), and truly surprising scale interactions could emerge beyond the studied range.

4. Key Insights and Innovations

Innovation 1: Code as a General-Purpose Representation Builder — Not Just a Domain-Specific Skill Trainer

Prior to this work, the dominant mental model for including code in pre-training was essentially utility-driven: if you want your model to generate code, include code; if not, it's optional or potentially harmful. The field had anecdotal evidence of transfer effects — practitioners observed that code-trained models seemed "smarter" — but lacked a framework for understanding why code should improve reasoning about non-code tasks. This paper fundamentally reframes code from a domain-specific dataset to a general-purpose representation learning tool. The core intellectual move is demonstrating that code data improves performance not just on code benchmarks (expected) but on natural language reasoning (+8.2% relative), world knowledge (+4.2% relative), and open-ended generative quality (+6.6% win-rate difference), all while boosting code performance 12× versus text-only training (Table 2, balanced → text vs. text-only).

What makes this a conceptual advance rather than merely a broader evaluation is the structure of the evidence. The paper doesn't just show that code helps — it shows that code helps through specific, controllable mechanisms that operate independently of the model's ultimate task. The initialization experiments (Figure 2) are the crucial diagnostic: a model trained exclusively on code for 200B tokens (code → text), then switched to text for another 200B tokens, outperforms a model trained on text-only for the full 400B tokens on natural language reasoning by 8.8% relative. The code-only phase contains no natural language reasoning tasks — yet it builds representations that accelerate later text training beyond what additional text training alone achieves. This is the signature of a general-purpose representation learning effect, not task-specific skill transfer.

This reframing matters because it changes how organizations should think about data investment. Under the domain-specific view, code data is a cost allocated to the code generation feature. Under the general-purpose view, code data is an investment in reasoning capability that pays dividends across all tasks. The paper's finding that the optimal code proportion for NL reasoning is 25% (Figure 4, left inset) — not 50% or 100% — reinforces this: there is a sweet spot where code provides maximum general benefit before it starts crowding out the factual knowledge needed for world knowledge tasks (which degrade monotonically with increasing code proportion). If code were merely a domain-specific skill, more code would monotonically improve code and monotonically degrade everything else. The observed non-linear relationship — reasoning peaks at 25%, knowledge degrades linearly, code improves linearly — is the signature of a genuine trade-off between code's representation-building benefits and its displacement of factual content.

Comparison to prior work: Muennighoff et al. (2023a) showed code helps in data-constrained regimes, but only measured perplexity — a proxy that could reflect improved token-level prediction without genuine reasoning transfer. Madaan et al. (2022b) showed code-trained models perform better on commonsense reasoning, but didn't systematically vary code proportion or isolate training stage effects. This paper is the first to demonstrate that code's benefits (a) persist in data-rich regimes, (b) transfer to explicit reasoning, knowledge, and generation tasks simultaneously, and (c) depend on controllable factors (proportion, quality, stage) that can be optimized independently.

Significance level: This is a fundamental reframing, not an incremental finding. It converts code from a "nice to have if you want code generation" into a "critical building block for general intelligence in LLMs," with direct implications for how pre-training budgets should be allocated. The paper's own data shows the stakes: the difference between text-only and the best code-inclusive variant is 8.2% relative on NL reasoning — comparable to the gains from significant architectural improvements or much larger models.

Innovation 2: Difficulty-Agnostic Quality Matters — Synthetic Verified Code's Outsized Impact Is a Diagnostic Signal

The most striking single result in the paper is that replacing a mere 10% of web-scraped code with synthetically generated, formally verified Python code — a dataset of only 3.2B tokens, dwarfed by the 139B tokens of web-code — yields a 9% relative improvement in NL reasoning and a 44.9% relative improvement in code performance compared to training on web-code alone (Figure 5a, code+synth vs. code). This is not just a "better data helps" finding; it's a diagnostic signal about what property of code data drives generalization.

The paper systematically disambiguates competing hypotheses about what makes code useful through its multi-property ablation design (Section 3.4):

If code's benefit comes from formal syntax and tree-structured representations, then markup-style languages (CSS, HTML, Markdown) — which share these properties but lack algorithms — should produce similar gains. The result: markup improves NL reasoning by 2.8% relative but degrades code performance by 15.7% relative (Figure 5a). Formal syntax helps reasoning but isn't sufficient — and it actively hurts code generation, likely because the model learns to treat declarative markup patterns as valid programming.
If code's benefit comes from the software engineering ecosystem (documentation, commit messages, Q&A discussions), then code-adjacent data should produce similar gains. The result: code-adjacent data improves NL reasoning by 6.3% relative but degrades code performance by 9.4% relative. The ecosystem matters for reasoning but dilutes code-specific skills.
If code's benefit comes from algorithmic correctness and structured problem-solving, then formally verified, guaranteed-correct code should produce outsized gains despite small volume. The result: synthetic verified code improves both NL reasoning (+9.0%) and code (+44.9%) simultaneously. This is the only intervention that improves both categories, and the code improvement is 2.9× larger than the next-best intervention (markup, which degraded code performance).

The intellectual contribution here is the identification of correctness as a critical quality dimension for pre-training data, separate from volume, diversity, or surface-level formatting. Web-scraped code contains an unknown but likely substantial fraction of buggy, incomplete, or unidiomatic code that may teach the model incorrect reasoning patterns. Verified synthetic code eliminates this noise while preserving the algorithmic structure. The outsized impact despite tiny volume (3.2B tokens = 2.3% of the total code corpus) is the empirical signature of a quality bottleneck: the model's reasoning capabilities are limited not by how much code it sees, but by how much correct code it sees. This has immediate implications for data curation strategies — it suggests that investing in high-quality code generation (via formal verification, expert annotation, or verified execution) may yield better returns than expanding web scraping, even at substantially higher per-token cost.

Transfer of benefits across training stages strengthens this interpretation. The balanced+synth → text variant — which includes synthetic code in both initial pre-training and continual pre-training phases — achieves 2% and 35% relative improvements over the same recipe without synthetic code in NL reasoning and code respectively (Figure 5b). This confirms that synthetic code's benefits are not merely a curiosity of code-only pre-training but persist when code is a minority (10%) component of a text-dominated mixture.

Comparison to prior work: Prior work on synthetic data for LLMs (Gunasekar et al., 2023; Li et al., 2023b) focused on using synthetic data to replace natural language data, demonstrating that models trained on "textbook-quality" synthetic text can match or exceed web-trained models. This paper provides a complementary finding: even when synthetic data is a small fraction of a web-dominated dataset, its quality has disproportionate impact. It also provides a specific mechanism: correctness matters more than volume. Prior code pre-training work treated code as monolithic — The Stack, GitHub scrapes — without investigating whether verified code produces qualitatively different representations.

Significance level: This is a fundamental empirical discovery with immediate practical implications. It identifies a new axis for data curation (correctness verification) that could drive the next generation of pre-training datasets. It also provides a concrete, testable hypothesis: the scaling of reasoning capabilities may be limited by the volume of verifiably correct structured data available for training, not by total data volume. If this hypothesis holds, it suggests that investment in formal verification and verified synthetic data generation may be one of the highest-leverage activities for improving LLM reasoning.

Innovation 3: Training Stage Determines Code's Function — Initialization, Reminder, and Finisher Are Distinct Roles

The paper's finding that code plays qualitatively different roles at different training stages — and that optimal performance requires code in all stages — represents a significant conceptual refinement over the prior implicit assumption that code is a uniform ingredient to be mixed into the training data. The experimental design (Table 2) maps code's contribution across three distinct functions:

Initialization (code as representation builder): The code → text and balanced → text variants (Section 3.1) show that code exposure during early training creates representations that accelerate later text learning. A model initialized from 200B tokens of pure code and then trained on 200B tokens of text outperforms a model trained on 400B tokens of text-only on NL reasoning (+8.8%, Figure 2). This demonstrates that code-derived representations are not merely preserved during text training — they actively improve the efficiency of text training, enabling the model to extract more reasoning capability per text token than it could from scratch.

Continual pre-training (code as distribution anchor): The paper's deliberate inclusion of 10% code in the continual pre-training mixture for code-initialized models — explicitly to "avoid a full distribution shift and maintain the benefits of code" — reveals an implicit hypothesis that code-derived representations are fragile and require sustained exposure. The finding that balanced → text and code → text both outperform balanced-only (which has the same total code token count but distributed differently across phases) supports this: 200B code tokens spread across initial + continual pre-training produce better NL reasoning than 200B code tokens concentrated in initial pre-training only. Code serves as a "reminder" that prevents text-dominated training from overwriting code-derived reasoning patterns.

Cooldown (code as capability finisher): The cooldown experiments (Section 3.5, Figures 6 and 7) reveal that adding code to the final 10% of training — a phase explicitly designed to up-weight high-quality data — produces benefits that are qualitatively different from simply adding more code during main pre-training. Cooldown with code improves NL reasoning by 3.6% and world knowledge by 10.1% relative to no cooldown (Figure 6), while cooldown without code provides no NL reasoning improvement and only 3.1% world knowledge improvement. This asymmetry is crucial: high-quality code in the final training phase provides reasoning benefits that high-quality text in the same phase does not. Something about code — its structure, its correctness, its compositional logic — makes it an especially effective "finisher" for the model's representations during the final optimization steps.

Comparison to prior work: Prior work established that cooldown (or "annealing") phases with high-quality data improve model quality (Parmar et al., 2024; Team et al., 2023), but didn't investigate whether which data appears in cooldown matters beyond general quality. The paper shows that code in cooldown is not merely "more high-quality data" — it provides distinct benefits from text in cooldown, even when the text is also high-quality. This implies that the cooldown phase may be the optimal time to introduce data with specific structural properties that the model can incorporate during its final representational refinement, and that code is an especially effective choice for this role.

Significance level: This is a conceptual refinement that changes how practitioners should think about training stage specialization. The insight that code serves three distinct functions (representation builder at initialization, distribution anchor during continued training, capability finisher during cooldown) provides a framework for optimizing data mixtures at each stage independently, rather than treating pre-training data as a single static mixture. This directly motivates future work on stage-specific data curation — for instance, maximizing algorithmic correctness in initialization code, diversity in continued code, and stylistic quality in cooldown code.

Innovation 4: Win-Rates as a Generative Quality Diagnostic Reveals Code's Hidden Benefits

A subtle but significant methodological contribution is the paper's use of LLM-as-a-judge win-rates on non-code open-ended prompts to measure code's impact on generative quality, revealing benefits that traditional discriminative benchmarks would miss. The finding that code-initialized models (both code → text and balanced-only) beat the text-only baseline by 6.6% in win-loss difference on the Dolly-200 English evaluation set (Section 3.1, Appendix C.1) is not merely an additional data point — it addresses a known tension in the literature.

Prior work (Üstün et al., 2024; Ouyang et al., 2022) has documented that improvements on academic benchmarks can come at the cost of degraded open-ended generation quality — models optimized for multiple-choice accuracy may produce less helpful, less fluent, or less instruction-following text. This creates a risk that code's benchmark improvements might reflect "teaching to the test" rather than genuine capability improvements. The win-rate results provide direct evidence against this interpretation: code-trained models generate responses that a capable judge LLM (Command-R+) prefers across a diverse set of open-ended, non-code prompts, even when the code-trained model has seen less total text data than the text-only baseline.

The cooldown results (Figure 7) further strengthen this finding. Cooldown with code achieves a 52.3% win-rate against the no-cooldown baseline — 4.1 percentage points higher than cooldown without code (48.2%). This isolates code's contribution to generative quality from the contribution of the high-quality text and instruct-style data that both cooldown variants share. The mechanism is plausibly that code teaches the model to produce outputs that are more structured, more logically coherent, and more directly responsive to the prompt — qualities that benefit both code generation and open-ended text generation but are not well-measured by multiple-choice accuracy.

Comparison to prior work: Zhang et al. (2024) studied code's impact in instruction fine-tuning and found reasoning improvements, but didn't evaluate generative quality separately. Zhu et al. (2024) reported general NL benchmark performance for code-trained models but didn't provide win-rate comparisons. This paper is the first to demonstrate that code in pre-training improves generative quality as judged by an independent LLM on non-code tasks, closing a gap between benchmark-centric evaluation and practical user-facing performance.

Significance level: This is an empirical validation of a claim the field had assumed but not proven: that code makes models "better" in ways humans or capable evaluators can perceive, not just in ways benchmark scores detect. It also establishes a methodological precedent — using LLM-as-a-judge win-rates to evaluate pre-training data decisions — that future work on data composition can adopt. The 6.6% win-rate difference is substantial in the context of LLM evaluation, where win-rate differences of 2–3% are often treated as meaningful.

Innovation 5: Trade-off Structure Between Code and Non-Code Performance Is Controllable, Not Inevitable

The paper reframes the "code vs. text" trade-off from a zero-sum constraint into a controllable optimization problem by demonstrating that the relative performance on code and non-code tasks can be tuned through independent levers: code proportion, code quality, initialization strategy, and training stage allocation. This is not obvious a priori — one might expect that any benefit code provides to reasoning necessarily comes at the cost of world knowledge (since code tokens displace factual text), or that maximizing code performance requires sacrificing NL performance. The paper shows these trade-offs are real but movable.

The evidence comes from comparing complete training recipes in Table 2:

Balanced-only achieves the best code performance (1.4× balanced → text's code score) but the worst NL reasoning among code-inclusive variants (2.5% relative below balanced → text in non-code tasks).
Balanced → text achieves the best NL performance (2.5% relative above balanced-only) but substantially worse code performance (55% relative below balanced-only).
Code → text achieves intermediate performance on both dimensions but, after cooldown with code, nearly matches balanced → text on NL tasks while exceeding balanced → text on code by 29% relative.

The critical insight is that these performance profiles are not fixed properties of "training with code" — they are engineering choices controlled by (a) the ratio of code to text in initial pre-training, (b) whether code is included in continual pre-training, (c) whether high-quality code appears in cooldown, and (d) the quality of the code itself. An organization targeting general-purpose reasoning can choose balanced → text with synthetic code and cooldown; an organization targeting code generation can choose balanced-only with synthetic code; an organization wanting a compromise can choose code → text with cooldown. The paper provides the first systematic map of this trade-off space.

Moreover, the finding that all variants with code outperform text-only on NL reasoning (Figure 2, all code-inclusive variants are above text-only) means the trade-off is asymmetric: code provides net positive benefits to reasoning regardless of configuration, while the cost is primarily in world knowledge (which degrades at high code proportions, Figure 4) and code performance (which varies dramatically with code proportion). This asymmetry means practitioners can choose their point on the trade-off curve without risk of falling below the text-only baseline on reasoning — a reassuring result that justifies the widespread practice of including code even when the primary use case is natural language.

Comparison to prior work: Prior work treated the code-text mixture ratio as a hyperparameter to tune for overall performance (Dubey et al., 2024), but didn't decompose the trade-off across capability categories or show that different training stages provide independent control knobs. This paper's demonstration that initialization, continual pre-training, and cooldown can be independently optimized to shift the trade-off frontier represents a new level of granularity in understanding data mixture effects.

Significance level: This is a practical engineering insight that converts an opaque trade-off into a controllable design space. It provides direct guidance for organizations making pre-training data decisions: if world knowledge matters, keep code below 50% and include cooldown with code; if reasoning matters most, use code → text initialization; if you need both, use balanced → text with synthetic code and cooldown. The paper doesn't claim to have found the Pareto-optimal frontier — there are infinitely many possible stage-wise allocations — but it establishes the existence of the frontier and provides waypoints.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted on the MATH benchmark (Hendrycks et al., 2021), specifically the split from Lightman et al. (2022): 12,000 training questions used for generating verifier training data and 500 test questions used for all reported evaluations. MATH consists of high-school competition-level mathematics problems requiring multi-step symbolic reasoning, with final answers that can be graded for exact match. This domain is chosen because the base model already possesses the necessary mathematical knowledge, and the challenge is drawing complex inferences — precisely the regime where the paper hypothesizes test-time compute should be most beneficial. For generative quality evaluation, the paper uses the Dolly-200 English dataset (Üstün et al., 2024; Singh et al., 2024b), consisting of 200 hand-picked open-ended prompts from Dolly-15K (Conover et al., 2023) that capture general-purpose, non-code use cases.
Base model(s). All experiments use PaLM 2-S* (Codey) (Anil et al., 2023) as the base language model. The paper states this model is "representative of the capabilities of many contemporary LLMs" (Section 4) and achieves non-trivial but far-from-saturated performance on MATH (roughly 10–19% pass@1 depending on prompt and sampling configuration). This places it in a useful regime for studying test-time compute: there is substantial room for improvement, and the model possesses enough capability that test-time strategies can amplify rather than being limited by near-zero baseline performance. For the FLOPs-matched comparison, a second model with approximately 14× more parameters is used as the pretraining-scaled baseline, with this larger model evaluated using only greedy decoding (no test-time compute augmentation).
Metrics. The primary metric is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground-truth answer exactly, as determined by the grading function released by Lightman et al. (2022) (Appendix G). For difficulty-dependent analysis, accuracy is reported separately within each of five difficulty quintiles. For generative quality, the metric is LLM-as-a-judge win-rate using Cohere Command-R+ as the judge: for each of 200 open-ended prompts, the judge compares two completions and selects the preferred one based on criteria including correctness of reasoning, responsiveness to the instruction, factual accuracy, and grammatical fluency, with win-rate computed as the fraction of comparisons where a given model's output is preferred (accounting for both orderings to control for position bias). For code evaluation, pass@1 on HumanEval-Python and MBPP is used.
Baselines. The paper compares against several standard approaches:
- Majority voting: Among N independently sampled solutions, select the most common final answer string using exact match. No learned verifier is involved.
- ORM best-of-N weighted: Score N complete solutions with an Outcome Reward Model (a single correctness score per solution), then apply best-of-N weighted selection where solutions arriving at the same final answer have their scores summed, and the answer with the highest total score is selected (Li et al., 2023).
- PRM best-of-N weighted: Same as ORM best-of-N weighted, but using the Process Reward Model's step-level scores aggregated to a single solution score via last-step aggregation (which the paper finds performs best among min/prod/last in Appendix E, Figure 13). This is the primary search baseline throughout Sections 5 and 7.
- Parallel sampling (for revisions): Generate N independent solutions from the revision model and select the best via either verifier-based scoring or majority voting, representing the non-sequential extreme of the revision space.
Generation budget / compute accounting. The universal unit of test-time compute is one generation, defined as one complete sampled answer from the base LLM. For best-of-N and beam search with N beams, the budget is simply N generations. For lookahead search with k lookahead steps, the cost is modeled as N × (k+1) generations to account for the additional rollout computation (Section 5.3). Budgets are swept across powers of 2, typically from 2⁰ (1 generation) to 2⁹ (512 generations), with a maximum of 256 generations for the main search experiments. For FLOPs-matched comparisons, pretraining FLOPs are approximated as 6ND_pretrain and inference FLOPs as 2ND_inference (following standard scaling law conventions), and the budget multiplier for the smaller model with extra test-time compute versus an M× larger model is computed as M + 3·(D_pretrain/D_inference)·(M−1), where the critical ratio R = D_inference/D_pretrain determines how much extra inference budget the smaller model receives.
Cross-validation / statistical protocol. The paper uses two-fold cross-validation within each difficulty quintile on the 500-question test set to select compute-optimal strategies without contaminating evaluation with strategy selection (Section 3.2). Specifically, each difficulty bin is split in half; the best-performing strategy configuration (e.g., which search algorithm, which sequential-to-parallel ratio) is selected on one fold and evaluated on the other, with results averaged across both folds. This prevents the circularity of using test-set performance to choose the strategy that is then evaluated on the test set. Difficulty bins themselves are computed using either oracle pass@1 (fraction of 2048 samples correct) or predicted difficulty (PRM final-answer score averaged over 2048 samples), with five quintile bins ranging from easiest (bin 1, highest pass@1) to hardest (bin 5, lowest pass@1).

Main Quantitative Results

Search Against PRM Verifiers (Section 5)

The headline result for PRM-guided search is that compute-optimal strategy selection per difficulty bin improves efficiency by up to 4× over best-of-N weighted, with 16 generations of compute-optimal search matching the accuracy of 64 generations of best-of-N weighted (Figure 4). This gain is achieved by adaptively routing easy problems to best-of-N (where beam search over-optimizes the verifier) and medium-hard problems to beam search (where the verifier's guidance genuinely helps). The absolute performance: compute-optimal search with oracle difficulty bins reaches approximately 39.5% accuracy at 256 generations versus approximately 37% for PRM best-of-N weighted at the same budget (Figure 4).

Aggregate search algorithm comparison (Figure 3, left). Across all 500 test questions evaluated at generation budgets ranging from 1 to 256, beam search with M=4 (fixed beam width) significantly outperforms best-of-N weighted at low budgets: at 4 generations, beam search achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted. However, this advantage diminishes and then reverses at high budgets. At 256 generations, best-of-N weighted reaches approximately 37% while beam search (M=4) plateaus around 34% — a crossover consistent with verifier over-optimization, where aggressive search exploits imperfections in the PRM's scoring rather than finding genuinely better solutions. Majority voting trails all verifier-based methods substantially, reaching only about 29% at 512 generations, confirming that learned verification provides signal beyond inter-answer agreement.

Lookahead search variants consistently underperform at equivalent generation budgets. Both k=1 and k=3 lookahead applied to beam search fall below standard beam search and best-of-N weighted across most budget levels (Figure 3, left). This counterintuitive result — the "strongest" optimizer performs worst — is explained by the cost model: a lookahead search with k=3 consumes 4× the generations per explored beam, meaning fewer candidate solutions are evaluated for the same budget. The additional accuracy in step-level scoring is outweighed by the reduction in exploration breadth. At the highest budgets (512 generations), lookahead search variants converge to roughly the same performance as other methods but never surpass them.

Difficulty-dependent behavior (Figure 3, right). The aggregate comparison masks crucial heterogeneity that the difficulty-bin analysis reveals:

Bin 1 (easiest, highest base model pass@1): Beam search degrades performance with increasing budget. At 4 generations, beam search achieves roughly 78% versus best-of-N weighted at 68%, but at 256 generations, beam search drops to roughly 77% while best-of-N weighted rises to 88%. This is the clearest evidence of PRM over-optimization: on easy problems where the base model already produces correct solutions at a high rate, aggressive beam search finds solutions that score well under the PRM but are actually incorrect — the verifier signal is being exploited rather than trusted.
Bin 2: Qualitatively similar to bin 1 but less extreme. Best-of-N weighted improves faster than beam search, from roughly 14% at 4 generations to 60% at 256 generations for best-of-N, versus roughly 14% to 32% for beam search. Best-of-N maintains a clear advantage at high budgets.
Bin 3 (medium difficulty): Beam search consistently outperforms best-of-N weighted across all budget levels. At 256 generations, beam search reaches roughly 34% versus roughly 23% for best-of-N weighted. This is the regime where the PRM provides genuine guidance: the model can produce correct solutions with non-trivial probability, and the verifier helps identify them more efficiently than random sampling.
Bin 4 (harder): Beam search shows the strongest relative advantage over best-of-N, reaching roughly 17% versus roughly 10% at 256 generations. Both methods achieve low absolute accuracy, but beam search is substantially more sample-efficient.
Bin 5 (hardest): Both methods hover near 1–3% regardless of budget. No amount of test-time compute helps — the base model simply lacks the capability to produce correct solutions for these problems, making search and verification irrelevant.

Compute-optimal search results (Figure 4). The compute-optimal policy selects the best search strategy per difficulty bin at each budget level. This produces the following efficiency gains:

At 16 generations, compute-optimal (oracle bins) achieves approximately 27% accuracy, matching PRM best-of-N weighted at 64 generations — a 4× reduction in required compute.
Compute-optimal with predicted difficulty bins (using PRM score distribution rather than ground-truth pass@1) tracks the oracle version closely. The two curves "largely overlap" per the authors, with the predicted version reaching approximately 37% at 256 generations versus roughly 39.5% for oracle.
Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations, Figure 14) and majority voting (around 29%), confirming that the PRM is essential for the performance ceiling and that adaptive allocation extracts more value from the same verifier.

PRM vs. ORM comparison (Appendix F, Figure 14). The PRM using last-step aggregation outperforms a separately trained ORM across all sample budgets, with the gap widening at higher sample counts. At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted, and roughly 30% for majority voting. The PRM's advantage is attributed to representation learning: even though last-step aggregation reduces the PRM to ORM-like behavior at aggregation time (using only the final step's score), the per-step training objective causes the model to learn better internal representations that generalize to solution-level quality assessment.

Revision Model Results (Section 6)

The headline result for revisions is that compute-optimal sequential-to-parallel ratio selection yields up to 4× efficiency gains over parallel-only sampling, with 64 generations of compute-optimal revisions matching 256 generations of parallel best-of-N weighted (Figure 8). The absolute performance: compute-optimal oracle revisions reach approximately 44% at 256 generations versus roughly 41% for PRM best-of-N weighted and approximately 37% for parallel-only (both also at 256 generations).

Revision model pass@1 trajectory (Figure 6, left). The revision model's per-step accuracy improves progressively through the revision chain: starting from approximately 18.2% pass@1 at step 1, accuracy rises to roughly 24–25% by steps 15–20 and remains in the 23–25% range through 64 sequential steps. This is evidence that the model has learned a generalizable revision skill — the improvement continues well beyond the 4-revision training horizon, suggesting the model can use its own previous outputs as informative context for producing better answers. However, the improvement is modest (approximately 6–7 percentage points absolute gain from step 1 to plateau), and the performance ceiling is well below what search achieves at equivalent budgets, indicating that revisions refine answers within a limited basin of attraction rather than exploring qualitatively different solution strategies.

Sequential vs. parallel comparison (Figure 6, right). At 64 total generations, the sequential configuration (one chain of 64 revisions) versus parallel (64 independent samples) shows:

Sequential + best-of-N weighted: approximately 41.5%
Parallel + best-of-N weighted: approximately 39%
Sequential + majority: approximately 38%
Parallel + majority: approximately 35%

Sequential sampling outperforms parallel under both verifier-based and majority-based selection, with a gap of roughly 2.5 percentage points for verifier selection and roughly 3 percentage points for majority voting. The fact that sequential outperforms parallel even under majority voting (which uses no learned verifier, only inter-answer agreement) suggests the benefit comes from the revision model genuinely producing better candidates in later steps, not merely from the verifier exploiting additional context from seeing previous revisions.

Sequential-to-parallel ratio sweep (Figure 7, left). For a fixed generation budget of 256, sweeping the ratio of sequential depth to parallel breadth reveals an optimal intermediate configuration: fully sequential (rightmost point) achieves approximately 42%, fully parallel (leftmost point) achieves approximately 40%, and the optimal mixed configuration (sequential-to-parallel ratio around 2¹ to 2³, meaning 2–8 sequential revisions per chain with multiple parallel chains) achieves roughly 43–44%. At lower budgets (8–32 generations), purely sequential is optimal — the curves are monotonically increasing with sequential ratio, meaning the model benefits more from refining a few high-quality attempts than from sampling many independent ones when the total budget is tight.

Difficulty-dependent optimal ratio (Figure 7, right). Breaking out sequential-to-parallel ratio sweeps by difficulty quintile at 128 generations reveals the familiar pattern:

Bin 1 (easiest): Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to how the budget is allocated — the model produces correct answers reliably regardless of strategy.
Bin 2: Slightly favors higher sequential ratios: approximately 63% at fully sequential versus 58% at fully parallel.
Bin 3 (medium): A clear optimum emerges at moderate sequential-to-parallel ratios (around 2¹ to 2³), achieving approximately 42% versus 35% at the fully parallel extreme and roughly 37% at fully sequential. This is the regime where balanced exploration-exploitation matters most.
Bin 4: Similar pattern to bin 3, with peak at moderate ratio reaching roughly 18% versus 14% at fully parallel.
Bin 5 (hardest): All ratios produce roughly 2–3% accuracy. No allocation strategy helps — the model cannot produce correct answers regardless of how revisions are organized.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin (with two-fold cross-validation):

At 64 generations, compute-optimal oracle achieves approximately 40%, matching PRM best-of-N weighted (parallel) at 256 generations — the 4× improvement.
Compute-optimal predicted bins perform slightly below oracle at high budgets (approximately 41% at 256 generations versus 44% for oracle), but still substantially outperform the parallel baseline (approximately 37%).
The parallel baseline appears to plateau around 36–37% at high budgets (Figure 8, the parallel-only curve flattens after 64–128 generations), while compute-optimal scaling continues to improve through 256 generations. This suggests that adaptive allocation not only shifts the efficiency curve but may actually increase the asymptotic performance ceiling — the model extracts more total capability from the same verifier and base model by adaptively choosing its inference strategy.

Revision model verifier distribution shift (Appendix J, Figure 15a). The PRM trained on base model outputs (from the search experiments) underperforms when applied to revision model outputs: sequential + base-LM PRM achieves roughly 40% at 64 generations versus sequential + revision-specific ORM at roughly 42%. This confirms that verifier distribution shift is a practical concern — the revision model's output distribution differs from the base model's, and a verifier trained on the wrong distribution loses accuracy. The revision-specific ORM is trained on revision model outputs and includes previous revisions in context (Appendix J, Figure 15b shows including revision history provides a small additional improvement of roughly 1–2 percentage points).

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

The headline result for the FLOPs-matched comparison is that a smaller model with compute-optimal test-time scaling can outperform a ~14× larger model, but only on easy-to-medium difficulty problems and primarily in low inference-to-pretraining ratio regimes. The results are difficulty- and R-dependent, and the paper does not claim universal superiority of test-time compute over pretraining.

Revisions (Figure 9, left). Comparing PaLM 2-S* with compute-optimal revisions against the ~14× larger model (greedy decoding, no test-time compute augmentation):

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy (bin 1)	+11.8%	+3.5%	−11.9%
Medium (bins 2–3)	+27.8%	+16.7%	+5.4%
Hard (bins 4–5)	+21.6%	negative (implied)	−37.2%

(These numbers come from the bar chart in Figure 1, top-right, which aggregates difficulty bins into three coarse groups: easy, medium, and hard.)

At R ≪ 1 (the inference-to-pretraining token ratio is very small, meaning the model is used for relatively few inference tokens compared to its training cost), test-time compute with the smaller model outperforms the larger model across all difficulty levels — including hard problems, where the 21.6% advantage is notable given that test-time compute provided essentially no benefit on bin 5 in the within-model experiments (Figure 3, right; Figure 7, right). This suggests that even on hard problems, the larger model may be so poorly utilized by greedy decoding that a smaller model with smart inference still wins, at least when the FLOPs accounting heavily favors the smaller model.

At R ≫ 1 (high inference volume relative to training), test-time compute remains preferable only on easy questions (−11.9% on easy actually means the larger model wins here, per the sign convention in the bar chart indicating larger model advantage). Medium questions show a small advantage (+5.4%), and hard questions show a substantial disadvantage (−37.2%). The crossover reflects the FLOPs accounting: when inference dominates the total FLOPs budget, the per-token cost advantage of the smaller model shrinks because inference FLOPs scale linearly with model size, and the larger model's better baseline performance eventually outweighs the smaller model's inference-time strategy advantages.

PRM search (Figure 9, right). The pattern for PRM search is starker, with test-time compute showing weaker benefits than revisions:

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy	+19.1%	+2.2%	+2.0%
Medium	0.0%	−35.3%	−30.8%
Hard	−3.6%	−35.3%	−52.9%

PRM search is only clearly preferable to the larger model on easy questions at low R. At moderate R (0.79), the comparison is essentially break-even on easy questions and substantially negative on medium and hard. At high R (22), the larger model dominates on medium and hard questions, with −52.9% relative disadvantage on hard questions. This is consistent with the within-model finding that PRM search is bottlenecked by verifier over-optimization (Section 5.3) — the search can redistribute probability mass among the base model's existing capabilities but cannot create new capabilities, making it less effective than pretraining when the base model's capability ceiling is the limiting factor.

The revision-based approach outperforms PRM search in the FLOPs-matched comparison because revisions modify the proposal distribution itself (potentially expanding the set of problems the model can solve), while PRM search only selects among existing proposals. This distinction becomes critical in the FLOPs-matched setting, where the baseline (the larger model) has fundamentally higher capability. Revisions can partially close the capability gap by iteratively improving answers; search cannot.

Figure 9 interpretation. The line plots in Figure 9 show accuracy per difficulty bin as test-time compute scales, with the ~14× larger model's greedy performance shown as horizontal lines (stars) at three x-axis positions corresponding to the three R values. Where the compute-optimal scaling curve is above the star, test-time compute wins. The key pattern: on bin 1 (easiest, purple line), the scaling curve is above all three stars for revisions, indicating test-time compute wins across all R regimes. On bin 5 (hardest, blue line), the curve is essentially flat near 0–5% and below all three stars for both revisions and search — no amount of test-time compute helps, and pretraining is always preferable. On intermediate bins, the crossover depends on R, with larger R shifting the advantage toward pretraining.

Ablation Studies and Robustness Checks

PRM step-wise score aggregation (Appendix E, Figure 13): Comparing "min" (minimum score across steps), "prod" (product of step-level correctness probabilities), and "last" (only the PRM's prediction at the final step) for aggregating per-step PRM scores into a single solution-level score reveals that "last" performs best, achieving roughly 37% at 256 samples versus roughly 35% for "min" and roughly 27% for "prod". This contradicts prior work by Lightman et al. (2023) and Wang et al. (2023) which found "min" to be optimal. The paper attributes the discrepancy to differences in PRM training — Monte Carlo soft labels rather than binary correctness labels — which changes the distribution of per-step scores such that intermediate step predictions become less reliable, while the final-step prediction remains well-calibrated. This finding effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM (Figure 14), indicating that the step-level training objective acts as beneficial representation learning even when intermediate predictions are discarded at inference.

PRM vs. ORM scaling behavior (Appendix F, Figure 14): The PRM consistently outperforms the ORM across all sample counts, with the gap widening at higher budgets: at 2048 samples, PRM reaches approximately 40% versus ORM's approximately 35% and majority voting's approximately 30%. This confirms that the PRM's training procedure — Monte Carlo rollout supervision at the step level — produces a verifier that scales better with increased compute than a standard outcome-level verifier, supporting the paper's emphasis on PRMs as the superior verifier architecture for test-time compute scaling.

Revision model verifier: base-LM PRM vs. revision-specific ORM (Appendix J, Figure 15a): Applying the base model's PRM to revision model outputs results in degraded selection quality: sequential + base-LM PRM achieves roughly 40% at 64 generations versus roughly 42% for a version-specific ORM trained on revision model data. This demonstrates distribution shift as a practical obstacle — verifiers trained on one model's output distribution do not transfer perfectly to another, even when the architectures are identical. The paper addresses this by training a separate ORM for revision experiments, but this adds engineering complexity and highlights a limitation of the current approach: each proposal distribution modification requires its own verifier adaptation.

Including revision history in verifier context (Appendix J, Figure 15b): The revision-specific ORM slightly benefits from seeing previous revisions in its context (approximately 1–2 percentage points improvement at 64 generations over the no-history ablation). However, both variants outperform the parallel baseline, confirming that the sequential sampling benefit is not solely an artifact of the verifier having access to more information. The revision model genuinely produces better candidates in later steps, and even a verifier blinded to revision history can select these better candidates more effectively than it can select among independent parallel samples.

Oracle vs. predicted difficulty bins (Figures 4 and 8): The compute-optimal strategy curves for oracle difficulty (computed from ground-truth pass@1 over 2048 samples) and predicted difficulty (computed from PRM final-answer score distribution over the same 2048 samples) largely overlap, particularly in the search setting (Figure 4). In the revision setting (Figure 8), predicted bins show slightly lower performance at high budgets (approximately 41% vs. 44% at 256 generations), but still substantially outperform the non-adaptive baselines. This is a critical robustness check: the compute-optimal approach works without access to ground-truth answers, using only the PRM's own assessment of solution quality as a difficulty signal. The small gap at high budgets in the revision setting suggests that the PRM's difficulty estimates are slightly noisier for revision model outputs (consistent with the distribution shift finding in Appendix J), but the adaptive strategy remains beneficial even with imperfect difficulty estimation.

Majority voting for revisions (Appendix B, Figure 10): The sequential-to-parallel ratio trends observed with verifier-based selection are qualitatively replicated using simple majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This robustness check confirms that the benefits of sequential revision are not an artifact of the verifier exploiting additional context — they persist with a selection mechanism that relies only on inter-answer agreement and has no access to revision history.

ReST^EM-trained revision model (Appendix K, Figure 16): An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024) — a reinforcement-learning-inspired training procedure — produces a negative result: additional sequential revisions substantially hurt performance with this model. At 256 generations, fully sequential performance with the ReST^EM model drops to approximately 33.5% compared to roughly 38.5% at the optimal intermediate ratio. The paper hypothesizes that on-policy data collection during ReST^EM training amplifies spurious correlations in revision trajectories, causing the model to learn degenerate revision behaviors that fail to generalize. This is an important finding because it demonstrates that revision model training is sensitive to the data generation procedure, and that naïve reinforcement-inspired optimization can destroy the revision capability rather than improving it. The offline data construction approach (pairing independently sampled correct and incorrect solutions using edit distance) appears to be crucial for the positive results in the main experiments.

Code proportion sweep: performance sensitivity (Section 3.3, Figure 4): Varying code proportion from 0% to 100% (200B tokens, six independent models) reveals that NL reasoning peaks at 25% code with a 3.4% relative gain over 0% code, remains near plateau through 50% code, degrades slowly at 75% code, and drops sharply at 100% code (−18.3% relative to 0% code). World knowledge shows a steady monotonic degradation with increasing code proportion, dropping 86.1% at 100% code relative to 0% code. Code performance increases near-linearly from 25% to 100% code, with 100% code yielding 2.6× the performance of 25% code. This validates that the optimal code proportion for general non-code performance involves a real trade-off, not a trivial "more code is better" relationship.

Model scale invariance (Section 3.2, Figure 3 [paper's Figure 9]): Scaling from 470M to 2.8B parameters with the same token budget increases average performance by 30–33% across NL reasoning and world knowledge for the three key variants (balanced, balanced→text, code→text), with world knowledge nearly tripling for all variants. Crucially, the relative ordering of variants is preserved: code→text and balanced→text outperform balanced-only on NL reasoning at both scales, while balanced-only dominates on code generation. The trade-off between NL tasks and code increases with scale — at 2.8B, code→text and balanced→text improve over balanced-only by 6.9% and 6.1% in NL tasks but fall behind by 43.1% and 46.3% in code generation. This scale-consistent finding strengthens the paper's claim that its results are not merely artifacts of limited model capacity, though the caveat remains that 2.8B is far from frontier scale.

Critical Assessment

Do the experiments support the claim that code provides "critical improvements to non-code performance" with 8.2% relative increase in NL reasoning, 4.2% in world knowledge, and 6.6% in win-rates?

Yes, with qualifications. The claim is supported by Table 2 and the associated figures, which show the balanced→text variant (after cooldown) outperforming text-only across all these metrics. However, several aspects deserve scrutiny:

The "best variant" is the result of extensive post-hoc selection. The paper tests initialization strategies (Section 3.1), code proportions (Section 3.3), code quality interventions (Section 3.4), and cooldown configurations (Section 3.5), then selects balanced→text with synthetic code and cooldown as the best recipe. The 8.2% relative improvement figure is for this specific configuration, discovered through the experimental process, not a pre-registered hypothesis test. Without a held-out validation set for meta-parameter selection (the cross-validation protocol applies only to the compute-optimal search/revision experiments in the other paper, not to this study's recipe selection), there is a risk of overfitting the reported gains to the test set through the choice of which variant to highlight. The paper does not report performance of the "best variant" on a separate validation split used for recipe selection; the 500-question test set appears to serve double duty as both the selection criterion (implicitly, through the authors' choice of which results to feature) and the evaluation metric.

The text-only baseline uses 400B tokens of SlimPajama with code removed, but the code-inclusive variants use SlimPajama plus additional code data — meaning the total token count differs. The paper reports total token budgets (Table 2) and they are matched across variants in terms of total tokens seen, but this matching means the text-only model sees 400B tokens of pure text while the balanced→text model sees 280B text tokens + 120B code tokens. If text is more informative than code for NL reasoning (as the code proportion results in Figure 4 suggest — 25% code is optimal, meaning 75% text is better than more code), then the balanced→text model is at a text disadvantage (280B vs. 400B text tokens). The fact that it still outperforms text-only on NL reasoning despite seeing 120B fewer text tokens is strong evidence for code's benefit. However, the improvement might be understated for world knowledge (where text quantity matters more) and overstated for reasoning if the text-only model is simply under-trained on the specific reasoning patterns that code also happens to teach.

The evaluation suite, while broad, has limitations for measuring "general performance." The NL reasoning category is comprised of 11 multiple-choice or classification benchmarks. These measure a specific kind of reasoning — selecting the correct answer from provided options — that may not capture the fluid, multi-step, generative reasoning that practitioners associate with "code improves reasoning." The Dolly-200 win-rate evaluation partially addresses this by measuring open-ended generation quality, but 200 examples evaluated by a single judge LLM (Command-R+) is a narrow window into generative capability. The judge model itself may have systematic biases — for instance, preferring completions that match its own training distribution, which might favor models with certain stylistic properties regardless of factual accuracy. The paper does not report human evaluation or inter-judge agreement, making it difficult to assess whether the 6.6% win-rate difference represents a perceptible improvement to human users or an artifact of the specific LLM judge.

The 12× boost in code performance is comparing text-only (essentially zero code capability) to the best code-inclusive variant, which is a straw-man comparison. The text-only model gets 0.4% pass@1 on code benchmarks (Table 2, text-only pre-training), while the best code variant (balanced-only, chosen here rather than balanced→text) gets 9.0%. The 12× figure reflects the ratio of a near-zero baseline to a moderate-but-not-impressive absolute score — 9% pass@1 on HumanEval and MBPP is far from state-of-the-art code generation. The paper's claim that "code is a critical building block for generalization far beyond coding tasks" is not directly illuminated by the 12× code improvement, since that improvement merely confirms that including code improves code generation (expected). The interesting question is whether code generation capability causally drives the NL improvements, or whether both co-occur from the same representation learning without one causing the other. The paper does not provide evidence to distinguish these hypotheses.

Do the experiments support the claim that "code quality and properties matter" with synthetic code providing 9% and 44.9% relative improvements?

Yes, strongly, but with an important caveat about the synthetic code dataset. The paper's synthetic code is described as "a proprietary synthetically generated code dataset that consists of Python programming problems that have been formally verified" (Section 2.1). The fact that this dataset is proprietary means the results are not independently reproducible — another research group cannot replicate the experiment without access to Cohere's internal synthetic code generation pipeline. This is a significant limitation for a paper whose central claim is about the importance of code quality, since "quality" is operationalized through a black-box dataset. Key questions that the paper does not answer: What fraction of the synthetic code's benefit comes from formal verification (correctness) versus other properties like consistent style, clear variable naming, or well-structured problem decomposition? Could an open-source synthetic code dataset (e.g., filtered and verified subsets of public repositories) achieve similar gains? The 44.9% relative improvement in code performance is impressive but expected — training on verified Python should improve Python code generation. The more surprising and important result is the 9% improvement in NL reasoning, which suggests that verified code teaches transferable reasoning patterns. Without knowing what makes the synthetic dataset "high quality," it is unclear which of these patterns matter most.

The code-adjacent and markup results are actually more informative for understanding mechanisms because they use publicly documented datasets (The Stack, SlimPajama) and show that different code properties produce different transfer patterns. The finding that code-adjacent data improves NL reasoning (+6.3%) but degrades code (−9.4%) is a clean demonstration of trade-off — the software engineering ecosystem teaches general reasoning but dilutes domain-specific code skill. This result is both more interpretable and more actionable than the synthetic code result, since practitioners can choose to include or exclude code-adjacent artifacts based on their performance priorities.

Do the experiments support the claim that "code in cooldown enables further improvement across all tasks"?

Yes, with the strongest evidence coming from the head-to-head comparison of cooldown with code versus cooldown without code (Figures 6 and 7). The finding that cooldown without code provides essentially no NL reasoning improvement (+0% relative to no cooldown, Figure 6) while cooldown with code provides +3.6% relative improvement is compelling because it isolates code's contribution from the general benefit of cooldown-phase training. The cooldown without code variant still up-weights high-quality text, math, and instruct-style data — if cooldown benefits came merely from "training on better data at the end," both variants would improve. The asymmetry — only the code-inclusive cooldown helps reasoning — suggests code plays a specific role in the final representational refinement.

The 52.3% win-rate for cooldown with code versus no cooldown (Figure 7) is also notable, but the 4.1 percentage point advantage over cooldown without code (48.2%) is a relatively small margin given the 200-example evaluation set. With 200 comparisons, a 4.1% difference represents roughly 8 additional wins, and the paper does not report confidence intervals or statistical significance testing for win-rate differences. The win-rate result should be interpreted as directional evidence rather than precise quantification.

Caveat on cooldown generalizability: The cooldown experiments use a single base model (balanced→text, 470M parameters) with a single cooldown data mixture. The 20% code fraction in cooldown is a specific choice that is not systematically ablated — the paper does not test whether 10% or 40% code in cooldown would be better or worse. The cooldown experiments are also not replicated at 2.8B scale, so scale dependence of cooldown effects remains unknown. Given that the paper's own results show that optimal code proportion depends on the training phase (25% in initial pre-training, 10% in continual pre-training, 20% in cooldown), there is likely an optimal cooldown code fraction that is not explored.

What experiments would have strengthened the paper but were not run?

1. Cooldown experiments at 2.8B scale. The paper validates that initialization trends transfer from 470M to 2.8B but does not replicate cooldown at the larger scale. Cooldown effects might be scale-dependent — larger models may benefit more (because they have more capacity to incorporate high-quality data at the final stage) or less (because they have already extracted more value from the main pre-training). The paper's claim that cooldown is beneficial rests entirely on 470M-scale evidence.

2. Code quality ablation with more granular quality levels. The synthetic code dataset is treated as a binary intervention: web-code vs. web-code + 10% synthetic. A dose-response curve — testing 5%, 10%, 20%, 50% synthetic — would reveal whether the benefits saturate (suggesting a minimum threshold of high-quality code is sufficient) or scale with synthetic proportion (suggesting web-code is fundamentally limited and should be replaced as much as possible). The current design cannot distinguish these hypotheses.

3. An open-source synthetic code baseline. Using a publicly available verified code dataset (e.g., filtered portions of The Stack with test-passing verification, or code from competitive programming platforms with known solutions) would make the central quality result reproducible and would provide a lower baseline for comparing the proprietary synthetic data's effectiveness. This is particularly important because the paper's strongest claim — "improvements to code quality have an outsized impact across all tasks" — is supported by a result that no one outside Cohere can verify.

4. Human evaluation of generative quality. The LLM-as-a-judge win-rates are an automated proxy. Human evaluation on the same Dolly-200 prompts would calibrate the judge's reliability and establish whether the 6.6% win-rate difference translates to human-perceptible quality improvements. Given the paper's emphasis on generative quality as a counterpoint to benchmark-centric evaluation, human validation would substantially strengthen this claim.

5. Mechanistic probing experiments. The paper demonstrates that code helps but does not investigate how. Do code-trained models develop better attention patterns for long-range dependencies? Do they learn more compositional representations? Do they acquire systematic syntactic generalization that transfers to natural language? Probing experiments (e.g., analyzing attention heads, representation similarity, or targeted behavioral tests) would convert the paper from a "what" study into a "why" study. The current results establish the empirical phenomena but leave the mechanisms entirely speculative.

6. Effect of code data volume on the quality findings. The synthetic code dataset is 3.2B tokens. The web-code baseline is 139B tokens. The 10% replacement means the model sees 12.5B fewer tokens of web-code and 3.2B more tokens of synthetic code — a net reduction in total code tokens. The observed improvements might partly reflect the benefits of data deduplication or reduction of harmful examples (removing buggy web-code) rather than the benefits of synthetic code per se. A baseline that removes 10% of web-code without replacement would help disambiguate: if simply removing random web-code also improves performance, then synthetic code's benefit may be overstated.

Where do the paper's claims hold conditionally, and what are the precise conditions?

Claim: Code improves NL reasoning. Condition: holds for all code proportions from 25% to 100% versus text-only (Figure 4), holds for all initialization strategies (Figure 2), and holds at both 470M and 2.8B scales (Figure 9 [paper's numbering]). However, the magnitude depends on code proportion (optimal at 25%, degrades at 100%) and training stage (code in cooldown provides additional gains beyond code in main pre-training).

Claim: Code improves world knowledge. Condition: holds only for balanced→text initialization (+4.2% relative to text-only), not for code→text or balanced-only (both underperform text-only on world knowledge, Figure 2). The improvement is fragile and depends on having substantial text in the training mixture — at high code proportions, world knowledge monotonically degrades (Figure 4). Code in cooldown provides an additional world knowledge boost (+10.1% relative to no cooldown for balanced→text with code cooldown, Figure 6), but cooldown without code also helps (+3.1%), so only part of the gain is code-specific.

Claim: High-quality synthetic code has outsized impact. Condition: demonstrated for one specific synthetic dataset (formally verified Python) at one replacement ratio (10%) in code-only pre-training (Figure 5a) and in continual pre-training (Figure 5b). The claim that quality drives the effect (as opposed to other properties of the synthetic dataset like style, naming conventions, or problem decomposition structure) is an interpretation, not a demonstrated fact — the paper does not ablate different quality dimensions of the synthetic data.

Claim: Code in all phases of training provides the best overall performance. Condition: this is a summary of the recipe comparison in Table 2, which shows that balanced→text with cooldown including code achieves the best NL reasoning, world knowledge, and generative win-rates among the studied variants. However, this configuration (balanced initial pre-training with synthetic code, text-dominated continual pre-training with 10% code, and cooldown with 20% code) is one specific point in a large combinatorial space. Different allocations of the token budget across phases might yield different optimal recipes, and the paper does not systematically sweep the token allocation per phase.

Claim: Test-time compute with a smaller model can outperform a ~14× larger model. This claim belongs to the companion paper (the test-time compute scaling study on MATH) and is not directly tested in this study. This paper's experiments do not include FLOPs-matched comparisons between different-sized models — the scale comparison (Section 3.2) only shows that trends persist at 2.8B vs. 470M, not that a 470M model with optimized code mixture can beat a larger model with a worse mixture. The abstract's mention of "models with sizes ranging from 470M to 2.8B" refers to the scale invariance validation, not a training-inference trade-off analysis.

Overall assessment of experimental rigor. The paper's experimental design is ambitious and thorough along the dimensions it investigates — 64 trained models at non-trivial scale with systematic variation of code proportion, quality, and training stage. The multi-category evaluation (NL reasoning, world knowledge, code, generative win-rates) is substantially more comprehensive than prior work. However, the absence of mechanistic probing, the reliance on a proprietary synthetic dataset for the strongest quality result, the lack of cooldown replication at 2.8B scale, and the implicit post-hoc selection of the "best variant" without a held-out validation set for recipe selection are genuine limitations. The paper succeeds in establishing that code matters across many dimensions and that quality, proportion, and training stage modulate the effect — but the why remains largely unaddressed, and the precise optimal recipes are likely specific to the model architecture, token budget, and data distributions used.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted For and Prohibitively Expensive

The assumption or constraint. The compute-optimal test-time scaling framework (Section 3.2 in the companion paper on MATH) requires estimating each prompt's difficulty before allocating the inference budget. The paper's method for doing so — generating 2,048 samples per question and averaging either ground-truth pass@1 (oracle) or PRM final-answer scores (predicted) — consumes more compute than any of the inference budgets studied. The paper acknowledges this explicitly:

"We note the cost associated with estimating difficulty in this way still incurs additional computation cost during inference, which we do not account for... our experiments do not account for this cost largely for simplicity" (Section 3.2, companion paper)

But the acknowledgment does not resolve the problem — it only identifies it.

The consequence. The headline claim of "4× better efficiency" over best-of-N is computed after difficulty is known, without amortizing the cost of estimating it. In a realistic deployment where difficulty estimation must be performed per-query, the total compute cost would be difficulty_estimation_cost + strategy_execution_cost. Since the estimation cost (2,048 generations + PRM scoring) exceeds even the largest evaluation budgets studied (256–512 generations) by a factor of 4–8×, the effective efficiency gain over best-of-N would be negative in a deployment setting where difficulty is unknown a priori. The paper presents compute-optimal scaling as a practical strategy for improving inference efficiency, but the results are closer to an upper bound on what is theoretically achievable if difficulty could be estimated for free.

What evidence exists in the paper. The difficulty estimation cost is explicitly stated: 2,048 samples per question, with PRM scoring applied to each sample. The total token cost of this step is not calculated, but can be roughly estimated — 2,048 complete solutions × average solution length (likely hundreds to low thousands of tokens for MATH problems) represents millions of generated tokens per query. The paper provides no experiment testing cheaper difficulty estimation methods (e.g., 4–8 samples with score variance as a difficulty proxy, or a lightweight classifier trained on question text). The predicted vs. oracle difficulty curves (Figures 4 and 8 in the companion paper) demonstrate that the PRM can substitute for ground-truth labels, but both methods require the same 2,048-sample generation budget.

Mitigation status. The paper partially acknowledges the limitation (Section 3.2 of the companion paper) and frames cheap difficulty estimation as "a key avenue for future work," suggesting that models could be trained to predict difficulty directly from the question text. However, no such model is developed or evaluated, and no evidence is provided that difficulty can be reliably estimated from question text alone without running the model. Given that difficulty is defined relative to the specific base model's capabilities (not as an intrinsic property of the question), text-based difficulty estimation would require learning a function that maps question text to the base model's pass@1 rate — a non-trivial meta-learning problem that is not addressed. This is the most consequential unaddressed limitation for anyone seeking to deploy the compute-optimal framework in practice, because it means the reported efficiency gains are currently unrealizable without incurring a cost that negates them.

6.2 The ~14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison (Section 7 of the companion paper) scales model parameters by a factor of ~14× while holding training data fixed, following the LLaMA training paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), where both parameters and data are scaled equally. The paper states this choice explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 4, companion paper)

Additionally, the ~14× larger model is evaluated using only greedy decoding — no test-time compute augmentation, no majority voting, no best-of-N sampling, no verifier-guided selection. This makes the comparison asymmetric: one side deploys compute-optimal test-time strategies, the other deploys none.

The consequence. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy-medium problems at R ≪ 1 with revisions, Figure 1 bar charts in the companion paper) are measured against a baseline that is almost certainly weaker than a properly optimized larger model would be. A Chinchilla-optimal ~14× larger model — scaling both parameters and training data such that D_pretrain ∝ N — would likely outperform the parameter-only-scaled model used in the comparison. Additionally, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or majority voting with 8–16 samples, which would cost a small fraction of the total inference FLOPs budget) would create a substantially stronger baseline. The paper's FLOPs-matched results should therefore be interpreted as a lower bound on the benefits of pretraining — the true frontier where test-time compute and pretraining are equally effective likely lies somewhere between the paper's reported results and a substantially higher performance level for the larger model.

What evidence exists in the paper. The paper provides no ablation testing the larger model with any form of test-time compute augmentation. The FLOPs-matched comparison in Figure 9 (companion paper) places the larger model's greedy accuracy as a horizontal line (stars) at three x-axis positions corresponding to different R values, and compares against the smaller model's compute-optimal scaling curve. There is no experiment where the larger model also receives test-time compute scaled proportionally to its size, which would be the fairest comparison. The paper acknowledges the data-scaling limitation in Section 4 but does not discuss the greedy decoding asymmetry as a potential confound for interpretability.

Mitigation status. The paper partially addresses the parameter-scaling limitation by acknowledging it and deferring to future work. The greedy decoding asymmetry is not acknowledged or justified. A defender might argue that the comparison is still informative because it represents two extreme points in a design space: "spend all FLOPs on pretraining with minimal inference" vs. "spend FLOPs on smaller pretraining with sophisticated inference." But without exploring intermediate points — e.g., a moderately larger model with moderate test-time compute — the paper cannot claim to have found the optimal allocation of a fixed total FLOPs budget between pretraining and inference. The results establish that test-time compute can substitute for pretraining under specific conditions, not that it should or that the reported substitution ratio (~14×) is the true exchange rate.

6.3 The Synthetic Code Quality Result Is Non-Reproducible Due to Proprietary Data

The assumption or constraint. The single most striking result in the paper — that replacing 10% of web-code with synthetically generated, formally verified Python code yields 9% relative improvement in NL reasoning and 44.9% relative improvement in code performance (Figure 5a) — is supported by a dataset that is explicitly described as proprietary:

"we use a proprietary synthetically generated code dataset that consists of Python programming problems that have been formally verified. We treat this as a high-quality source of code data" (Section 2.1)

The paper does not release this dataset, describe its generation procedure in sufficient detail for replication, or provide examples of the synthetic code. It also does not test against any publicly available high-quality code dataset that could serve as a reproducibility baseline.

The consequence. The paper's central quality-related claim — that code quality has outsized impact and that correctness matters more than volume — cannot be independently verified. This matters for three reasons specific to this paper: (1) the synthetic code dataset is described as "formally verified," which could mean anything from "manually checked by humans" to "machine-checked against full functional specifications" — the strength of the verification directly affects the interpretation of the result; (2) the dataset is only 3.2B tokens, meaning its content could be highly curated along dimensions other than correctness (style, naming conventions, problem complexity, comment quality), and any of these could drive the observed improvements; (3) the 44.9% code improvement is measured on Python benchmarks (HumanEval, MBPP), and training on verified Python problems would be expected to improve Python generation — the extent to which this improvement reflects genuine transfer of reasoning capability versus domain-specific training on the benchmark language cannot be determined without access to the data.

What evidence exists in the paper. The paper provides Figure 5a (code+synth vs. code-only), Figure 5b (balanced+synth→text vs. balanced→text), and a brief description: "synthetically generated code dataset that consists of Python programming problems that have been formally verified" (Section 2.1). There are no examples of the synthetic code, no description of the formal verification methodology, no statistics on the dataset beyond total token count (3.2B), and no public release. The paper also does not test any open-source alternative — such as verified subsets of The Stack (filtering for repositories with passing CI/CD tests), code from competitive programming platforms (e.g., Codeforces with known correct solutions), or publicly available synthetic code datasets — which could provide a reproducibility floor.

Mitigation status. Not addressed. The paper does not claim the synthetic dataset will be released, does not provide a detailed-enough description for replication, and does not test an open-source proxy. This is a significant limitation for a paper whose abstract states "improvements to code quality have an outsized impact across all tasks" as a key finding — the finding is supported by a dataset that the research community cannot access, making the claim unfalsifiable in its current form. The code-adjacent and markup results (Figure 5a), which use publicly documented datasets, are actually more valuable for the research community because they can be replicated and built upon, even though their effect sizes are smaller.

6.4 Hardest Problems Show Near-Zero Improvement Regardless of Compute or Strategy

The assumption or constraint. The paper's compute-optimal test-time scaling framework assumes that increasing inference compute can improve performance. This assumption breaks down for problems where the base model's capability is fundamentally insufficient — when pass@1 is near zero, no amount of search, revision, or verifier guidance can help, because there are essentially no correct solutions in the model's output distribution to find or refine. The paper demonstrates this systematically: difficulty bin 5 (the hardest quintile of MATH problems, where the base model's pass@1 over 2,048 samples is lowest) shows accuracies of 1–3% across all methods, all budgets, and all allocation strategies (Figure 3 right, Figure 7 right, Figure 9 in the companion paper).

The consequence. Test-time compute can amplify existing capability but cannot create it from nothing. For any prompt where the base model's probability of generating a correct answer is below some threshold (the paper's bin 5 corresponds to pass@1 rates presumably in the low single digits or below), additional inference compute provides negligible returns regardless of how intelligently it is allocated. This is a fundamental ceiling on the approach: it offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path, and the FLOPs-matched comparison (Figure 9) confirms this: on bin 5, the ~14× larger model with greedy decoding substantially outperforms the smaller model with compute-optimal test-time scaling at all but the most favorable R ratios. The practical implication is that organizations deploying compute-optimal test-time strategies must accept that some fraction of queries will be "unreachable" regardless of inference budget, and must have a fallback strategy (e.g., routing to a larger model, flagging for human review, or declining to answer).

What evidence exists in the paper. The difficulty-bin analyses in Figures 3 (right), 7 (right), and 9 (companion paper) all show bin 5 performance near floor (1–3%) irrespective of method, budget, or allocation strategy. The FLOPs-matched comparison (Figure 9) shows the bin 5 scaling curve as essentially flat and below the larger model's performance at all R values. The paper explicitly acknowledges this limitation in Section 8 (companion paper): "These results suggest that if the base model's pass@1 is near zero on a problem class, no amount of test-time compute can help." However, the paper does not quantify the pass@1 threshold below which test-time compute becomes ineffective, which would be practically useful for predicting when the approach will fail.

Mitigation status. The paper acknowledges the limitation transparently but does not attempt to address it. This is appropriate given the paper's framing as an analysis of test-time compute scaling rather than a solution for hard problems, but it means the approach has a known failure mode with no proposed remedy. The paper's recommendation to prefer pretraining for hard problems (Section 7 takeaway) is a diagnosis, not a mitigation — it tells you when the approach will fail but does not help you make it work in that regime. A potential mitigation (combining compute-optimal test-time strategies with some form of retrieval-augmented generation or tool use to expand the base model's effective capability on hard problems) is not explored.

6.5 Sequential Revision Strategies Introduce Latency That Is Not Accounted For

The assumption or constraint. The paper measures test-time compute in units of "generations" (complete sampled solutions) and treats all generations as having equal cost. This is a reasonable proxy for total FLOPs but ignores latency — the wall-clock time required to produce a final answer. Sequential revision strategies, which represent a substantial fraction of the compute-optimal allocation on easy-to-medium difficulty problems (Figure 7), require generating a chain of revisions where each step depends on the output of the previous step. This is inherently serial: revision t+1 cannot begin until revision t is complete. Parallel best-of-N sampling, by contrast, can execute all N generations simultaneously given sufficient hardware.

The consequence. A compute-optimal allocation that uses 64 sequential revisions × 2 parallel chains (128 total generations) takes roughly 64 × generation_time in wall-clock time, while a purely parallel allocation of 128 generations takes 1 × generation_time. The sequential strategy is therefore ~64× slower in latency despite using the same total FLOPs. For latency-sensitive applications — interactive assistants, real-time decision-making, user-facing chatbots — this makes sequential-heavy strategies impractical regardless of their accuracy advantages. The paper's compute-optimal policies, which favor sequential revisions on easy problems (Figure 7 right, bin 2) and balanced sequential-parallel on medium problems (Figure 7 right, bins 3–4), would produce unacceptable latency in such settings. The paper's framing of these strategies as "more efficient" (4× fewer generations for equivalent accuracy) is therefore only correct under a throughput metric (total FLOPs per correct answer), not a latency metric (time to produce one answer).

What evidence exists in the paper. The paper does not measure or report latency for any experiment. There is no discussion of wall-clock time, no comparison of sequential vs. parallel latency at equivalent generation budgets, and no acknowledgment of this trade-off in the limitations section. The paper's cost model (generations as the unit of compute) implicitly assumes a throughput-optimized setting where many queries are processed simultaneously and the total FLOPs budget is the binding constraint — batch inference, data generation, or offline evaluation. This is a reasonable setting for some use cases (and is explicitly the setting of the MATH benchmark evaluation), but it is not universal, and the paper does not caution readers that the recommended strategies may be unsuitable for latency-constrained deployments.

Mitigation status. Not addressed at all. The paper does not mention latency as a consideration, does not report generation times, and does not test latency-constrained variants of the compute-optimal policy (e.g., maximizing accuracy subject to a maximum sequential depth). This is a significant omission because it means the paper's practical recommendations — "use sequential revisions on easy problems" — cannot be applied in many real-world settings without modification. A latency-aware extension of the compute-optimal framework would treat sequential depth as a cost multiplier, which would likely shift the optimal policy toward more parallel strategies, potentially reducing or eliminating the reported gains from sequential revisions.

6.6 Difficulty Bins Are Static, Coarse, and Depend on a Fixed Base Model

The assumption or constraint. The compute-optimal framework bins prompts into five discrete difficulty quintiles based on the base model's pass@1 rate (oracle) or PRM score distribution (predicted), estimated once from 2,048 samples. These bins are treated as fixed categories: a question in bin 3 receives the same strategy regardless of where it falls within the bin (near the bin 2 boundary or near the bin 4 boundary). Additionally, the difficulty bins are specific to the base model — PaLM 2-S* in all experiments — and would change for a different model with different capabilities.

The consequence. The coarse discretization means that questions at opposite ends of a difficulty bin receive identical treatment despite potentially benefiting from different strategies. For instance, a question at the hard end of bin 3 (medium difficulty) might benefit more from balanced sequential-parallel allocation, while a question at the easy end might perform better with purely sequential revisions, but both receive the same strategy because they share a bin. This intra-bin misallocation reduces the efficiency of the compute-optimal policy relative to a continuous or finer-grained difficulty estimate. More importantly, the difficulty bins provide no mechanism for dynamic adjustment during computation. The strategy is chosen once based on a pre-computed difficulty estimate and then executed in full. An adaptive policy could start with a few parallel samples, assess the score distribution, and then decide in real-time whether to switch to beam search, continue with revisions, or allocate the remaining budget differently — potentially achieving better results than any static strategy. The paper's binning approach represents a coarse approximation of such an adaptive policy, and the gap between the binning approach and a truly dynamic policy is unknown.

What evidence exists in the paper. The paper's binning methodology is described in Section 3.2 (companion paper): 2,048 samples per question, five quintiles based on pass@1 or PRM score, with two-fold cross-validation within each bin for strategy selection. The bin boundaries are determined by the pass@1 distribution of the specific test set (500 MATH questions), meaning the bins are dataset-dependent as well as model-dependent. The paper does not test sensitivity to the number of bins (e.g., 3 bins vs. 5 bins vs. 10 bins), does not evaluate continuous difficulty-conditioned policies, and does not explore dynamic mid-computation strategy switching. The "predicted" difficulty bins use the PRM's score distribution rather than ground-truth pass@1, but still require the same 2,048-sample generation budget and produce the same discrete binning.

Mitigation status. The paper acknowledges the exploration-exploitation trade-off in difficulty estimation (Section 3.2 of the companion paper) but treats it primarily as a cost issue (how to estimate difficulty more cheaply) rather than a representational issue (how to use a continuous or dynamic difficulty estimate). The paper does not propose or test adaptive policies, finer-grained binning, or model-agnostic difficulty estimation. This is a reasonable scope limitation for a first study establishing the existence of difficulty-dependent optimal strategies, but it means the reported results should be interpreted as a lower bound on what a more sophisticated (continuous, dynamic) allocation policy could achieve. The practical advice — "use five bins based on PRM score distribution" — is specific to this experimental setup and may not be optimal in other settings with different base models, different problem distributions, or different bin counts.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper transforms code data in pre-training from an anecdotal best practice into an empirically characterized design space. Before this work, the field's approach to code was essentially binary and faith-based: practitioners included code because other practitioners included code, and the fraction crept upward (from Llama 2's 4.5% to Llama 3's 17%) without rigorous justification for why a specific proportion was chosen or what mechanisms drove the observed improvements. The paper demolishes this heuristic by demonstrating that code's benefits are not uniform — they depend on code proportion (NL reasoning peaks at 25%, world knowledge degrades monotonically, code improves linearly; Figure 4), code quality (10% synthetic verified code produces larger gains than 20% markup or 15% code-adjacent data; Figure 5a), and training stage (code serves distinct functions at initialization, during continual pre-training, and during cooldown; Figures 2, 6, and Table 2).

The magnitude of this shift is best characterized as a reframing with practical teeth — not a paradigm shift (the idea that code helps reasoning was already in the air), but a conversion of vague intuition into a controllable engineering discipline. The paper's key conceptual move is establishing that code is a general-purpose representation learning tool, not merely a domain-specific dataset for code generation. The evidence for this reframing is the initialization experiments (Section 3.1, Figure 2): a model trained on 200B tokens of pure code then 200B tokens of text outperforms a model trained on 400B tokens of text-only on natural language reasoning by 8.8% relative. The code-only phase contains no NL reasoning tasks — yet it builds representations that make subsequent text training more efficient than additional text training would have been. This is the signature of a transfer effect, not task-specific skill acquisition.

The paper also resolves a latent tension in the literature that had not been explicitly articulated. Prior work had produced a fragmented picture: Muennighoff et al. (2023a) showed code helps in data-constrained regimes on perplexity; Madaan et al. (2022b) showed code-trained models perform better on commonsense reasoning; Kim et al. (2024) showed code improves entity tracking; and various papers on mathematical reasoning (Liang et al., 2022; Shao et al., 2024) reported code-related gains. But these findings existed in isolation, with no framework for understanding whether they reflected a common mechanism or unrelated phenomena. This paper doesn't resolve the mechanistic question — it doesn't explain how code transfers to reasoning — but it establishes that the transfer is broad-spectrum (affecting NL reasoning, world knowledge, code generation, and generative quality simultaneously) and controllable through independent levers. This unification makes the prior piecemeal findings coherent: they are all manifestations of the same underlying representation-learning benefit, modulated by whether the code data was high-quality, appropriately proportioned, and present at the right training stages.

The paper shifts which research directions become attractive or unattractive:

More attractive:

Code quality curation as a primary research investment. The finding that 3.2B tokens of verified synthetic code produces larger gains than 139B tokens of web-code (Figure 5a) implies that the bottleneck is correctness, not volume. Research on formal verification of training data, automated code correctness checking, and synthetic code generation with correctness guarantees becomes directly relevant to pre-training data strategy, not merely a niche concern for formal methods researchers.
Stage-specific data optimization. The demonstration that code serves distinct functions at initialization, continual pre-training, and cooldown (Sections 3.1, 3.4, 3.5) implies that the optimal data mixture is not a single static recipe but a function of training phase. This opens a research direction on dynamic data scheduling — learning curricula that adjust data composition as training progresses — which is both more complex and potentially higher-impact than the static mixture tuning that dominates current practice.
Synthetic data with correctness properties. The outsized impact of verified synthetic code (9% NL reasoning gain from 10% replacement) suggests that synthetic data generation pipelines that prioritize correctness (not just diversity or volume) may be one of the highest-leverage investments for improving model reasoning. This reframes synthetic data research from "generate more data" to "generate provably correct data."
Generative quality evaluation in pre-training ablations. The paper's use of LLM-as-a-judge win-rates to evaluate pre-training data decisions (Figures 7, 8, and Appendix C.1) establishes a methodological precedent. Prior pre-training ablation studies typically evaluated only on discriminative benchmarks; this paper demonstrates that code affects generative quality too, and that these effects are separable from benchmark effects. Future pre-training data studies that omit generative evaluation risk missing important effects.

Less attractive:

Naïve code proportion scaling. The paper's finding that NL reasoning peaks at 25% code and degrades at 100% code (Figure 4) undermines the implicit assumption that "more code is better" which had been driving the trend toward higher code fractions (Llama 2's 4.5% → Llama 3's 17%). Simply increasing code proportion without attention to quality and training stage may be wasteful or counterproductive for general-purpose models.
Treating code as a monolithic data category. The markup, code-adjacent, and synthetic ablations (Figure 5a) demonstrate that different code properties produce qualitatively different transfer patterns — markup helps NL reasoning but hurts code; code-adjacent helps reasoning but hurts code; only synthetic helps both. Research that treats "code" as an undifferentiated category and asks "does code help?" is asking an ill-posed question. The relevant question is "which properties of code help which capabilities at which training stages?"
Pre-training data research that neglects training stage dynamics. The paper's finding that code in cooldown provides distinct benefits from code in main pre-training (Figure 6: cooldown without code gives no NL reasoning improvement, cooldown with code gives +3.6%) means that studying pre-training data mixtures without controlling for training stage conflates qualitatively different effects. The training stage is not merely a nuisance variable — it determines what function the data serves.

Follow-Up Research This Work Enables

1. Open-source replication of the synthetic code quality effect with publicly available verified code datasets. The paper's strongest result — 9% NL reasoning and 44.9% code improvement from 10% synthetic verified code (Figure 5a) — is supported by a proprietary dataset that cannot be inspected or replicated. A high-priority follow-up would substitute publicly available high-quality code sources to determine whether the effect generalizes beyond Cohere's internal pipeline. Concrete candidates: (a) code from competitive programming platforms (Codeforces, AtCoder) where solutions are known to pass test suites — this provides a weaker but still meaningful correctness signal; (b) a filtered subset of The Stack where only repositories with passing CI/CD pipelines are retained, creating a binary "builds successfully" quality threshold; (c) the APPS dataset (Hendrycks et al., 2021, different from the MATH benchmark), which contains programming problems with test cases — using only solutions that pass all tests as "verified" code. A successful replication would demonstrate the effect at, say, 5–7% NL reasoning improvement and 30–40% code improvement at 10% replacement, establishing a reproducibility floor. A null result (no improvement from any public verified dataset) would suggest the proprietary synthetic data's benefits come from unstated properties beyond verification (e.g., uniform coding style, consistent problem difficulty, absence of certain anti-patterns) and would substantially weaken the paper's claim that correctness per se is the critical quality dimension.

2. Dose-response curve for synthetic code quality: at what proportion does the effect saturate, and does it ever become harmful? The paper tests exactly one replacement ratio: 10% synthetic code, 90% web-code (Section 3.4, Figure 5a). This leaves a fundamental question unanswered: if 10% synthetic produces 9% NL reasoning gain (relative to 0% synthetic), would 20% synthetic produce 18% gain? 50% synthetic produce a larger gain? Or does the effect saturate — perhaps 10% already captures most of the benefit — or even reverse at high proportions because synthetic code lacks the diversity of web-code? A follow-up experiment would train models at synthetic replacement ratios of 0%, 5%, 10%, 20%, 50%, and 100% (all at the same total code token budget, 200B tokens, matching the paper's code-only pre-training setting), measuring NL reasoning, code performance, and world knowledge. If the dose-response curve is concave (diminishing returns), the practical implication is that a small investment in verified code goes a long way, and organizations should prioritize obtaining a modest amount of high-quality code rather than maximizing its proportion. If the curve is sigmoidal (threshold effect), there may be a minimum proportion below which verified code provides negligible benefit. If the curve eventually declines (too much synthetic code hurts), it would suggest synthetic code lacks some property of real-world code diversity that matters for generalization — a finding that would complicate the "correctness is all you need" hypothesis.

3. Mechanistic probing: do code-trained models develop better compositional representations, systematic syntactic generalization, or long-range dependency tracking? The paper demonstrates that code improves NL reasoning, world knowledge, and generative quality, but provides no evidence about how. A natural follow-up would apply representation-level probing techniques to the code-initialized models (code→text, balanced→text) versus the text-only baseline at checkpoints throughout training to understand what changes in the model's internal representations. Specific probes: (a) Compositional generalization: test whether code-trained models better handle novel compositions of known linguistic structures using benchmarks like COGS (Kim & Linzen, 2020) or SCAN (Lake & Baroni, 2018), which measure systematic generalization to unseen compositional patterns. The hypothesis: code's compositional syntax (function composition, nested expressions, variable scoping) teaches the model to represent hierarchical structure more explicitly, which transfers to compositional language understanding. (b) Long-range dependency tracking: measure attention entropy and effective context length on synthetic tasks requiring tracking of entities across long distances, testing the hypothesis that code's long-range dependencies (function calls across files, variable references across scopes) improve the model's ability to maintain coherent representations over long contexts. (c) Abstract reasoning: evaluate on benchmarks like ARC (Abstraction and Reasoning Corpus, Chollet, 2019) or synthetic Raven's Progressive Matrices tasks, testing whether code exposure improves the model's ability to perform abstract pattern recognition independent of natural language content. A positive result on any of these probes would provide mechanistic grounding for the paper's empirical findings; a null result across all probes would suggest code's benefits operate through a different mechanism (e.g., improved optimization dynamics, better calibration, or reduced surface-form overfitting) that would require different experimental tools to detect.

4. Dynamic data scheduling: can a learned curriculum that adjusts code proportion by training stage outperform the static recipes in this paper? The paper's key insight about training stage (code serves distinct functions at initialization, continual pre-training, and cooldown) is derived from comparisons between separately trained models with different stage-specific recipes, not from a single model whose data mixture changes dynamically during training. A follow-up would implement a dynamic data scheduler that adjusts the code proportion as a function of training progress, potentially using a simple heuristic (e.g., start at 50% code to build representations, decay to 10% code over the main pre-training phase to transfer to text, spike to 20% high-quality code during cooldown) or a learned policy (e.g., using a small proxy model to estimate the gradient of downstream task performance with respect to data mixture at each training step). The evaluation would compare a dynamically scheduled model against the paper's best static recipe (balanced→text with synthetic code and cooldown) at matched total token budgets, measuring NL reasoning, world knowledge, code, and generative win-rates. A positive result — dynamic scheduling outperforms the best static recipe — would establish dynamic data composition as a new axis for pre-training optimization, expanding the design space beyond the fixed recipes that current practice and this paper both assume. A null result — dynamic scheduling performs no better than the best static recipe — would simplify the practitioner's decision problem (choose one good static mixture and stick with it) but would be surprising given the paper's evidence that optimal code proportion varies by training stage.

5. Does code in cooldown scale with model size, and is there an optimal cooldown code fraction? The paper's cooldown experiments (Section 3.5, Figures 6–7) are conducted exclusively at 470M parameters with a single cooldown code fraction (20%) and a single cooldown duration (40B tokens, 10% of the total budget). Three critical follow-up questions remain: (a) Does the cooldown benefit scale with model size? The paper validates that initialization trends transfer from 470M to 2.8B (Section 3.2, Figure 3 [paper's Figure 9]), but does not replicate cooldown at 2.8B. Larger models may benefit more from cooldown (because they have more capacity to absorb the high-quality data signal) or less (because they have already extracted more value from the main pre-training). A follow-up would run cooldown with and without code at 2.8B, matching the 470M cooldown protocol (40B tokens, 20% code, linear annealing to 1e−6), and compare relative improvements over no-cooldown baselines at both scales. (b) What is the optimal cooldown code fraction? The paper uses 20% based on the rationale that code is up-weighted during cooldown, but does not sweep this fraction. A follow-up would test cooldown code fractions of 0%, 10%, 20%, 30%, and 50% (all at the same total token budget) on the balanced→text base model, measuring which yields the best NL reasoning and world knowledge. (c) Does the cooldown benefit depend on the cooldown duration? The paper uses 40B tokens (10% of total budget) without ablating this choice. A follow-up would test cooldown durations of 5%, 10%, 20%, and 30% of the total token budget, mapping the efficiency frontier: how much cooldown is needed to capture most of the benefit, and at what point does additional cooldown produce diminishing returns?

6. Interaction between code quality and model scale: does the synthetic code advantage grow, shrink, or remain constant as models get larger? The paper demonstrates that synthetic code's benefits transfer from code-only pre-training to continual pre-training (Figure 5a → Figure 5b) and that initialization trends transfer from 470M to 2.8B (Section 3.2), but it does not test the interaction between code quality and scale: does the advantage of synthetic code over web-code increase with model size (because larger models can better exploit the correctness signal) or decrease (because larger models can extract more value from noisy web-code, reducing the quality gap)? A follow-up would compare balanced→text with and without synthetic code at both 470M and 2.8B scales. If the synthetic code advantage grows with scale — e.g., 2% NL reasoning gain at 470M but 6% gain at 2.8B — this would suggest that code quality becomes more important at larger scales, making synthetic code generation a critical investment for frontier model training. If the advantage shrinks with scale — e.g., 2% gain at 470M but 1% gain at 2.8B — this would suggest that larger models are more robust to noisy training data, and quality-focused curation matters most at smaller scales. This interaction is directly relevant to organizations deciding where to invest their data curation budget as they scale up model size.

Practical Applications and Downstream Use Cases

1. Pre-training data budget allocation for general-purpose LLMs. Organizations developing general-purpose language models (not specifically code-focused) can use the paper's findings to allocate their data curation budget more efficiently. The key numbers: NL reasoning peaks at 25% code proportion (Figure 4) — quadrupling code from this level to 100% yields only linear code improvement at the cost of 18.3% relative reasoning degradation and 86.1% world knowledge collapse. The practical implication is that teams should target roughly 25% code in their pre-training mixture, invest disproportionately in producing or curating high-quality verified code within that budget (given that 10% synthetic code replacement yields 9% NL reasoning gain, Figure 5a), and include code in their cooldown phase (given that cooldown with code improves NL reasoning by 3.6% and generative win-rates by 4.1% over cooldown without code, Figures 6–7). Concretely, a team with a 10T token pre-training budget might allocate ~2.5T tokens to code, with ~250B of those tokens being high-quality synthetic code (10% of the code budget), and an additional ~200B tokens of code in a cooldown phase at the end of training. The paper's results suggest this allocation would substantially outperform both a text-only model and a model with a higher code fraction but lower-quality code.

2. Code model pre-training as a general-purpose initialization backbone. The finding that a model trained on 200B tokens of pure code then 200B tokens of text outperforms 400B tokens of text-only on NL reasoning (+8.8%, Figure 2) has direct implications for training pipeline architecture. Organizations can develop specialized code-LM "backbones" — large models trained primarily on high-quality code — and then offer these as initialization checkpoints for downstream text-focused training. This is more efficient than training general-purpose models from scratch because the code-LM backbone does the "heavy lifting" of learning structured, compositional representations, and the text-focused phase can be shorter and more targeted. A concrete deployment: train a 7B-parameter code-LM on 500B tokens of code (with 10% synthetic verified code), release it as an open-source initialization, and enable other teams to continue training on their domain-specific text data for only 100–200B additional tokens to achieve NL reasoning performance equivalent to a 7B model trained on 1T tokens of mixed text-code data from scratch. This saves the downstream team compute and allows them to specialize the model to their text domain without needing to manage the code-text mixture themselves.

3. Code quality investment for reasoning-focused applications. For applications where reasoning capability is the primary performance metric — automated theorem proving, mathematical problem-solving, logical inference engines, code-assisted reasoning tools — the paper's finding about synthetic verified code has direct procurement implications. Generating or licensing 3.2B tokens of formally verified code (the size of the synthetic dataset used in this paper) is substantially cheaper than scraping and filtering 139B tokens of web-code, yet produces larger reasoning improvements (+9% NL reasoning from the synthetic addition vs. baseline web-code, Figure 5a). A small organization developing a reasoning-focused LLM could invest in generating verified code data (via formal verification tools, automated program synthesis with correctness checking, or licensing from companies that produce verified software) rather than building large-scale web scraping infrastructure. The 3.2B token figure provides a concrete target: a dataset of this size is achievable with modest resources (it represents roughly 50,000–100,000 verified Python programs of ~300–600 tokens each). The expected return is a ~9% improvement in reasoning benchmark performance and a ~45% improvement in code generation — gains that would require a much larger investment in web-code volume or model scale to achieve through other means.

4. Cooldown phase as a standard component of pre-training pipelines with code. The paper provides evidence that cooldown with code improves performance across all measured dimensions — NL reasoning (+3.6%), world knowledge (+10.1%), code (+20%), and generative win-rates (+4.1% over cooldown without code) relative to the model before cooldown (Figure 6). This establishes cooldown as a high-leverage, low-cost intervention: at only 10% of the total token budget (40B out of 440B), cooldown provides disproportionate gains. For any team currently pre-training models without a cooldown phase, adding one — with the paper's recommended mixture of high-quality text, math, code (20%), and instruct-style data, using linear learning rate annealing to 1e−6 — is a straightforward engineering change with predictable benefits. The 20% code fraction and 40B token budget can be used as starting defaults, with the understanding that these specific numbers come from the paper's 470M-parameter, 400B-total-token setting and may need adjustment for substantially different scales. The generative win-rate improvement (52.3% win-rate for cooldown with code vs. no cooldown, Figure 7) is particularly actionable for user-facing products where instruction-following and response quality matter more than benchmark scores.