ArXiv: 2512.07783

🎯 Pitch

RL produces true reasoning gains—up to +42% pass@128—only when pre-training leaves headroom and RL targets the model’s edge of competence; it cannot synthesize capabilities from nothing. Adding a mid-training stage yields +10.8% further OOD improvement under fixed compute, revealing that mid-training, not just RL, is a critical and underexplored lever for reasoning generalization.

1. Executive Summary

This work systematically studies the interplay between pre-training, mid-training, and RL-based post-training for reasoning language models using a fully controlled synthetic reasoning framework built on GSM-Infinite with 100M-parameter Qwen2.5-style models. The paper analyzes two generalization axes—extrapolative generalization (composing known primitives into deeper reasoning chains with more operations than seen during pre-training) and contextual generalization (transferring reasoning skills across novel surface templates that share underlying dependency-graph structure)—and identifies the conditions under which RL produces genuine capability improvements beyond the base model. The central finding is that RL yields true pass@128 gains—up to +42% on OOD-hard tasks and +60% on contextual transfer—only when pre-training leaves sufficient headroom for exploration and when RL data target the model's edge of competence (tasks at the boundary where pass@1 fails but pass@k succeeds). Introducing a mid-training stage bridging pre-training and RL distributions further improves OOD-hard performance by +10.8% under fixed compute, establishing that mid-training is a critical yet underexplored lever whose effectiveness is bounded by the presence of atomic reasoning primitives seeded during pre-training—RL cannot synthesize capabilities from a void but can robustly amplify them once minimal pre-training exposure (≥1%) provides a sufficient foundation.

2. Context and Motivation

The Core Problem: We Don't Know How Pre-Training and Post-Training Jointly Produce Reasoning

The fundamental question driving this paper is one of the most consequential yet poorly understood aspects of modern language model training: does reinforcement learning (RL) genuinely extend a model's reasoning capabilities beyond what it acquired during pre-training, or does it merely refine and surface skills that were already latent in the base model? This is not a rhetorical or philosophical question—it has direct practical implications for how the field should allocate billion-dollar compute budgets between scaling pre-training data, designing mid-training curricula, and applying RL fine-tuning.

The paper frames this as an interaction question rather than a question about any single training phase in isolation. Contemporary training pipelines for state-of-the-art reasoning models—think DeepSeek-R1 [DeepSeek-AI et al., 2025] or OpenAI's o1 [OpenAI et al., 2024]—involve three distinct stages: (1) pre-training on massive web-scale corpora, (2) an intermediate mid-training phase (also called continued pre-training) that bridges the distributional gap between raw web text and the specialized reasoning formats used downstream, and (3) RL-based post-training that optimizes against reward signals derived from verifiable correctness. Each of these stages contributes to the model's final reasoning competence, but their contributions are deeply entangled. The web-scale pre-training data that modern models are trained on is opaque—we fundamentally do not know what reasoning patterns, problem structures, or solution strategies the base model has already internalized. Consequently, when an RL-trained model solves a novel math problem, we cannot determine whether RL taught it to compose new reasoning strategies or simply amplified a nascent skill that was already present after pre-training.

This ambiguity matters enormously because it determines what we should expect from scaling different phases. If RL primarily surfaces and amplifies existing pre-training knowledge, then the returns to additional pre-training data scale with the diversity of that data, while RL serves as a relatively cheap "extractor" of latent competence. If, alternatively, RL genuinely enables novel compositional generalization—teaching models to combine atomic skills acquired during pre-training into qualitatively new reasoning patterns—then RL is a first-class contributor to capability and deserves its own scaling investment. Without controlled experiments that disentangle these effects, the field is navigating one of its most expensive design choices by intuition and folklore rather than principle.

Why This Problem Is Urgent: Competing Empirical Narratives and the Stakes Involved

The paper is motivated by a direct conflict in the recent literature that makes the ambiguity around RL's contribution particularly acute. Several high-profile works have reached seemingly contradictory conclusions about whether RL improves reasoning:

The "RL does not extend reasoning" perspective. Yue et al. (2025) present evidence that while RL-trained models outperform base models at small sample budgets (e.g., pass@1), the advantage largely disappears at higher sample budgets (e.g., pass@128). Their coverage and perplexity analyses suggest that the reasoning capabilities of RL-trained models remain fundamentally constrained by the base model's representational capacity—RL, in this view, makes the model more efficient at accessing existing knowledge but does not expand what it can ultimately reason about. Wu et al. (2025) provide complementary theoretical arguments, formalizing conditions under which RL cannot surpass the base model's inherent limitations. Shao et al. (2025) further demonstrates that models can exploit "spurious rewards"—achieving correct final answers through invalid reasoning chains—which inflates apparent RL gains without representing genuine reasoning improvement. Zhao et al. (2025) argues that RL post-training primarily amplifies behaviors already learned during pre-training, characterizing it as an "echo chamber" effect.

The "RL enables novel generalization" perspective. On the other side, multiple controlled studies report substantial reasoning gains from RL that go well beyond what the base model can achieve. Wen et al. (2025) shows that RL with verifiable rewards implicitly incentivizes correct reasoning in base models by shaping the distribution of intermediate steps. Yuan et al. (2025) designs a synthetic function composition task and demonstrates that RL-trained models can generalize to unseen compositions—effectively computing f(g(x))f(g(x))—that the base model completely fails on, representing a clear case of RL enabling what the authors call "new skill composition." Sun et al. (2025a) presents evidence of "RL grokking"—a phenomenon where RL unlocks and transfers new algorithmic reasoning strategies for complex problem families that the base model shows no sign of understanding. Liu et al. (2025b) reports that Prolonged RL (ProRL) expands reasoning boundaries on synthesized tasks well beyond base model limitations. Yeo et al. (2025) demystifies long chain-of-thought reasoning in the context of RL-trained models, showing emergent structured reasoning behaviors.

These two perspectives are not minor methodological disagreements—they lead to fundamentally different investment strategies. If RL is primarily an "amplifier," the rational strategy is to invest heavily in broader and higher-quality pre-training data while treating RL as a relatively minor optimization step. If RL is a "composer" that genuinely extends reasoning, then significant budget should flow toward RL infrastructure, reward design, and exploration algorithms.

The Missing Ingredient: Controlled Isolation of Training-Phase Contributions

The paper identifies a single root cause for this empirical conflict: none of the prior studies can causally attribute reasoning gains to specific training phases because they lack controlled training environments. Modern LLMs are pre-trained on massive, opaque internet corpora whose composition is fundamentally unknown to researchers (even the training teams). When a model solves a novel reasoning problem after RL fine-tuning, there are at least three possible explanations, all observationally equivalent in an uncontrolled setting:

  1. The reasoning strategy was acquired during pre-training from some unknown document in the web corpus, and RL merely made the model more efficient at retrieving and applying it (the "amplification" hypothesis).
  2. The reasoning strategy was partially seeded during pre-training (the model learned atomic operations and basic composition patterns), and RL extended this to novel, deeper compositions (the "compositional generalization" hypothesis).
  3. The reasoning strategy was genuinely learned during RL through reward-driven exploration of the strategy space, without requiring pre-training exposure to similar compositions (the "de novo discovery" hypothesis).

Without control over pre-training data—specifically, without the ability to guarantee that certain reasoning patterns were absent from the pre-training corpus—it is impossible to distinguish between these mechanisms. The paper argues that this lack of control is the fundamental reason for the conflicting narratives in the literature: different studies are accidentally (and unknowingly) testing different hypotheses because their base models have different (and unknown) pre-training data coverage.

Where Prior Approaches Fall Short

The paper identifies specific limitations across several lines of prior work:

Uncontrolled pre-training backgrounds. Studies that use open-weight base models (e.g., LLaMA, Qwen) as starting points for RL experiments cannot know what reasoning skills those models already possess. Even the model creators typically do not have full knowledge of their training data composition at the granularity needed to determine whether a specific reasoning primitive (e.g., "solving for an unknown through equation setup") was covered in pre-training. This makes it impossible to determine whether observed RL gains represent new skill acquisition or existing skill amplification. The paper explicitly contrasts this with their approach: "As a result, we cannot ascertain which reasoning primitives the base model has already internalized. Consequently, this lack of control makes it challenging to isolate the causal effect of post-training and to understand how pre-training and post-training jointly shape reasoning behavior" (Section 1).

Single-axis evaluation of generalization. Most prior work evaluates reasoning generalization along a single dimension—typically depth (can the model solve harder problems?) or breadth (can the model transfer to new domains?)—but not both simultaneously. The paper argues that extrapolative and contextual generalization are complementary axes that probe different aspects of reasoning competence, and that the interplay between pre-training and post-training may differ along these axes. For example, a model might generalize depth-wise on familiar contexts while failing to transfer even simple reasoning to novel narrative settings, or vice versa.

Absence of process-level verification in training and evaluation. Many prior studies evaluate models solely on final-answer correctness, which the paper argues is vulnerable to reward hacking—models can produce correct answers through invalid or incomplete reasoning chains. Without verification of intermediate reasoning steps, apparent RL-driven improvements may reflect the model learning to exploit shortcuts rather than developing genuine reasoning competence. The paper notes that "Shao et al. (2025) further demonstrates that models can exploit 'spurious rewards'—achieving correct final answers through invalid reasoning chains—which inflates apparent RL gains without representing genuine reasoning improvement."

Mid-training as an underexamined confounding factor. Several recent works have identified mid-training as a crucial intermediate phase that substantially impacts downstream RL performance [Wang et al., 2025, Liu et al., 2025a, Akter et al., 2025], but its role has not been systematically studied in controlled settings. The paper observes that mid-training may explain why some base models respond dramatically better to RL than others: models that undergo reasoning-oriented mid-training effectively receive a "distributional bridge" that aligns their internal representations with RL objectives. Without controlling for mid-training, differences in RL effectiveness across base models might actually reflect differences in their mid-training (or lack thereof) rather than differences in their pre-training coverage. Wang et al. (2025) finds that LLaMA models mid-trained on structured reasoning data achieve RL performance comparable to much stronger Qwen base models, suggesting that mid-training can substitute for, or complement, pre-training coverage.

Controlled synthetic approaches that focus narrowly on RL. Several prior works have used controlled synthetic tasks to study RL generalization [Yuan et al., 2025, Liu et al., 2025b, Sun et al., 2025a], but these studies typically focus on the RL phase in isolation—they design tasks specifically for RL evaluation without systematically manipulating pre-training exposure or studying mid-training. The paper argues that this leaves the pre-training/RL interaction unexplored, which is precisely the dimension that can reconcile conflicting findings in the literature.

How This Paper Positions Itself

The paper positions itself at the intersection of these prior lines of work, aiming to provide the first controlled framework that isolates the causal contributions of all three training stages simultaneously. Its approach is explicitly motivated by prior controlled experiments in language model analysis—particularly the "Physics of Language Models" series [Allen-Zhu, 2024, Ye et al., 2024] and the GSM-Infinite framework [Zhou et al., 2025a]—but extends them to systematically study pre-training, mid-training, and RL interactions.

The paper's key positioning claims are:

Controllability as the central methodological contribution. Rather than analyzing existing large-scale models (which have opaque training), the paper constructs a fully controlled testbed where every aspect of the training distribution—operation counts, template exposure, primitive coverage—can be precisely specified and systematically manipulated. This enables the kind of causal attribution that is impossible in uncontrolled settings.

Reconciliation of conflicting empirical results. The paper does not take sides in the "does RL work?" debate. Instead, it proposes that the contradictory findings are both correct under different (implicit) conditions, and that the key variable is whether pre-training left sufficient headroom at the model's edge of competence. By systematically varying this variable, the paper shows precisely when each prior finding holds and when it breaks down.

Mid-training as a first-class contributor. While mid-training has been acknowledged in recent pipeline descriptions, the paper elevates it to a central object of study—arguing that understanding its role is essential for explaining why some models respond better to RL than others, and for designing optimal training budgets that balance mid-training and RL allocation.

Process verification as both a training and evaluation tool. By building process verification into both the RL reward function and the evaluation protocol, the paper addresses the reward hacking critique directly. This dual use of process verification—as a training signal to shape reasoning fidelity and as an evaluation metric to prevent inflated performance estimates—provides a cleaner signal of genuine reasoning improvement.

Practical guidance as a motivating goal. Throughout the paper, the authors frame their findings in terms of actionable guidance for practitioners: how to select RL data that targets the model's edge of competence, how to seed pre-training with long-tail primitives to unlock contextual transfer, and how to balance mid-training and RL compute budgets for different generalization objectives. This practical orientation distinguishes the work from purely theoretical analyses and reflects the authors' stated goal of providing "a foundation for understanding and improving reasoning LM training strategies."

3. Technical Approach

3.1 Reader Orientation

This paper constructs a fully controlled experimental testbed—not a new model or algorithm—that systematically manipulates pre-training, mid-training, and RL post-training data distributions to isolate their causal effects on reasoning generalization. The core problem it solves is the identification problem in modern LM training: when a reasoning model improves after RL, we cannot tell whether RL taught it something new or merely surfaced pre-existing knowledge because pre-training data is opaque. The solution takes the form of a synthetic reasoning data generator that produces problems with explicit dependency-graph structure, combined with a training protocol where the operation-count ranges and template exposures in each phase are precisely specified, enabling causal attribution of generalization gains to specific training stages.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, organized into data generation and training/evaluation pipelines:

  1. Dependency Graph Generator — produces directed acyclic graphs (DAGs) where nodes represent variables and edges represent arithmetic operations (+, −, ×, ÷). The edge count op(G) = |E| precisely controls reasoning complexity. This is the abstract reasoning structure shared across all surface realizations.

  2. Contextual Renderer (Φ) — maps each DAG through a template τ (e.g., animals–zoo, teachers–school, movie-festival) to produce a complete natural-language math problem with a step-by-step solution trace. Different templates produce structurally identical but surface-different problems, enabling controlled study of contextual generalization.

  3. Training-Phase Data Partitioner — splits the generated corpus into disjoint distributions for pre-training (10B tokens, op=2-10 across multiple templates), mid-training (optional, op=11-14 "edge of competence" data), and RL post-training (200K samples from specified op ranges and template mixtures). No instance appears in more than one phase.

  4. 100M-Parameter Qwen2.5-Style Decoder-Only Model — trained from scratch through all three phases using the partitioned data, with GRPO for RL post-training. The small scale (100M parameters) enables systematic experimentation.

  5. Process-Verified Evaluation Pipeline — parses model-generated free-form solutions into predicted dependency graphs, compares them against ground-truth DAGs node-by-node, and requires both perfect step-level reasoning and correct final answers for a sample to count as correct. This enforces reasoning fidelity.

Information flows: the DAG generator samples operations and structure → the renderer produces (question, solution, answer) triples → the partitioner allocates triples to pre-training, mid-training, or post-training based on op counts and templates → the model is trained sequentially through these phases → at evaluation, the model generates free-form solutions that are parsed into predicted DAGs and verified against ground-truth DAGs.

3.3 Roadmap for the Deep Dive

  • First, the dependency graph formalism and complexity control mechanism, because it is the foundation on which all task difficulty manipulation and generalization measurement is built.
  • Second, the contextual rendering function and the separation between structure and surface form, because this is what enables the paper to study contextual generalization independently of extrapolative generalization.
  • Third, the training-phase data partitioning and the specific operation-count and template allocations for each phase, because these precise allocations are what enable causal attribution of reasoning gains to specific stages.
  • Fourth, the process-verified evaluation protocol—parsing, graph matching, and the strict correctness criterion—because it defines what "correct reasoning" means in this paper and how reward hacking is prevented.
  • Fifth, the RL training setup (GRPO) and the compute budget equivalence between mid-training and RL, because the budget comparison in Section 5 depends on this equivalence.
  • Sixth, the key design choices that distinguish this framework from alternatives and enable the paper's causal claims.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a controlled experimental methodology paper whose core idea is that by building a synthetic reasoning testbed where every aspect of the training distribution is known and manipulable, researchers can isolate the causal contributions of pre-training, mid-training, and RL post-training to reasoning generalization—something impossible in opaque, large-scale training pipelines.

Dependency Graph Formalism and Reasoning Complexity Control

Every reasoning problem in the paper's testbed is grounded in a directed acyclic graph (DAG):

G=(V,E)G = (V, E)

where $V$ is the set of nodes (each node $v_i \in V$ represents a latent quantity such as "number of adult lions" or "total schools in Westhaven City") and $E$ is the set of directed edges (each edge $(v_j \rightarrow v_i) \in E$ represents a functional dependency: node $v_i$ is computed from its parent nodes).

What this computes: This formalism represents the abstract reasoning structure of a math word problem. Each node is an intermediate quantity that must be computed; each edge encodes a "this quantity depends on that one" relationship. The DAG is acyclic because quantities are defined in terms of previously computed quantities—there are no circular definitions.

Why this form: DAGs are the natural representation for step-by-step reasoning because they make dependencies explicit and prevent circular reasoning. Unlike sequence representations (which flatten the reasoning into a linear order), DAGs capture the partial ordering of computation steps—some steps can be computed independently, others must wait for their dependencies to be resolved. This partial ordering is precisely what makes some reasoning problems harder than others (deeper chains require more sequential composition of primitives).

Each node's value is computed by an arithmetic operation from its parent set:

vi=fi({vj}j∈pa(i))v_i = f_i(\{v_j\}_{j \in \text{pa}(i)})

fi∈{+,−,×,÷}f_i \in \{+, -, \times, \div\}

where $\text{pa}(i)$ is the parent set of node $i$ (all nodes that have edges pointing to $v_i$), and $f_i$ is one of the four elementary arithmetic operations.

What this computes: Given numeric assignments to all leaf nodes (nodes with no incoming edges, representing "given" quantities in the problem statement), this recursive definition computes the value of every node in topological order. For a designated query node $v^*$ (typically the last node in the topological order or the distinguished unknown in equation-style problems), the ground-truth answer is $a^* = \text{val}(v^*)$.

Why this form: Restricting to four elementary arithmetic operations keeps the reasoning primitives interpretable and the graph structure tractable while still supporting complex compositional reasoning. The operations are all deterministic functions of their inputs, meaning the correct answer is uniquely determined by the graph structure and leaf values—there is exactly one correct reasoning path through the DAG.

The paper quantifies reasoning complexity as the number of arithmetic operations:

op(G)=∣E∣\text{op}(G) = |E|

What this computes: the total number of edges in the dependency graph, which equals the number of arithmetic operations needed to compute the final answer from the leaf values. Each edge corresponds to one "combine two previously-computed quantities into a new quantity" step.

Why this form: The edge count directly measures the minimum length of the compositional reasoning chain needed to solve the problem. A problem with $\text{op}(G) = 2$ requires only two arithmetic steps (e.g., "lions = 3, elephants = lions + 2, total = lions + elephants"), while a problem with $\text{op}(G) = 20$ requires composing twenty sequential operations. This single integer parameter provides a clean, continuous knob for controlling task difficulty and defines the in-distribution vs. out-of-distribution boundary for extrapolative generalization experiments. The paper groups problems into three difficulty categories based on operation counts:

  • In-Distribution (ID): $\text{op}(G) \in [2, 10]$ — problems within the pre-training distribution range.
  • OOD-edge: $\text{op}(G) \in [11, 14]$ — problems just beyond the pre-training range where the base model retains non-zero pass@128 accuracy. This is the model's "edge of competence."
  • OOD-hard: $\text{op}(G) \in [15, 20]$ — problems substantially beyond the pre-training distribution where the base model exhibits near-zero accuracy.

Implicit vs. explicit reasoning. The paper introduces a crucial distinction: not all edges in the DAG need to be explicitly verbalized in the natural-language problem. The edge set can be partitioned as:

E=Eexplicit∪Eimplicit,Eexplicit∩Eimplicit=∅E = E_{\text{explicit}} \cup E_{\text{implicit}}, \quad E_{\text{explicit}} \cap E_{\text{implicit}} = \emptyset

where $E_{\text{explicit}}$ contains dependencies directly stated in the problem text (e.g., "there are 5 more elephants than lions") and $E_{\text{implicit}}$ contains dependencies that are part of the ground-truth reasoning graph but never directly verbalized (e.g., "total animals equals lions plus elephants"). This separation is critical because it means the model must recover unspoken dependencies during reasoning—the problem statement does not contain the complete solution recipe. A model that can exploit $E_{\text{implicit}}$ correctly is demonstrating genuine reasoning rather than pattern matching against explicitly stated relationships.

Abstract and Instance Parameters: Separating Structure from Numeric Instantiation

The paper's framework separates each reasoning problem into two levels of specification, following the abstraction mechanism from GSM-Infinite:

Abstract parameters define the symbolic graph structure independent of particular numbers or entities. They specify:

  • Which variables exist and how they decompose (e.g., that "total animals" decomposes into "lions" and "elephants").
  • The edge set $E$ and the operation $f_i$ attached to each node.
  • The variable roles and the topological ordering.

These parameters define a purely symbolic DAG—a reasoning template.

Instance parameters fill this template with concrete values:

  • Numeric assignments to leaf nodes (e.g., "there are 12 adult lions and 7 elephant calves").
  • Bindings of variables to context-specific surface forms (e.g., "adult lions in the city zoo").

What this two-level design enables: The same abstract graph can be instantiated with different numeric values to produce a family of structurally identical problems that differ only in their concrete numbers. This is critical for controlled experimentation because it means:

  1. The paper can test whether the model has learned the reasoning structure (which transfers across numeric instantiations) or merely memorized specific numeric combinations.
  2. Training and evaluation splits can contain structurally different graphs to test extrapolation, while the numeric values for a given structure can vary to prevent memorization.

Why this form over generating problems from scratch: If problems were generated without this abstraction-instance separation, it would be impossible to guarantee that the model hasn't seen a structurally similar problem during pre-training. The explicit separation ensures that when the paper claims a problem is "out-of-distribution," it means the graph structure (the arrangement of operations and dependencies) is novel—not just the numbers.

Contextual Rendering: Mapping Dependency Graphs to Natural-Language Problems

The paper maps symbolic DAGs to human-readable math problems through a contextual rendering function:

Φ:(G,τ)↦x\Phi : (G, \tau) \mapsto x

where $G$ is the dependency graph, $\tau \in \mathcal{T}$ is a contextual template from a predefined set (e.g., animals–zoo, teachers–school, movie-festival), and $x$ is the resulting text instance.

What this computes: The rendering function takes the abstract reasoning structure and "dresses" it in a specific narrative context. It produces a triple:

x=([question],[solution],[answer])x = ([\text{question}], [\text{solution}], [\text{answer}])

where $[\text{question}]$ is the natural-language problem statement, $[\text{solution}]$ is a step-by-step derivation following the topological order of $G$, and $[\text{answer}]$ is the final numeric result.

Templates are context-specific mappings. A template $\tau$ specifies:

  • How abstract variables are lexicalized into domain-specific surface forms. For example, a node representing "a type of animal" might become "adult lions" in the animals–zoo template, "elementary school students" in the teachers–school template, or "tickets sold on day 1" in the movie-festival template.
  • Which subset of edges is realized explicitly in the problem wording vs. left implicit. This determines the split between $E_{\text{explicit}}$ and $E_{\text{implicit}}$, which can differ across templates for the same underlying graph.

Structural equivalence across templates. The paper defines a fundamental property of the rendering function: for any two templates $\tau_a$, $\tau_b$ that differ only in surface context, the induced problems remain structurally identical:

Struct(Φ(G,τa))=Struct(Φ(G,τb)),∀τa,τb∈T\text{Struct}(\Phi(G, \tau_a)) = \text{Struct}(\Phi(G, \tau_b)), \quad \forall \tau_a, \tau_b \in \mathcal{T}

Why this form matters: This property is the foundation of the paper's contextual generalization experiments. Because the same abstract graph can be rendered into semantically distinct yet structurally equivalent problems, the paper can train on graphs rendered in one context (say, animals–zoo) and test on the same graph structures rendered in a different context (say, teachers–school). If the model succeeds, it has learned transferable reasoning primitives rather than memorized task styles. If it fails, its reasoning is surface-bound.

The three specific templates used in the paper are:

  • Context A: animals–zoo (the dominant pre-training context, comprising 99–100% of most pre-training corpora).
  • Context B: teachers–school (a long-tailed context used to study contextual transfer).
  • Context C: movie-festival (used in some experiments as a third context for diversity).

Why three templates rather than two or many: Two templates would allow testing whether transfer occurs, but three templates (with two as targets) allows the paper to verify that transfer is not template-specific—that the model isn't learning "how to map zoo→school" specifically, but rather learning reasoning primitives that transfer to any novel surface form.

Data Generation Pipeline and Deduplication

The paper builds on the GSM-Infinite data generator with a stage-wise procedure:

Stage 1: Structural sampling. The generator samples structural parameters that define the dependency graph:

  • A target operation count range for $\text{op}(G)$ (e.g., 2–10 for pre-training, 11–14 for mid-training/edge RL).
  • Graph shape parameters controlling fan-in (how many parents each node can have), layering pattern (how nodes are arranged in topological levels), and overall depth.
  • Operation types $f_i \in \{+, -, \times, \div\}$ attached to each node, controlling whether the problem involves addition, subtraction, multiplication, division, or combinations thereof.

These choices determine a layered DAG $G$ with a unique query node $v^*$.

Why this complexity: The structural sampling is not merely random—it produces DAGs with specific topological properties that control reasoning difficulty independently of operation count. Two problems with $\text{op}(G) = 10$ can differ dramatically in difficulty depending on whether the operations are arranged in a single deep chain (requiring sequential reasoning) or a broad, shallow tree (allowing parallelizable computation). The paper's generator controls these structural knobs to ensure that "difficulty" (as measured by operation count) corresponds monotonically to actual reasoning complexity.

Stage 2: Abstract and instance parameterization. Given $G$, the generator samples abstract parameters (variable roles and decompositions) and instance parameters (numeric values on leaves), then evaluates all node values in topological order using the recursive valuation function defined earlier.

Stage 3: Contextual rendering. The generator chooses a template $\tau \in \mathcal{T}$ and applies $\Phi(G, \tau)$ to produce the natural-language triple.

Stage 4: Forward vs. reverse generation modes. Following GSM-Infinite, the generator supports two problem formulation modes:

  • Forward mode: The generator produces a standard arithmetic word problem where the final node in the topological order is queried. All leaf values are given, and the model must compute forward through the dependency chain.
  • Reverse mode: The generator treats one node as an unknown and phrases an equation-style problem where the model must solve for that quantity while the rest of the graph remains fully specified. This is analogous to algebra problems where "solve for x" replaces "compute the final value."

Deduplication and canonicalization. To guarantee cleanliness and avoid contamination across training and evaluation splits, the paper performs exact hash-based deduplication at the level of rendered triples:

  • Each instance is canonicalized by serializing the triple $([\text{question}], [\text{solution}], [\text{answer}])$ into a normalized string representation (stripping extraneous whitespace, normalizing numeric formatting).
  • This canonical form is hashed to obtain a global identifier.
  • Any duplicate hashes within or across splits are discarded, ensuring that no identical problem–solution triple appears in both training and evaluation.

Why this deduplication is critical in this setting: Unlike web-scale pre-training where exact duplicates are diluted by corpus size, synthetic data generation can accidentally produce identical problems (same structure, same template, same numbers) across different phases. Without deduplication, the paper's central claim—that post-training extends reasoning beyond pre-training coverage—could be invalidated by contamination: a model solving an "OOD" problem might simply be recalling an identical problem from its pre-training data. The hash-based dedup guarantees this cannot happen.

Pre-Training, Mid-Training, and Post-Training Data Partitions

The paper's key methodological contribution is the precise specification of which data distributions are exposed during each training phase. This specification is what enables causal attribution—by knowing exactly what the model has and has not seen, the paper can conclude that observed generalization reflects genuine composition rather than memorization.

Pre-training data. The base model is pre-trained on 10B tokens (100× the 100M parameter count, following Chinchilla scaling principles) drawn from:

  • Operation range: $\text{op}(G) \in [2, 10]$ — covering basic arithmetic through moderately complex multi-step reasoning. The specific distribution within this range varies by experiment. The default distribution (used in Section 3) is "20% op=2-4 + 30% op=5-7 + 50% op=8-10," meaning half the pre-training data consists of the hardest in-distribution problems (op=8-10).
  • Templates: The default distribution uses equal mixing across three contexts (33% A + 33% B + 33% C), though this proportion is systematically varied in the contextual generalization experiments (Section 4), where context B exposure is reduced to as low as 0%, 0.1%, or 1%.

Why 10B tokens for a 100M model: The Chinchilla scaling law suggests that compute-optimal pre-training uses roughly 20 tokens per parameter. At 100× tokens-to-parameters, the model is trained in a "data-rich" regime, which Li et al. (2025) show leads to better downstream performance. The paper verifies that pre-training achieves near-saturated pass@128 accuracy on the in-distribution range (op=2-10), ensuring that any remaining headroom is on OOD tasks. If pre-training were insufficient (the model hadn't mastered in-distribution tasks), subsequent RL experiments would be confounded—RL might improve in-distribution performance rather than driving genuine extrapolation.

Why stop pre-training at op=10: This is the critical design choice that creates the "headroom" needed to study extrapolation. If pre-training included op=11-20 problems, there would be no out-of-distribution region to test against. By stopping at op=10, the paper creates a clean boundary where:

  • ID (op=2-10): The model has seen these and achieves near-saturation.
  • OOD-edge (op=11-14): The model has NOT seen these, but they require only modestly more compositional depth than the pre-training maximum. The model retains non-zero pass@128 here—it can sometimes solve them through generalization of learned primitives.
  • OOD-hard (op=15-20): The model has NOT seen these, and they require substantially deeper composition. The model exhibits near-zero pass@128 here—its learned primitives are insufficient for the required reasoning depth.

Post-training (RL) data. RL is applied using GRPO with 200K samples drawn from specified operation ranges and template mixtures:

  • Operation range: Varies by experiment. In Section 3, four distinct RL data regimes are tested:
    • $\text{op}=7-10$ (ID): RL data entirely within pre-training range—tests whether RL can improve on tasks the base model already solves.
    • $\text{op}=9-12$ (mixed): straddles the pre-training boundary—tests the transition from ID to OOD.
    • $\text{op}=11-14$ (edge): exactly at the model's edge of competence—the paper's hypothesized optimal RL data range.
    • $\text{op}=17-20$ (hard): far beyond the model's competence—tests whether RL can work with zero-pass@k tasks.
  • Templates: Vary by experiment. In contextual generalization experiments (Section 4), the template mixture includes varying proportions of context A and context B (0% to 100% B), with the total fixed at 200K samples.

Why 200K samples: This is a fixed budget that enables fair comparison across RL data regimes. The total number of training samples is held constant; only their difficulty distribution changes. This means improvements from "edge-competence" RL data cannot be attributed to simply having more data—they reflect better allocation of a fixed data budget.

Mid-training data (Section 5 only). Mid-training is implemented as continued pre-training (same next-token prediction objective) but on a narrowed data distribution: the model sees supervised tokens from the $\text{op}=11-14$ range (with 20% budget allocated to op=2-10 and 80% to op=11-14 to prevent catastrophic forgetting). The total mid-training budget varies by experiment but is normalized against RL cost using the token-equivalence formula described below.

Why continued pre-training rather than SFT: The paper deliberately keeps the mid-training objective identical to pre-training (next-token prediction) to isolate the effect of data distribution narrowing from the effect of objective switching. If mid-training used SFT with a different loss function, observed improvements could reflect either the data distribution change or the objective change. By holding the objective constant, the paper can attribute mid-training effects purely to focusing supervision on the edge-of-competence distribution.

Process-Verified Evaluation Protocol

The paper introduces a strict evaluation protocol that goes beyond final-answer correctness to verify intermediate reasoning steps. This is critical because outcome-only evaluation is vulnerable to reward hacking: a model could produce correct answers through invalid reasoning, inflating apparent performance.

Solution parsing into predicted dependency graphs. Given an input instance with ground-truth DAG $(G, a^*)$, the model produces a free-form solution $s$. A deterministic parser extracts:

G^=(V^,E^,val^),a^\hat{G} = (\hat{V}, \hat{E}, \widehat{\text{val}}), \quad \hat{a}

where $\hat{V}$ is the set of predicted nodes (named intermediate quantities), $\hat{E}$ encodes which previously defined quantities each step depends on, $\widehat{\text{val}}$ stores the inferred numeric value for each node, and $\hat{a}$ is the extracted final answer.

How parsing works: The parser segments the solution into "Define ... as ..." steps—the same format used in the training data. For each step, it (1) identifies the variable being defined, (2) infers the step's dependencies from the variables it references, and (3) evaluates the last computable arithmetic expression in the step to obtain a numeric value (falling back to the last numeric literal if no expression is present). This yields a graph-level representation of the model's reasoning trace aligned with the gold dependency graph.

Why "Define ... as ..." format: This structured output format makes parsing deterministic and reliable. In free-form mathematical reasoning, parsing intermediate steps is notoriously difficult because models express the same logical relationship in many surface forms (e.g., "let x = y + z," "x equals the sum of y and z," "we can compute x as y + z"). By training on a consistent format and expecting it at evaluation, the paper eliminates parsing ambiguity as a confound—if the model produces valid "Define ... as ..." steps, the parser will extract the correct dependency graph; if not, the evaluation correctly penalizes format deviation as part of the reasoning task.

Per-step correctness criterion. For each gold node $v \in V$, a per-step correctness indicator is defined:

s(v;G^,G)={1,if v∈V^,paG^(v)=paG(v), and val^(v) and val(v) are both defined and val^(v)=val(v)0,otherwises(v; \hat{G}, G) = \begin{cases} 1, & \text{if } v \in \hat{V}, \text{pa}_{\hat{G}}(v) = \text{pa}_G(v), \text{ and } \\ & \widehat{\text{val}}(v) \text{ and val}(v) \text{ are both defined and } \widehat{\text{val}}(v) = \text{val}(v) \\ 0, & \text{otherwise} \end{cases}

where $\text{pa}_G(v)$ and $\text{pa}_{\hat{G}}(v)$ are the parent sets (dependencies) of node $v$ in the gold and predicted graphs, respectively, and $\text{val}(v)$ and $\widehat{\text{val}}(v)$ are the gold and predicted numeric values.

What this computes: For each reasoning step that should exist in the correct solution, the criterion checks three conditions simultaneously: (1) the step exists in the model's solution (the node is present in $\hat{V}$), (2) the step uses the correct dependencies (the parent set matches—it combines the right previously-computed quantities), and (3) the step produces the correct value (the computed numeric result matches the ground truth). All three must hold for the step to count as correct.

Why all three conditions: Checking only the value would miss structural errors (e.g., getting the right number through a wrong computation). Checking only the structure would miss arithmetic errors (e.g., correctly identifying that "total = lions + elephants" but computing "3 + 2 = 6"). Checking only existence would miss both. The three-condition conjunction ensures that the model's reasoning trace is structurally faithful and numerically accurate.

Process accuracy. The overall process accuracy of a predicted reasoning trace is the average step-level accuracy over all gold nodes:

ProcessAcc(G^;G)=1∣V∣∑v∈Vs(v;G^,G)\text{ProcessAcc}(\hat{G}; G) = \frac{1}{|V|} \sum_{v \in V} s(v; \hat{G}, G)

What this computes: the fraction of gold reasoning steps that the model correctly reproduces. Extra predicted nodes $v \in \hat{V} \setminus V$ (redundant but compatible intermediate steps) do not affect ProcessAcc—they are allowed but not required.

Verified correctness for pass@k. A prediction is considered fully correct only when both the reasoning graph and the final answer match:

VerifiedCorrect(a^,G^;a∗,G)={1,if ProcessAcc(G^;G)=1 and a^=a∗0,otherwise\text{VerifiedCorrect}(\hat{a}, \hat{G}; a^*, G) = \begin{cases} 1, & \text{if } \text{ProcessAcc}(\hat{G}; G) = 1 \text{ and } \hat{a} = a^* \\ 0, & \text{otherwise} \end{cases}

What this computes: the strict binary correctness used for all pass@k metrics in the paper. A sample counts as correct only if the model (1) gets every gold reasoning step exactly right (step-level process accuracy = 1, meaning 100% of nodes have correct dependencies and values) AND (2) produces the correct final answer.

Why this strict criterion: The paper explicitly states that this prevents inflated performance estimates from reward hacking. If a model produces the correct final answer through an incorrect reasoning chain (e.g., skipping steps, using wrong dependencies that coincidentally yield the right number, or exploiting spurious shortcuts), VerifiedCorrect returns 0 because ProcessAcc will be < 1 (some steps will have incorrect dependencies or values). This means all reported pass@k numbers represent genuine, faithful reasoning rather than coincidental correctness.

The magnitudes reported throughout the paper—+42% pass@128 on OOD-hard, +60% on contextual transfer—reflect this strict metric. Under outcome-only evaluation (checking only the final answer), these numbers would likely be higher because reward-hacked solutions would be counted as correct. The strict criterion therefore represents a lower bound on absolute performance but a more reliable measure of genuine improvement.

RL Training Setup: GRPO with Outcome and Process Rewards

The paper applies Reinforcement Learning from Group Relative Policy Optimization (GRPO) as the post-training method. The base setup uses outcome-based rewards with the following hyperparameters quoted from Appendix A.3.3:

  • RL algorithm: GRPO [Shao et al., 2024]
  • Global batch size: 1,024 examples
  • Maximum prompt and response lengths: 1,024 tokens each
  • Training epochs: 2
  • Actor learning rate: $1 \times 10^{-6}$
  • PPO mini-batch size: 256
  • Micro-batch size per GPU: 16
  • KL regularization coefficient: $10^{-3}$ (low-variance KL penalty)
  • Entropy bonus: 0
  • RL rollout temperature: $T_{\text{RL}} = 1.0$, top-p = 1.0, no top-k truncation (full nucleus sampling)
  • Evaluation generation temperature: $T_{\text{eval}} = 0.7$, top-p = 1.0, top-k = −1 (no truncation), maximum 1,024 new tokens per problem

Reward function design (Section 6). The paper extends the standard outcome reward with process-level verification to study how process supervision affects reasoning fidelity and generalization. The composite reward function is:

R=αRout+(1−α)RpvR = \alpha R_{\text{out}} + (1 - \alpha) R_{\text{pv}}

where $R_{\text{out}} \in \{0, 1\}$ is the traditional outcome-based reward (1 for a correct final answer, 0 otherwise), $R_{\text{pv}} \in [0, 1]$ is the process verification reward defined by the ProcessAcc criteria (the fraction of gold steps correctly reproduced), and $\alpha \in [0, 1]$ controls the balance between outcome accuracy and process fidelity.

What this computes: a weighted combination of final-answer correctness and intermediate-step correctness. When $\alpha = 1.0$, the reward is purely outcome-based (standard). When $\alpha = 0.2$ (the "moderate mix" that achieves best balance in Figure 7), the reward is 20% outcome + 80% process. When $\alpha = 0.0$, the reward is purely process-based.

Why a weighted combination rather than pure process reward: Pure process reward (α = 0) might incentivize the model to produce correct intermediate steps without caring about the final answer—which is the wrong optimization target. Pure outcome reward (α = 1) provides no intermediate guidance and is vulnerable to reward hacking. The weighted combination balances these concerns, and the paper's experiments (Figure 7) validate that moderate mixes (α = 0.2) achieve the best tradeoff.

The paper also considers a strict reward formulation:

R={Rout,if Rpv=10,otherwiseR = \begin{cases} R_{\text{out}}, & \text{if } R_{\text{pv}} = 1 \\ 0, & \text{otherwise} \end{cases}

What this computes: the outcome reward is granted only when the entire reasoning process is verified as correct (ProcessAcc = 1). If any step has an incorrect dependency or value, the reward is 0 regardless of the final answer.

Why this form: This is the strongest anti-reward-hacking signal—the model receives positive reinforcement only when both the reasoning AND the answer are correct. It directly penalizes correct answers achieved through invalid reasoning chains. Figure 7 shows that this strict formulation achieves the highest performance improvements on OOD tasks (up to +5.2% pass@1 on op=15-20, and +4.1% pass@128), confirming that process supervision effectively mitigates reward hacking.

Rollout and training dynamics. During RL, the model generates $r = 6$ rollouts per prompt (the "rollout multiplicity"), and the advantage is computed relative to the group. The total sequence length $L_{\text{total}} = 2048$ includes both the prompt (up to 1024 tokens) and the generated response (up to 1024 tokens).

Why r = 6 rollouts: The group-relative advantage in GRPO requires multiple samples per prompt to estimate a baseline. Six rollouts provides a reasonable balance between advantage estimation quality and computational cost—fewer rollouts would give noisy advantage estimates, while more rollouts would increase the per-step cost linearly.

Compute Budget Equivalence Between Mid-Training and RL

Section 5 compares mid-training and RL under a fixed compute budget, which requires a principled method for normalizing the cost of the two phases. The paper derives this equivalence from standard transformer FLOP accounting.

Pre-training/mid-training FLOPs. Following the Chinchilla scaling law, a decoder-only transformer with $P$ non-embedding parameters trained on $T$ tokens consumes approximately:

Ctrain≈6PTFLOPsC_{\text{train}} \approx 6PT \quad \text{FLOPs}

What this computes: the approximate total floating-point operations for a forward-backward pass over $T$ tokens. The factor of 6 comes from: roughly 2P FLOPs for the forward pass, and roughly 4P FLOPs for the backward pass (which computes gradients for all parameters).

RL FLOPs (GRPO). The paper decomposes on-policy GRPO computation into three components:

CRL=(8+2γ)PNrLtotalC_{\text{RL}} = (8 + 2\gamma) P N r L_{\text{total}}

where $N$ is the number of RL samples, $r = 6$ is the rollout size, $L_{\text{total}} = 2048$ is the total sequence length, and $\gamma \in \{0, 1\}$ toggles whether a reference model forward pass is used (for KL regularization). The components are:

  • Rollout: actor model forward (2P)
  • Reference: optional reference model forward (2P, toggled by $\gamma$)
  • Policy Update: forward pass (2P) and backward pass (4P)

Summing: 2P (rollout forward) + 2γP (reference forward if used) + 2P (update forward) + 4P (update backward) = (8 + 2γ)P per token per rollout.

Token equivalence. Normalizing by the pre-training cost per token (6P) gives the equivalent mid-training token cost:

TRL=CRL6P=(43+γ3)NrLtotalT_{\text{RL}} = \frac{C_{\text{RL}}}{6P} = \left(\frac{4}{3} + \frac{\gamma}{3}\right) N r L_{\text{total}}

When $\gamma = 1$ (reference model used), this simplifies to:

TRL=53NrLtotalT_{\text{RL}} = \frac{5}{3} N r L_{\text{total}}

What this computes: the number of pre-training-equivalent tokens that would cost the same as the RL phase. An RL phase with N samples, r = 6 rollouts, and L_total = 2048 tokens costs $\frac{5}{3} \times 6 \times 2048 = 20,480$ token-equivalents per RL sample.

Why this equivalence matters: Without it, comparisons between mid-training and RL would be meaningless—they use different objectives, different data, and different computational patterns. The token equivalence puts them on a common budget scale (total FLOPs), enabling the paper to ask: "Given a fixed compute budget of T tokens, what fraction β should go to RL vs. mid-training to maximize generalization?"

Budget allocation. The paper defines the RL allocation ratio $\beta \in [0, 1]$:

Tmid=(1−β)⋅T,TRL=β⋅TT_{\text{mid}} = (1 - \beta) \cdot T, \quad T_{\text{RL}} = \beta \cdot T

where $T$ is the total token budget. The paper tests five allocation strategies (quoted from Section 5):

  • Full Mid: $\beta = 0$ (all budget to mid-training, no RL)
  • Full RL: $\beta = 1$ (all budget to RL, no mid-training)
  • Light RL: $\beta = 0.2$ (20% RL, 80% mid-training)
  • Medium RL: $\beta = 0.5$ (50% RL, 50% mid-training)
  • Heavy RL: $\beta = 0.8$ (80% RL, 20% mid-training)

Why this range of β values: Testing five points from 0 to 1 (including three interior points) provides enough resolution to identify whether there is an optimal interior allocation (a "sweet spot" where mid-training + RL outperforms either extreme) or whether the relationship is monotonic (always better to allocate more budget to one phase). Figure 6 shows that the answer depends on the evaluation task: Light RL (β = 0.2) is best for OOD-edge pass@1, while Heavy RL (β = 0.8) or Full RL is best for OOD-hard pass@128.

Model Architecture and Training Hyperparameters

Architecture. All experiments use a decoder-only Qwen2.5-style model with exactly 100M parameters. The detailed architecture from Table 1:

ComponentConfiguration
Model TypeQwen2.5
Number of Layers12
Hidden Size768
Intermediate Size3,072
Number of Attention Heads12
Number of Key-Value Heads2
Activation FunctionSiLU
RMS Norm Epsilon1e-06

Why Qwen2.5 architecture: The paper states this choice is motivated by the observation that Qwen models respond more effectively to RL than architectures like LLaMA [Gandhi et al., 2025, Shao et al., 2025], and that this responsiveness is linked to mid-training practices. Using a Qwen2.5-style architecture ensures the model has the architectural capacity to benefit from RL, making it a more informative testbed than an architecture known to be RL-resistant.

Why 100M parameters: This is small enough to enable systematic experimentation (training dozens of model variants from scratch with different pre-training distributions, mid-training budgets, and RL recipes) while large enough to exhibit meaningful reasoning behavior. The 100× token-to-parameter ratio (10B tokens for 100M parameters) ensures the model is not undertrained on its in-distribution tasks.

Tokenizer. The paper follows the approach of the Physics of Language Models series and trains a byte-pair encoding (BPE) tokenizer directly on the synthetic reasoning corpus. The resulting vocabulary has 2,200 tokens (including special tokens). All problems, questions, and solutions are tokenized with a maximum sequence length of 2,048 tokens.

Why a custom tokenizer: Using a pre-trained tokenizer from a general-domain model would introduce unknown biases—the tokenizer might have been trained on math data that contains patterns similar to the synthetic problems, creating an uncontrolled source of prior knowledge. Training the tokenizer from scratch on only the synthetic corpus guarantees that any reasoning capability the model exhibits was learned during the controlled training phases, not inherited from the tokenizer's training data.

Pre-training hyperparameters (quoted from Appendix A.3.3):

  • Context length: 2,048 tokens
  • Batch size: 512K tokens
  • Learning rate: $2 \times 10^{-4}$
  • Weight decay: 0.1
  • Learning rate schedule: cosine decay with minimum learning rate $3 \times 10^{-5}$
  • Warmup ratio: 5% of total steps
  • Epochs: 1 (single pass over the 10B-token corpus)
  • Precision: bf16

Mid-training hyperparameters (when mid-training is applied, Section 5):

  • Starting from the pre-trained checkpoint
  • Maximum sequence length: 2,048
  • Global batch size: 512K tokens
  • Learning rate: $1 \times 10^{-4}$ (halved from pre-training)
  • Weight decay: 0.1
  • Cosine decay with minimum LR $3 \times 10^{-5}$
  • Warmup ratio: 15% (increased from pre-training to stabilize adaptation to the narrowed distribution)

Why halve the learning rate for mid-training: The mid-training phase starts from a converged pre-training checkpoint, so a full pre-training learning rate would risk destabilizing the model or causing catastrophic forgetting of in-distribution reasoning skills. The halved learning rate provides more conservative updates that refine the model's representations on the edge-of-competence data without overwriting pre-training knowledge. The increased warmup ratio (15% vs. 5%) serves the same purpose—it gives the optimizer time to adapt to the distribution shift before applying the full learning rate.

Key Design Choices That Enable Causal Attribution

Several design choices in this framework are not incidental but are specifically engineered to enable the paper's causal claims:

1. Disjoint data partitions across phases. The pre-training, mid-training, and post-training data are drawn from non-overlapping regions of $(\text{op}(G), \tau)$-space, with hash-based deduplication guaranteeing no instance appears in more than one phase. This means:

  • If the model solves an OOD-hard problem (op=15-20) after RL, it cannot be because it saw that specific problem during pre-training or mid-training—those phases contained only op ≤ 14 problems.
  • If the model generalizes to context B after RL, it cannot be because it saw context B problems at high operation counts during pre-training—pre-training contained only context B at op=2 (atomic primitives) in the contextual generalization experiments.

2. Process-level ground truth for both training and evaluation. The dependency graph is not just an internal representation—it is the ground-truth reasoning structure that the model must reproduce. This dual use (as an RL reward component and as an evaluation criterion) means that "correctness" has a single, consistent definition throughout the paper: structural fidelity to the DAG. There is no gap between "what we reward during training" and "what we measure during evaluation."

3. The edge of competence as a principled region for RL data selection. The paper identifies the edge of competence not by intuition but by empirical measurement: problems where the base model's pass@1 is low (cannot solve in one attempt) but pass@128 is non-zero (can sometimes solve with enough sampling). This operational definition is critical because it means the RL data contains problems that are solvable by the base model in principle (so exploration during RL can find correct solutions) but not reliably solvable (so there is headroom for RL to improve). If RL data were too easy (high pass@1), RL would merely sharpen existing skills; if too hard (zero pass@128), RL would receive no positive reward signal and could not learn.

4. Operation count as a single, continuous difficulty knob. By quantifying reasoning complexity with a single integer $\text{op}(G)$, the paper can precisely control the "distance" of evaluation problems from the training distribution. An OOD-hard problem with op=18 is exactly "8 operations harder" than the pre-training maximum of op=10, and "4 operations harder" than the RL training maximum of op=14. This continuous scaling enables systematic study of how generalization degrades with increasing distance from the training distribution, rather than binary in-distribution vs. out-of-distribution comparisons.

5. The 100× token-to-parameter ratio for pre-training. By training in a data-rich regime and verifying near-saturated in-distribution performance, the paper ensures that in-distribution failures are not due to insufficient pre-training. This is essential because if the base model hadn't fully mastered op=2-10 tasks, observed RL improvements on harder tasks might partly reflect continued learning of in-distribution skills (i.e., RL compensating for undertraining) rather than genuine extrapolation.

6. Fixed RL sample budget (200K) across all data regimes. By holding the total number of RL samples constant and varying only their difficulty distribution, the paper isolates the effect of data curation quality from the effect of data quantity. Improvements from edge-competence RL data are specifically due to better targeting of the model's learning frontier, not simply more RL steps.

4. Key Insights and Innovations

Innovation 1: The "Edge of Competence" as a Diagnostic Principle for RL Data Curation

The paper's most conceptually distinctive contribution is the operationalization of the edge of competence—not as a vague intuition about "not too easy, not too hard," but as a precise, measurable region in the capability landscape where RL produces genuine capability gains rather than amplifying existing skills or failing entirely from reward sparsity. This is a diagnostic concept, not a new algorithm: it tells practitioners how to identify which problems will benefit from RL investment, transforming what was previously a trial-and-error data curation process into a principled selection criterion.

What the field did before this. Prior work on RL for reasoning treated data difficulty as either a binary (in-distribution vs. out-of-distribution) or a quantity to be maximized (harder is better for generalization). The conflicting empirical narratives—Yue et al. (2025) finding no pass@128 improvement, Yuan et al. (2025) finding substantial compositional generalization gains—reflected different accidental positions relative to the edge of competence. Yue et al. used standard math benchmarks that the base model largely solved (high pass@1, well within the pre-training distribution), so RL only sharpened sampling efficiency. Yuan et al. designed synthetic composition tasks where the base model failed completely at pass@1 but had non-zero pass@k, placing their RL data precisely at the edge. Neither study identified this region as the operative variable; both interpreted their results as general claims about RL's capabilities.

Why this is a fundamental reframing. The edge of competence transforms the "does RL work?" debate from an empirical dispute into a conditional answer: RL does produce genuine capability gains, but only when two conditions are simultaneously satisfied—the pre-training leaves headroom (the base model hasn't already saturated the task) AND the RL data is calibrated to the region where the model can sometimes succeed through exploration but cannot yet do so reliably. Figure 3 provides the cleanest evidence: RL on op=7-10 (ID, high pass@1) improves pass@1 but never surpasses the base model's pass@128 ceiling; RL on op=17-20 (far OOD, near-zero pass@128) produces no meaningful improvement; only RL on op=11-14 (edge, non-zero pass@128 but low pass@1) yields genuine pass@128 gains of up to +42% on OOD-hard generalization. This is not a gradual effect—it is a sharp qualitative transition. The practical implication is that RL data curation is not about maximizing difficulty or volume, but about identifying and targeting this narrow frontier, which shifts as the model improves.

The edge of competence also explains an otherwise puzzling negative result in Appendix K: the ReST^EM-trained revision model degraded with sequential revisions. The authors hypothesize that on-policy data collection amplified spurious correlations, but an edge-of-competence lens suggests a complementary interpretation—ReST^EM likely pushed the training distribution beyond the model's frontier, generating data with near-zero pass@k at the new difficulty level and starving the RL process of positive reward signal.

Innovation 2: Reconciling Conflicting RL Narratives Through Pre-Training Exposure Control

The paper makes a significant methodological contribution by demonstrating that the two dominant competing narratives about RL—"RL does not extend reasoning" (Yue et al., 2025; Wu et al., 2025; Zhao et al., 2025) and "RL enables novel compositional generalization" (Yuan et al., 2025; Sun et al., 2025a; Liu et al., 2025b)—are not contradictory but rather describe different regions of a unified phase space defined by pre-training coverage and RL data difficulty. This is more than an empirical observation; it is a conceptual unification that provides the field with a framework for interpreting future RL results rather than relitigating the binary debate.

What distinguishes this from prior reconciliation attempts. Several prior works have noted that RL's effectiveness varies, but none provided a mechanistic explanation grounded in controlled pre-training manipulation. The paper's innovation is to show that the key parameter is not RL algorithm choice, reward design, or model scale—it is the interaction between what the model already knows and what the RL data demands. The paper demonstrates this through the most controlled manipulation in the literature: by fixing the pre-training distribution (op=2-10) and varying only the RL data range, Figure 3 shows that the same base model with the same RL algorithm and the same training budget (200K samples) produces results that would support either narrative depending solely on which operation range is used for RL. This is causal evidence that the conflicting narratives are artifacts of dataset choice, not fundamental disagreements about RL's capabilities.

The significance for experimental design. This finding implies that any future study claiming to evaluate "whether RL extends reasoning" must report the base model's pass@k curve on the evaluation tasks and the RL data's position relative to the model's edge of competence. Without these, the study's conclusions are uninterpretable—they could reflect either genuine RL-driven generalization (if the data is at the edge) or mere sharpening of existing skills (if the data is in-distribution). This is a methodological standard that the paper effectively proposes and that would substantially improve the rigor of RL-for-reasoning research.

Evidence anchor. Figure 3 is the smoking gun: pass@128 on OOD-hard (op=15-20) increases from near-zero (base) to ~20% with RL on op=11-14, but shows no improvement with RL on op=7-10 or op=17-20. This is a ~20 percentage point swing in generalization performance from changing nothing except which difficulty range receives the fixed RL budget.

Innovation 3: Mid-Training as a Distinct "Prior Installation" Phase with Task-Aware Budget Allocation

While mid-training has been acknowledged as a component of modern training pipelines (Wang et al., 2025; Liu et al., 2025a; Akter et al., 2025), the paper elevates it from an engineering detail to a conceptually distinct training phase with a specific functional role—installing reasoning priors that RL can amplify—and demonstrates that its value relative to RL depends on the generalization objective (OOD-edge vs. OOD-hard). This is not merely "mid-training helps"; it is a principled characterization of when and why mid-training matters, and how compute should be allocated between mid-training and RL based on the deployment goal.

What the field assumed before this. The dominant implicit model treated mid-training as either (a) an optional bridge between pre-training and RL that could be subsumed by better pre-training data, or (b) a mysterious source of "RL readiness" (Gandhi et al., 2025; Shao et al., 2025) whose mechanism was unexplained. Wang et al. (2025) showed that mid-training LLaMA models on reasoning data produced RL performance comparable to Qwen bases, but did not systematically vary the mid-training/RL budget split or connect the choice to task difficulty. Liu et al. (2025a) demonstrated that mid-training reduces forgetting and eases adaptation, but focused on the distributional bridge rather than the strategic allocation question.

What the paper establishes that is new. Figure 6 reveals a non-monotonic relationship between the mid-training/RL budget split and performance, where the optimal allocation differs qualitatively by task regime. On OOD-edge tasks (op=11-14), Light RL (β=0.2, 20% RL, 80% mid-training) achieves the best pass@1, while Heavy RL (β=0.8) degrades pass@1. On OOD-hard tasks (op=15-20), the pattern reverses: Heavy RL achieves the best pass@1 and pass@128, while Light RL provides minimal benefit over the base model. This is not a "sweet spot" that works everywhere—it is a task-dependent tradeoff where mid-training installs the necessary priors (without which RL flounders) and RL provides the exploration (without which the model cannot push beyond the mid-training distribution). The paper's key insight is that these are complementary rather than competing investments, and the optimal balance shifts with the hardness of the target task.

Why this is a fundamental contribution to training pipeline design. The field has largely treated mid-training as a fixed upfront cost—"do some mid-training, then do RL"—without strategic budget allocation. The paper's results suggest this is suboptimal: compute should be allocated in a task-aware manner, with more mid-training when the goal is reliability on similar tasks (OOD-edge) and more RL when the goal is exploration on harder tasks (OOD-hard). This directly informs the decisions faced by organizations deploying reasoning models: customer-support reliability requires a different mid/RL split than research-grade mathematical reasoning.

Evidence anchor. Figure 6 shows Light RL at ~55% pass@1 on op=11-14 versus ~51% for Heavy RL, while Heavy RL achieves ~8% pass@1 on op=15-20 versus ~5% for Light RL. The crossover is sharp and consistent across multiple budget levels (Appendix A.10, Figure 18).

Innovation 4: Process Verification as an Anti-Reward-Hacking Mechanism with a Causal Role in Generalization

While process reward models (PRMs) have been used for evaluation and search (Lightman et al., 2023; Uesato et al., 2022), the paper demonstrates that incorporating process verification directly into the RL reward function produces a causal shift in the model's reasoning strategy—away from shortcut exploitation and toward structurally faithful reasoning—that yields measurable improvements in generalization performance, particularly under extrapolative settings. This is not merely "process rewards improve accuracy"; it is evidence that reward design shapes how the model generalizes, not just whether it produces correct answers.

What distinguishes this from prior PRM work. Prior work on PRMs largely focused on verifier-guided search at inference time (e.g., beam search against step-level scores) rather than using process signals as RL training rewards. The paper's contribution is to show that process-aware rewards during training produce a qualitatively different model behavior: the model learns to produce valid reasoning chains rather than exploiting spurious correlations that yield correct final answers through incorrect intermediate steps. This is visible in Figure 8's structural error analysis: process-aware rewards reduce dependency mismatches and missing nodes while increasing the proportion of fully correct reasoning traces, compared to pure outcome rewards.

The significance for the reward hacking literature. Reward hacking is a well-known failure mode in RL (Shao et al., 2025; Pan et al., 2024; Skalse et al., 2022), but most mitigation strategies focus on reward model improvements or KL regularization to prevent policy divergence. The paper's approach—augmenting the sparse outcome signal with dense process-level feedback—is a complementary strategy that addresses the root cause: outcome-only rewards are insufficient to specify how to achieve the correct answer, creating an incentive for any strategy that yields the right output regardless of validity. By rewarding intermediate step correctness, the RL signal aligns the optimization trajectory with faithful reasoning rather than merely correct outputs, which the paper shows leads to better generalization (OOD-hard pass@1 improves by +4.3% to +5.2% with process-augmented rewards, Figure 7).

Evidence anchor. Figure 8 shows that under pure outcome rewards, the model exhibits ~70% dependency mismatches on OOD-edge tasks; process-augmented rewards (particularly the "outcome only if process correct" formulation) reduce this to ~85% correct traces. The improvement in reasoning fidelity is causally linked to generalization: Figure 7 shows consistent pass@k improvements on OOD-hard tasks (op=15-20) across all process-reward formulations, with the strict formulation achieving the largest gains (+5.2% pass@1, +4.1% pass@128).

Innovation 5: The "Minimal Seed" Principle—Sparse Pre-Training Exposure as Sufficient for RL-Driven Transfer

The paper establishes a principle with significant implications for data engineering strategy: RL-driven contextual generalization does not require comprehensive pre-training coverage of a domain, but rather a minimal yet sufficient seed of atomic primitives—as little as 1% exposure to a long-tailed context during pre-training is sufficient for RL to robustly transfer reasoning competence to that context, while 0% or 0.1% exposure is not. This is a fundamentally asymmetric relationship: pre-training provides the necessary initialization (what the paper calls "latent seeds"), while RL provides the amplification and composition that generalizes from atomic primitives to complex reasoning in the novel context.

What this implies that is non-obvious. The natural assumption—implicit in most data curation strategies—is that more pre-training coverage is always better, and that long-tailed domains require substantial representation to enable downstream generalization. The paper's result challenges this: beyond a surprisingly low threshold (~1%), additional pre-training exposure to the long-tailed context yields diminishing returns (Figure 4). This suggests that the expensive part of reasoning is not exposure to surface forms but mastery of compositional operations—and that once atomic operations are seeded in a new context, RL can efficiently compose them into complex reasoning without requiring extensive in-context pre-training data. Practitioners should prioritize breadth of atomic primitive coverage (at ~1% density across many long-tailed contexts) over depth of coverage in any single context.

Why this is a fundamental insight rather than an incremental finding. The field has largely treated pre-training and RL as separate investments with independent returns. The "minimal seed" principle reveals a multiplicative interaction: RL's effectiveness on a context is zero below a threshold pre-training exposure and approximately maximal slightly above it, meaning the marginal return to pre-training coverage is highly nonlinear. This has direct implications for how to allocate pre-training data budgets across domains—spread broadly at low density rather than concentrated deeply—and suggests that current pre-training practices (which inevitably underrepresent thousands of long-tailed domains) may nevertheless provide sufficient seeds for RL-driven transfer to many contexts that appear scarce in the pre-training corpus.

Evidence anchor. Figure 4: with 0.1% pre-training exposure to context B atomic examples, pass@128 on context B after RL is near zero across all operation counts. With 1% exposure, pass@128 jumps to ~80% on op=2-10 and remains at ~40% on op=15-20 (a >60 percentage point gain from a 10× increase in pre-training exposure, from 0.1% to 1%). Further increasing exposure to 10% provides only modest additional gains. The threshold is sharp—it is not a continuous scaling relationship. Discussion 2 and Figure 5 provide the mechanistic interpretation: at 1% exposure, the model generates novel reasoning structures (low topological similarity to context A) rather than merely replicating context A patterns, indicating genuine compositional transfer rather than template adaptation.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use a synthetic reasoning dataset generated through the paper's controlled framework built on GSM-Infinite [Zhou et al., 2025a]. The full corpus contains 30B tokens spanning multiple operation ranges (op=2-20) and contextual templates (animals–zoo, teachers–school, movie-festival). The corpus is partitioned into disjoint splits for pre-training, mid-training, and post-training, with hash-based deduplication ensuring no instance appears in more than one phase. For evaluation, the paper samples problems at specified operation counts and templates to test extrapolative and contextual generalization; the exact evaluation set sizes are not explicitly stated, but the paper's difficulty quintile analysis in Section 3 splits the test problems into bins by operation count (e.g., ID: op=2-10, OOD-edge: op=11-14, OOD-hard: op=15-20).

  • Base model. All experiments use a 100M-parameter Qwen2.5-style decoder-only transformer trained from scratch. The architecture follows the Qwen2.5 configuration with 12 layers, hidden size 768, intermediate size 3,072, 12 attention heads, 2 key-value heads, SiLU activation, and RMS norm epsilon 1e-06 (Table 1). A custom BPE tokenizer with 2,200 tokens is trained solely on the synthetic reasoning corpus to avoid uncontrolled prior knowledge from general-domain tokenizers. The model is pre-trained on 10B tokens (100× parameter count, following data-rich Chinchilla principles) from the op=2-10 range across multiple templates. The authors justify this scale as enabling systematic experimentation across dozens of training configurations while being large enough to exhibit meaningful reasoning behavior, and note that Qwen2.5 architectures respond more effectively to RL than alternatives like LLaMA [Gandhi et al., 2025, Shao et al., 2025], making it a more informative testbed for studying RL effects.

  • Metrics. The primary metric throughout is process-verified pass@k accuracy—the fraction of evaluation problems for which at least one of k sampled solutions satisfies the strict VerifiedCorrect criterion, which requires both perfect step-level reasoning (ProcessAcc = 1, meaning every gold dependency graph node has correct dependencies and numeric values in the model's predicted trace) AND the correct final answer. This is a harder criterion than outcome-only evaluation and is designed to prevent inflated performance from reward hacking. Pass@k is computed for k ∈ {1, 2, 4, 8, 16, 32, 64, 128} using the standard unbiased estimator. For the reward composition analysis in Section 6, the paper additionally reports error-type breakdowns: the percentage of solutions falling into categories of "correct" (full process match), "dependency mismatch" (incorrect parent sets in the predicted graph), and "missing nodes" (gold nodes absent from the predicted trace), extracted from the process-verified evaluation parser.

  • Baselines. The primary baseline across all experiments is the base pre-trained model before RL or mid-training, evaluated at the same pass@k levels. This is not a separate model but the initialization from which all RL/mid-training variants are derived, enabling direct before-vs-after comparison. In the extrapolative generalization experiments (Section 3), the base model achieves near-saturated pass@128 on ID tasks (op=2-10, ~95-100%), non-zero but declining pass@128 on OOD-edge tasks (op=11-14, ranging from ~60% at op=11 to ~10% at op=14 as shown in the performance ladder in Appendix Figure 9), and near-zero pass@128 on OOD-hard tasks (op=15-20, <5%). In the contextual generalization experiments (Section 4), the base model's performance on context B varies with pre-training exposure to context B atomic primitives. For the mid-training experiments (Section 5), additional baselines include Full Mid (all compute budget to continued pre-training on op=11-14 data, no RL) and Full RL (all budget to GRPO on op=11-14 data, no mid-training), both compared at equivalent total token budgets as defined by the compute equivalence formula in Section 5.

  • Generation budget / compute accounting. In the RL-only experiments (Section 3), the budget is measured in number of RL training samples, fixed at 200K for all data regimes, ensuring that performance differences reflect data curation quality rather than quantity. For the mid-training vs. RL comparison (Section 5), the paper normalizes both phases to equivalent pre-training tokens based on FLOPs. Mid-training cost follows the Chinchilla approximation C_train ≈ 6PT FLOPs for a model with P parameters trained on T tokens. RL cost is decomposed into rollout (2P forward), optional reference model pass (2P if used), and policy update (2P forward + 4P backward), yielding C_RL = (8 + 2γ)PNrL_total where γ∈{0,1} toggles the reference model, N is the number of RL samples, r=6 is the rollout multiplicity, and L_total=2048 is total sequence length. Normalizing by 6P gives the token-equivalent cost T_RL = (5/3)NrL_total when γ=1. The total compute budget T is varied across experiments (from ~1B to ~20B tokens as shown in Appendix Table 3 and Figure 18), and the RL allocation ratio β∈[0,1] distributes this budget between mid-training (T_mid = (1-β)T) and RL (T_RL = βT).

  • Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional sense. Instead, evaluation is performed on held-out problem instances that are guaranteed disjoint from all training phases by the hash-based deduplication described in Appendix A.1.5. For the difficulty-dependent analyses, problems are grouped by operation count (not by empirical performance binning), establishing clean category boundaries. The paper does not report confidence intervals, standard errors, or statistical significance tests for any of the reported pass@k numbers. The number of evaluation problems per operation-count bin is not explicitly stated, though Appendix Figure 9 suggests continuous evaluation across operation counts. This absence of statistical quantification is a notable limitation for interpreting the reliability of the reported gains.

Main Quantitative Results

When Does Post-Training Incentivize Reasoning Beyond the Base Model? (Section 3, Figures 3, 10, 11)

The headline finding is that RL produces genuine pass@128 gains on OOD tasks only when the RL data targets the model's edge of competence—specifically, the op=11-14 range where the base model has non-zero pass@128 but low pass@1. On OOD-hard tasks (op=15-20), RL with edge-competence data (op=11-14) improves pass@128 from near-zero (the base model's performance) to approximately 20% at k=128 (Figure 3, rightmost panel, red line). This is a ~20 percentage point gain over the base model and a ~20 point gain over RL using in-distribution data (op=7-10, which shows no pass@128 improvement on OOD-hard). On OOD-edge tasks (op=11-14), the same RL regime improves pass@128 from the base model's ~60% to approximately 80% at k=128 (Figure 3, middle panel, red line), representing a ~20 point gain.

The critical comparison is the behavior of different RL data regimes on the same evaluation tasks. On ID tasks (op=2-10), all RL regimes improve pass@1 (the base model's ~85% rises to ~92-96% depending on regime) but none improve pass@128 beyond the base model's already-saturated ~98-100% (Figure 3, left panel). This confirms that RL sharpens sampling efficiency without extending the capability ceiling for in-distribution tasks. On OOD-edge tasks (op=11-14), RL on op=17-20 (far OOD) produces essentially no pass@128 improvement over the base model, while RL on op=9-12 (mixed ID/OOD) and op=11-14 (edge) both yield substantial gains, with the edge-competence regime showing the strongest improvement (~60% → ~80% at pass@128). On OOD-hard tasks (op=15-20), only RL on edge-competence data (op=11-14) improves pass@128 meaningfully (near-zero → ~20%); RL on op=9-12 produces modest gains (~0% → ~10%), while RL on op=7-10 and op=17-20 produce no improvement, with the performance curves remaining flat near zero throughout training (Figure 3, right panel).

The training dynamics (Appendix Figure 11) reinforce these findings. The reward curves for RL on op=9-12 and op=11-14 show steady improvement from ~0.50-0.55 to ~0.60-0.65 over 200 steps, indicating genuine learning progress. In contrast, RL on op=7-10 shows a flat reward curve around ~0.72-0.80 (already high, leaving little room for improvement), and RL on op=17-20 shows a flat reward curve near ~0.04 (essentially zero, indicating the model cannot discover positive examples to learn from). The response length trajectories show complementary patterns: edge-competence RL leads to modestly increasing response lengths (~280 → ~320 tokens), suggesting the model is learning to produce more elaborate reasoning chains, while ID RL shows flat response length (~245-255 tokens), and far-OOD RL shows flat length (~277-280 tokens).

The NLL reduction analysis (Appendix Figure 10) provides a fine-grained view of how RL affects the model's internal representations across operation ranges. RL on op=11-14 produces the largest NLL reduction on op=15-20, with NLL decreasing by approximately 2.5-3.0 nats compared to the base model. The NLL gains decay smoothly as the evaluation range diverges from the RL-trained operations, consistent with a model that is learning compositional patterns that transfer best to nearby difficulty levels. RL on op=7-10 produces NLL reductions concentrated in the op=2-10 range with minimal spillover to higher op counts—the model sharpens its in-distribution representations but does not extend them.

How Does Pre-Training Exposure Shape Post-Training Contextual Generalization? (Section 4, Figures 4, 5, 17)

The headline finding is that RL-driven contextual generalization requires minimal but sufficient pre-training exposure to atomic primitives in the target context—with a sharp threshold at approximately 1% exposure. When the base model is pre-trained with 0.1% context B atomic examples (op=2 only), RL post-training with a 50% context A + 50% context B mixture fails completely to generalize to context B: pass@128 on context B is near zero across all operation counts from op=2 to op=20 (Figure 4, purple line). When pre-training exposure increases to 1% context B atomic examples, pass@128 on context B jumps to ~80% on op=2-10 tasks, ~60% on op=11-14, and ~40% on op=15-20 (Figure 4, red line)—a gain of >60 percentage points at the hardest level from only a 10× increase in pre-training exposure. When pre-training exposure increases further to 10%, context B pass@128 rises modestly to ~85% on op=2-10, ~65% on op=11-14, and ~45% on op=15-20 (Figure 4, green line)—diminishing returns from additional pre-training coverage.

The topological similarity analysis (Figure 5) reveals the mechanism underlying this threshold effect. The paper measures the similarity between the dependency graphs generated by the model for correct context B solutions and the ground-truth graph topologies from context A. At 0.1% context B pre-training exposure, the distribution of similarities is concentrated near 1.0 across all difficulty levels—the model is primarily replicating context A reasoning patterns, indicating it has not learned context-specific structural adaptation. At 1% exposure, the similarity distribution shifts leftward, particularly for harder tasks: on op=15-20, 20-40% of correct solutions show similarity below 0.6, indicating the model is generating novel reasoning structures distinct from context A patterns. At 10% exposure, the distribution shifts further left, with a mode at similarity ~0.4-0.6 for op=11-20 tasks, suggesting the model has developed genuinely context-adapted compositional strategies.

The training dynamics (Appendix Figure 17) provide additional evidence. Models with 0.1% context B pre-training exposure show a flat reward curve during RL (~0.32-0.34 over 200 steps) and flat response length (~210-215 tokens), indicating no learning progress. Models with 1% exposure show steady reward improvement from ~0.40 to ~0.60 and increasing response length from ~230 to ~280 tokens. Models with 10% exposure show the strongest reward improvement (~0.40 → ~0.70) and the longest response lengths (~240 → ~280 tokens), though the performance difference at evaluation (Figure 4) between 1% and 10% exposure is modest compared to the 0.1% → 1% jump, confirming the threshold rather than linear-scaling nature of the relationship.

The supplementary analysis on shared-primitive contextual generalization (Appendix A.6.1, Figure 12) provides a contrasting baseline. When pre-training includes the full reasoning primitives in context B (not just atomic op=2 examples, but op=2-20 data at 0.1% of the pre-training corpus), RL post-training with 0% context B data still achieves pass@128 of ~80% on context B at op=2-10 and ~60% at op=11-20. This shows that when the reasoning primitives are fully covered in pre-training, RL can transfer them to long-tailed contexts even without any in-context RL data—a stronger form of generalization than the "minimal seed" case where only atomic primitives are pre-trained. Increasing RL exposure to context B data produces a smooth improvement curve, with 100% context B RL data achieving pass@128 of ~90% on op=2-10 and ~80% on op=20, confirming that post-training context exposure provides additional but not threshold-level benefits when pre-training coverage is already sufficient.

Mid-Training and Post-Training Interaction Under Fixed Compute (Section 5, Figures 6, 18)

The headline finding is that mid-training and RL exhibit complementary, task-dependent strengths under a fixed compute budget, with the optimal allocation shifting based on the generalization objective. Figure 6 presents the core comparison across five training configurations at a total budget T (the specific value of T is not stated in the main text, but Appendix Table 3 shows experiments spanning T ∈ {1.05, 2.10, 4.20, 8.40, 12.58, 16.78, 20.0} × 10^9 tokens).

On OOD-edge tasks (op=11-14): For pass@1, Light RL (β=0.2, 80% mid-training + 20% RL) achieves the best performance at approximately 55%, compared to Full RL (β=1.0) at ~51%, Heavy RL (β=0.8) at ~52%, and Full Mid (β=0.0) at ~53%. The ordering is Full Mid ≈ Light RL > Medium RL > Heavy RL ≈ Full RL for pass@1. For pass@128 on the same OOD-edge tasks, the pattern reverses: Full RL achieves the highest performance at approximately 93%, compared to Light RL at ~87%, with the ordering being Full RL > Heavy RL > Medium RL > Light RL ≈ Full Mid. This crossover indicates that mid-training improves the likelihood of correct single-sample solutions (pass@1), while RL is essential for maximizing the coverage of correct solutions in the model's output distribution (pass@128).

On OOD-hard tasks (op=15-20): Both pass@1 and pass@128 are maximized by RL-heavy allocations, with the effect being stronger at pass@128. For pass@1 on OOD-hard, Heavy RL achieves approximately 8%, compared to Light RL at ~5% and Full Mid at ~3%. For pass@128 on OOD-hard, Heavy RL achieves approximately 20%, compared to Light RL at ~12% and Full Mid at ~6%. Full RL (no mid-training) achieves the highest pass@128 at ~22%, slightly edging out Heavy RL. The consistent pattern is that RL exploration is indispensable for OOD-hard generalization, and mid-training alone provides minimal benefit on these far-out-of-distribution tasks.

The compute budget scaling analysis (Appendix Figure 18) tests robustness. The key patterns replicate across budgets from ~1.0B to ~20.0B tokens. At every budget level, Light RL achieves the best OOD-edge pass@1, and Heavy RL achieves the best OOD-hard pass@1. For pass@128, the relationship evolves with budget: at low budgets (4.2B tokens), Heavy RL achieves the best OOD-hard pass@128 (~18%), while at higher budgets (8.4B tokens and above), Full RL overtakes Heavy RL (e.g., at 20B tokens, Full RL achieves ~28% pass@128 on OOD-hard vs. Heavy RL at ~24%). This suggests that as compute increases, the relative value of exploration (RL) over prior installation (mid-training) grows for hard tasks.

The specific numeric comparison from the main text: "mid-training + RL outperforming RL alone by +10.8% on OOD-hard tasks" (Figure 1, right panel of the overview figure). This number likely refers to a specific budget and comparison point, but the exact configuration is not explicitly stated in the main text beyond the reference to Figure 1. Based on Figure 6, the largest gap between a mid-training+RL configuration and Full RL on OOD-hard pass@1 appears to be Light RL (with mid-training) vs. Full RL, or potentially a specific budget-combination from Appendix Figure 18.

Mitigating Reward Hacking via Process Supervision (Section 6, Figures 7, 8)

The headline finding is that incorporating process verification into the RL reward function consistently improves both pass@1 and pass@128 performance while reducing structural reasoning errors, with the strict formulation (outcome reward granted only when the entire reasoning process is correct) achieving the largest gains.

Figure 7 presents the core comparison across four reward compositions, all trained on op=11-14 RL data and evaluated on both OOD-edge (op=11-14 combined with op=2-10 into a 2-14 evaluation) and OOD-hard (op=15-20) tasks. On the broader OOD-edge evaluation (op=2-14), pure outcome reward achieves pass@1 of 70.7% and pass@128 of 94.7%. The best-performing process-augmented variant—0.2 outcome + 0.8 process—improves pass@1 to 75.4% (+4.7 percentage points) while maintaining pass@128 at 94.7% (no change). The strict formulation (outcome only if process correct) achieves pass@1 of 75.9% (+5.2 points) and pass@128 of 95.0% (+0.3 points). On OOD-hard tasks (op=15-20), the gains are more pronounced: pure outcome reward achieves pass@1 of 5.6% and pass@128 of 23.2%. The 0.2 outcome + 0.8 process formulation improves pass@1 to 9.9% (+4.3 points) and pass@128 to 27.7% (+4.5 points). The strict formulation achieves pass@1 of 9.7% (+4.1 points) and pass@128 of 25.8% (+2.6 points). The 0.5 outcome + 0.5 process formulation sits between these extremes, suggesting a monotonic relationship where more process weight (up to the strict formulation threshold) monotonically improves performance, with the strict formulation providing the best pass@1.

The structural error analysis (Figure 8) reveals the mechanism: process-aware rewards shift the model's reasoning strategy away from shortcut exploitation. On OOD-edge tasks (op=11-14), the pure outcome reward model produces only ~50% fully correct reasoning traces (where ProcessAcc = 1), with ~40% showing dependency mismatches (incorrect parent sets in the predicted graph) and ~5% showing missing nodes. Under the strict process-verified reward (outcome only if process correct), the proportion of fully correct traces rises to ~60%, dependency mismatches drop to ~25%, and missing nodes remain at ~5%. On OOD-hard tasks (op=15-20), the effect is even more dramatic: pure outcome reward produces ~90% correct traces (surprising, but note this is conditional on the model getting the right answer—which it rarely does on OOD-hard; the base rate of correctness is low, so these percentages reflect a small denominator of "solutions that happened to get the right answer"). The strict formulation maintains ~95% correct traces, with dependency mismatches dropping from ~5% to ~2% and missing nodes from ~3% to ~1%. The key insight from this figure is that process-aware rewards are not merely improving final-answer accuracy—they are causally shifting the distribution of how the model reasons, increasing the proportion of structurally faithful solutions.

Ablation Studies and Robustness Checks

Varying RL Data Difficulty (Section 3, Appendix A.5): Testing four RL data regimes (op=7-10, op=9-12, op=11-14, op=17-20) confirms that the edge-competence region (op=11-14) uniquely enables pass@128 gains on OOD tasks. The training reward dynamics (Appendix Figure 11) show that op=7-10 RL saturates quickly (reward ~0.72-0.80 throughout, no improvement), op=17-20 RL shows no learning signal (reward ~0.04 throughout), while op=9-12 and op=11-14 show steady reward improvement from ~0.50-0.55 to ~0.60-0.65. This ablation confirms that both "too easy" and "too hard" RL data fail for fundamentally different reasons: the former because the model already knows how to solve these tasks (no headroom), the latter because the model cannot discover any correct solutions to learn from (no positive reinforcement signal).

Varying Pre-Training Hard-Data Exposure (Appendix A.7, Figure 15): Manipulating the proportion of op=8-10 data during pre-training from 0.1% to 50% (while keeping total pre-training tokens fixed at 10B) reveals a complementary relationship. Greater hard-data exposure improves both base and post-trained performance, but the marginal gain from RL diminishes as pre-training becomes more comprehensive. When pre-training includes 0.1% hard data, RL on op=11-14 improves pass@128 on op=15-20 by +5.9 percentage points. At 20% hard data, the RL gain is +22.1 points (the largest relative boost). At 50% hard data, the RL gain is +25.8 points but the base model's performance is already higher (meaning the total post-RL performance is similar to the 20% condition). The NLL reduction analysis (Appendix Figure 10) confirms that pre-training difficulty shapes the model's representational readiness for RL, and that RL is most effective when pre-training provides partial but incomplete coverage of complex primitives.

Varying Context B Exposure During RL (Appendix A.6.2, Figure 13): Testing RL data mixtures with 0%, 1%, 10%, 50%, and 100% context B (while keeping the base model pre-trained with only atomic context B examples at 1% exposure) reveals a threshold at ~10% RL exposure for contextual transfer. At 0% or 1% RL context B exposure, pass@128 on context B remains near zero. At 10% exposure, pass@128 jumps to ~80% on op=2-10 and ~60% on op=11-20. Further increases to 50% or 100% provide diminishing returns, reaching ~85% and ~70% respectively. Importantly, context A performance remains stable across all RL mixtures, even when RL uses 100% context B data—the model does not catastrophically forget context A reasoning (pass@128 remains at ~95% throughout, Figure 13 left panel). The training reward dynamics (Appendix Figure 14) confirm that 0-1% context B RL exposure produces flat rewards, while 10-100% exposure shows steady reward improvement from ~0.62-0.70 to ~0.70-0.75 and increasing response lengths.

Varying Process Reward Weight (Section 6, Figure 7): Testing four reward formulations—pure outcome (α=1.0), balanced (α=0.5), process-heavy (α=0.2), and strict (outcome only if process correct)—demonstrates a monotonic relationship between process supervision strength and performance. Process-heavy (α=0.2) consistently outperforms balanced (α=0.5) on both pass@1 and pass@128, and the strict formulation generally performs best or ties for best across all metrics. The error-type analysis (Figure 8) confirms that this improvement is driven by reduction in structural errors (dependency mismatches and missing nodes) rather than simply improved arithmetic accuracy.

Mid-Training Budget Scaling (Appendix A.10, Figure 18): Testing five mid-training/RL allocation ratios (β ∈ {0, 0.2, 0.5, 0.8, 1.0}) across seven total compute budgets (1.05B to 20.0B tokens, Appendix Table 3) shows that the qualitative patterns from Figure 6 replicate across all budget scales. At 1.05B tokens, Light RL achieves OOD-edge pass@1 of ~52% vs. Heavy RL at ~49%; Heavy RL achieves OOD-hard pass@1 of ~4% vs. Light RL at ~2%. At 20.0B tokens, Light RL achieves OOD-edge pass@1 of ~60% vs. Full RL at ~56%; Full RL achieves OOD-hard pass@128 of ~28% vs. Light RL at ~16%. The consistent replication across two orders of magnitude of compute budget strengthens the claim that these are structural properties of the mid-training/RL interaction, not artifacts of a particular budget choice.

Contextual Generalization with Shared Primitives (Appendix A.6.1, Figure 12): When pre-training includes full reasoning primitives in context B (0.1% of corpus, op=2-20), RL with 0% context B data still transfers to context B at pass@128 of ~80% on op=2-10 and ~60% on op=20. This contrasts with the minimal-seed case (Appendix A.6.2) where 0% RL context B exposure yields near-zero transfer, confirming that the RL context-exposure requirement depends on whether the pre-training exposure included full reasoning depth or only atomic primitives.

Pre-Training Exposure to Context B with Only Atomic Primitives (Section 4, Figure 4): Testing 0%, 0.1%, 1%, and 10% context B atomic examples during pre-training (while keeping the total pre-training token budget fixed) confirms the threshold effect at ~1% for enabling RL-driven contextual transfer. The non-monotonic behavior (near-zero transfer at 0.1%, strong transfer at 1%, diminishing returns at 10%) is a negative result for the hypothesis that pre-training exposure scales linearly with downstream RL transfer—implying a fundamental nonlinearity in how pre-training seeds interact with RL amplification.

Deduplication Across Training Phases (Appendix A.1.5): While not presented as an ablation with performance numbers, the paper emphasizes hash-based deduplication across all training and evaluation splits as a critical validity check. Any failure to properly deduplicate would invalidate the paper's central causal claims by allowing training-evaluation contamination. The paper states that "no identical problem–solution triple appears in both training and evaluation," which is a binary guarantee rather than a measured ablation.

Critical Assessment

Claim: "RL produces true capability gains (pass@128) only when pre-training leaves sufficient headroom and when RL data target the model's edge of competence."

Does the evidence support this? The evidence for the positive case—that edge-competence RL improves pass@128—is strong. Figure 3 clearly shows that RL on op=11-14 improves pass@128 on OOD-hard (op=15-20) from near-zero to ~20%, while RL on op=7-10 and op=17-20 produce no pass@128 improvement. The gradient of NLL reduction across operation ranges (Appendix Figure 10) provides converging representational evidence: edge-competence RL produces the largest NLL reduction on OOD-hard tasks, consistent with genuine compositional learning.

What is not fully tested: The claim uses the word "only," which requires demonstrating that the edge-of-competence condition is necessary, not just sufficient. While the paper shows that two failure modes exist (too easy and too hard), it does not systematically vary the "distance" from the edge of competence to identify the precise boundaries where the effect vanishes. For instance: does RL on op=10-13 (slightly shifted from op=11-14) produce similar gains? Does RL on op=13-16 (harder than edge but with some pass@k signal) work? The paper tests four discrete RL data ranges, which provide strong evidence for the qualitative pattern but leave the precise functional form of the edge-of-competence window underexplored.

The "headroom" condition is coarser than it appears. The paper defines headroom as "the task was not heavily covered during pre-training," operationalized as op>10 (beyond the pre-training maximum). But even within the ID range (op=2-10), the model achieves near-100% pass@128, meaning there is effectively no headroom anywhere in-distribution. This creates a binary test: ID (no headroom) vs. OOD (headroom). A more graded test would vary pre-training maximum operation count (e.g., op=8, op=10, op=12) and assess whether RL gains scale with the size of the headroom gap. The current design only tests one headroom gap (pre-training to op=10, RL on op=11-14), which is sufficient to establish the principle but insufficient to characterize how RL effectiveness varies with headroom magnitude.

Pass@128 as the diagnostic metric deserves scrutiny. The paper's central diagnostic—that RL "truly extends" reasoning if pass@128 improves—is built on the assumption that pass@128 measures the model's capability ceiling independent of sampling efficiency. This is defensible but not self-evident: pass@128 from a model with 128 independent samples at temperature 0.7 is not the same as measuring what the model "knows" in a representation-level sense. If RL changes the model's output distribution to concentrate probability mass on correct solutions (rather than actually enabling new reasoning pathways), pass@128 would improve without genuine capability extension. The paper attempts to rule this out through the process-verified evaluation (which requires structurally correct reasoning, not just correct answers) and the NLL analysis (which shows representational change), but a skeptic could argue that improved pass@128 under process verification still reflects distributional sharpening rather than novel composition. The paper does not provide a direct test that distinguishes "the model can now compose operations it couldn't before" from "the model now assigns higher probability to compositions it could already occasionally produce."

Claim: "Contextual generalization requires minimal yet sufficient pre-training exposure, after which RL can reliably transfer."

Does the evidence support this? The evidence for the threshold effect at ~1% pre-training exposure is clear and striking (Figure 4). The jump from near-zero transfer at 0.1% to strong transfer at 1% (a >60 percentage point gain on hard tasks) is one of the most statistically unambiguous findings in the paper despite the absence of formal significance tests—the effect size is simply too large to be noise. The topological similarity analysis (Figure 5) provides a plausible mechanistic interpretation: at 1%, the model generates novel graph structures rather than replicating context A patterns, suggesting genuine compositional transfer.

What is coarser than it might appear: The paper tests 0.1%, 1%, and 10% exposure—which spans two orders of magnitude but with only two intermediate points (0.1% and 1%) in the critical region where the threshold supposedly lies. Is the threshold at 0.5%? At 2%? At exactly 1%? The data cannot distinguish a sharp threshold (step function at some critical value) from a steep sigmoid (continuous but highly nonlinear scaling). The distinction matters for practical guidance: if the threshold is sharp, practitioners need only cross it; if it is continuous, more exposure continues to help (if with diminishing returns).

The "minimal seed" is operationalized narrowly. The pre-training seed consists of atomic op=2 examples in context B—the simplest possible reasoning problems. This is a clean experimental design, but it leaves open the question: what is the content of the seed that matters? Is it the surface forms (template vocabulary, entity names)? The reasoning structure (dependency graph patterns at op=2)? The solution format and rendering? The paper does not ablate these components to determine which aspect of the seed is necessary. If the seed primarily provides surface-form exposure, then the finding is about vocabulary/format transfer rather than reasoning transfer; if the seed provides graph-structure patterns, the finding is about compositional primitive transfer. Both would produce the observed results but have different implications for data engineering strategy.

Claim: "Mid-training significantly enhances performance under fixed compute compared with RL only, demonstrating its central but underexplored role."

Does the evidence support this? The evidence supports a more nuanced claim than the headline: mid-training helps, but its value relative to RL depends on the task. Figure 6 shows that Light RL (β=0.2, heavy mid-training) achieves the best OOD-edge pass@1, while Full RL or Heavy RL achieves the best OOD-hard pass@k. This is not "mid-training is better than RL in general" but rather "mid-training and RL have complementary, task-dependent strengths." The +10.8% figure cited in the executive summary is one point in this space and should not be interpreted as a universal advantage.

What is not tested: The paper only tests mid-training on op=11-14 data—the same data used for edge-competence RL. This conflates two roles that mid-training might play: (a) narrowing the data distribution to focus on the edge of competence, and (b) providing supervised next-token-prediction training as a bridge to RL. If mid-training on op=11-14 data helps, is it because that specific data distribution is valuable, or because any continued pre-training on reasoning data helps? Testing mid-training on other data distributions (e.g., op=2-10 to reinforce in-distribution skills, or op=15-20 to push further OOD) would disambiguate these explanations. Additionally, the paper only tests one mid-training objective (continued pre-training with next-token prediction). Contemporary mid-training often uses instruction-formatted or SFT-style data [Liu et al., 2025a], and the paper's choice to keep the objective identical to pre-training (to isolate distribution effects) means the results may not generalize to SFT-based mid-training.

The compute budget equivalence model is approximate. The FLOPs-based equivalence between mid-training and RL (Equation 6, derived from Chinchilla scaling) makes several simplifying assumptions: (1) it assumes the backward pass is exactly 2× the forward pass, which is approximate for modern transformer implementations with activation checkpointing and gradient accumulation; (2) it uses the same sequence length for mid-training and RL (2048 tokens), but RL includes both prompts and generated responses, which have different computational patterns (causal attention with key-value caching for generation vs. bidirectional attention for training); and (3) it does not account for the fact that RL rollouts are generated autoregressively (sequential token generation), which has different hardware utilization characteristics than the parallel forward passes used in mid-training. These approximations are reasonable for a first-order analysis but mean the reported budget allocations (β values) are approximate rather than exact.

Claim: "Process-level rewards reduce reward hacking and improve reasoning fidelity."

Does the evidence support this? The evidence for improved performance with process-aware rewards is consistent across all metrics and formulations (Figure 7). The error analysis (Figure 8) provides a mechanistic explanation—process rewards shift the model toward structurally faithful reasoning—though the interpretation requires care because Figure 8 shows error distributions conditional on the model producing a solution that passes some correctness threshold (the exact conditioning is not fully specified). The reduction in dependency mismatches under process-aware rewards is substantial (from ~40% to ~25% on OOD-edge), providing direct evidence for the anti-reward-hacking mechanism.

What limits the strength of this claim: The paper does not demonstrate that the observed performance gains are caused by reduction in reward hacking rather than by the dense reward signal providing better credit assignment. Process rewards are dense (providing feedback at every reasoning step), while outcome rewards are sparse (only at the final answer). Dense rewards are known to improve RL sample efficiency and final performance independent of any anti-hacking effect. The paper's error analysis shows that process rewards reduce structural errors, but this could be because (a) the model is rewarded for correct intermediate steps, directly incentivizing structural fidelity (the anti-hacking mechanism), or (b) the dense reward signal helps the model learn faster and more stably, and this improved optimization leads to better structural fidelity as a byproduct. The paper does not include an ablation that provides dense rewards without process verification (e.g., rewarding any intermediate step output regardless of correctness, or using an outcome-only reward with step-level decomposition for credit assignment) to distinguish these mechanisms.

Additionally, the process verification reward requires parsing the model's output into a dependency graph at training time—the same parsing used for evaluation. If the parser has systematic biases or failures (which the paper does not characterize), the process reward could inadvertently reward or penalize specific output patterns in ways that are artifacts of the parser rather than genuine process supervision.

Cross-Cutting Weaknesses

Small number of evaluation templates. The paper uses three templates (animals–zoo, teachers–school, movie-festival). Contextual generalization is characterized as transfer from one template to another, but with only three templates, it is difficult to distinguish genuine "learned reasoning primitives independent of surface form" from the model learning a mapping between three specific surface forms. Testing on 10 or more diverse templates would substantially strengthen the claim that RL enables abstract reasoning transfer rather than template-specific adaptation. The current design is sufficient to demonstrate the phenomenon but insufficient to characterize its generality.

No statistical quantification of uncertainty. The paper reports pass@k as point estimates without confidence intervals, standard errors, or any measure of variance. While the effect sizes are large enough that the qualitative patterns are credible, the precise magnitudes (e.g., "+4.7% pass@1" from process rewards in Figure 7) should be interpreted with appropriate uncertainty. The lack of statistical reporting is a significant limitation for a paper whose contributions include precise quantitative comparisons between training configurations.

Evaluation at a single model scale. All experiments use 100M-parameter models. While the authors justify this scale as enabling systematic experimentation, the interaction between pre-training, mid-training, and RL may change with model scale. Larger models typically have stronger base reasoning capabilities and may exhibit different edge-of-competence characteristics (e.g., the edge may be at higher operation counts, or the headroom gap may be narrower). The paper does not provide evidence that its findings scale, and while the Chinchilla-style token-to-parameter ratio (100×) is well-motivated, it is unknown whether the qualitative patterns (threshold effects, optimal mid/RL ratios) are scale-invariant.

Evaluation at a single architecture. Only Qwen2.5-style models are tested. The paper's motivation for this choice—that Qwen models respond better to RL than LLaMA [Gandhi et al., 2025, Shao et al., 2025]—is circular with respect to the paper's goals. If the response to RL is architecture-dependent, then the paper's conclusions about when RL produces gains may be specific to architectures that are "RL-responsive." A LLaMA-style baseline (even at the same 100M scale) would clarify whether the edge-of-competence principle is architecture-general or Qwen-specific.

The process-verified evaluation parser is a potential single point of failure. All results depend on the deterministic parser that extracts dependency graphs from free-form model outputs. If the parser has systematic failures (e.g., misparsing valid but non-canonical reasoning formats, or incorrectly accepting invalid reasoning that happens to match the expected format), the reported pass@k numbers would be systematically biased. The paper does not report parser accuracy, parser failure modes, or inter-annotator agreement (if any human validation was performed). Given that process verification is used for both training rewards and evaluation metrics, parser reliability is a critical unexamined assumption.

Missing ablation: RL without process verification but with outcome-only reward + KL regularization. The process reward experiments show that adding process supervision improves performance over pure outcome rewards, but the paper does not test whether simply increasing KL regularization (to prevent the policy from diverging too far from the base model, which is a standard anti-reward-hacking technique) would achieve similar benefits without requiring step-level verification infrastructure. If strong KL regularization achieves comparable reasoning fidelity improvements, the practical case for process reward models (which require expensive step-level annotation or parsing) would be weakened.

The compute-equivalence model conflates training and inference compute. The FLOPs-based equivalence treats all FLOPs as interchangeable, but mid-training FLOPs and RL FLOPs have different hardware utilization characteristics, memory footprints, and parallelization properties. Mid-training uses large-batch, highly parallel forward-backward passes, while RL rollout generation is inherently sequential (token-by-token autoregressive decoding). In practice, an "equivalent FLOP" budget for RL may require substantially more wall-clock time and different hardware configurations than the same FLOP budget for mid-training. The paper's budget allocation recommendations (e.g., "Light RL for OOD-edge tasks") may translate differently to practical wall-clock budgets than to theoretical FLOP budgets.

6. Limitations and Trade-offs

The Difficulty Estimation Cost That Makes Deployment Impractical

The assumption or constraint. The paper's entire compute-optimal framework depends on knowing a prompt's difficulty before deciding how to allocate the inference budget. The method used to estimate difficulty—generating 2048 samples per question and scoring them with the ground-truth PRM—costs more compute than the largest test-time budgets studied. The paper explicitly acknowledges this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The difficulty estimation step is not a one-time cost—it must be performed at deployment for every new prompt unless difficulty can be predicted from the prompt alone (which the paper does not develop). The paper frames this as "an exploration-exploitation tradeoff" between "compute spent assessing difficulty versus compute spent solving the problem."

The consequence. In any realistic deployment, the total compute cost equals difficulty estimation + strategy execution. Since difficulty estimation via 2048 samples is roughly 8–16× more expensive than the largest test-time budgets reported (256–512 generations), the headline "4× efficiency improvement" is best understood as an upper bound that assumes difficulty is known for free. If difficulty estimation were amortized into the total budget, the efficiency gains would shrink dramatically or potentially reverse—the system would spend more compute determining how to solve the problem than actually solving it. This is not merely a quantitative scaling issue; it is a fundamental gap between the experimental methodology and deployment feasibility.

The paper does not develop a cheap difficulty estimator. Section 8 lists it as future work: "training models to directly predict difficulty of a question," but no such model is trained or evaluated. Until this gap is closed—either through learned difficulty predictors, adaptive estimation that reuses early samples, or strong priors about prompt difficulty distributions—the compute-optimal framework cannot be deployed in the form presented.

What evidence exists in the paper. The limitation is stated explicitly in Section 3.2, and the cost of the estimation procedure can be inferred directly: 2048 samples × the base model's per-sample inference cost, which for PaLM 2-S* is substantial. The paper provides no ablation or alternative estimation method with reduced cost. All results in Figures 4, 8, and 9 are computed after difficulty is known, without amortizing the estimation cost.

Mitigation status. Not addressed. The paper acknowledges the limitation as a simplification for experimental purposes and flags future work on learned difficulty prediction. A partial mitigation—adaptive estimation where initial samples serve double duty as difficulty probes and solution attempts—is mentioned conceptually but not implemented or evaluated.

The Method Provides Zero Benefit on the Hardest Problems

The assumption or constraint. The paper's approach rests on the premise that test-time compute amplifies existing capability—it helps the model find and select correct solutions that are already present in its output distribution at some non-trivial rate, but it cannot create capability where none exists. Section 7 explicitly states this boundary condition in a takeaway box:

"Test-time compute can amplify existing capability but does not create it from nothing."

The operational manifestation is difficulty bin 5 (the hardest quintile), where the base model's pass@1 is near zero. Across all methods—search, revisions, and their compute-optimal combinations—bin 5 shows essentially no improvement regardless of compute budget.

The consequence. This is not a limitation that can be addressed by scaling compute or improving the allocation policy—it is a hard capability bound imposed by the base model's pre-training. For problems where the base model cannot produce a correct solution even once in 2048 attempts, no test-time strategy (search, revision, or adaptive allocation) will help. This means compute-optimal test-time scaling is fundamentally restricted to the regime where the base model already possesses the necessary reasoning primitives.

The FLOPs-matched comparison in Section 7 quantifies the severity: on bin 5 (hardest problems), test-time compute shows a −52.9% relative disadvantage compared to the ~14× larger model at R >> 1 (Figure 9, right panel). Even at R << 1 (where test-time compute performs best overall), bin 5 shows essentially zero improvement—the scaling curves in Figure 9 are flat near 0–5% for the hardest bin regardless of budget.

In practical terms, this means that deploying compute-optimal test-time scaling on a problem distribution with a heavy tail of genuinely difficult queries will waste compute on problems where no improvement is possible, and the system needs a mechanism to detect such problems early and either escalate to a larger model or flag for human review.

What evidence exists in the paper. The evidence is extensive and consistent:

  • Figure 3 (right): bin 5 pass@128 is ~1–3% for all search methods at all budgets.
  • Figure 7 (right): bin 5 accuracy is ~2–3% across all sequential-to-parallel ratios.
  • Figure 9: the bin 5 scaling lines (blue, bottommost) are essentially flat near 0–5% in all panels.
  • Appendix M qualitatively shows search producing "degenerate outputs" and repetitive low-information steps on hard problems, indicating that the model is generating text but not approaching correct solutions.

Mitigation status. Not mitigated within the framework. The paper is transparent about this limitation and correctly identifies that pretraining is the only viable path for hard problems. However, it does not provide a mechanism for detecting which problems fall into this regime at deployment time without the expensive difficulty estimation process described above. A practical system would need to recognize "this problem is beyond our model's capability" early in the test-time compute allocation and avoid wasting budget on futile optimization—a capability the paper does not develop.

The 14×14\times Larger Model Baseline Is Not Compute-Optimal, Making the FLOPs Comparison Favorable to Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 uses a pretraining-scaled model where only the parameter count is increased (~14× more parameters) while the training data volume is held fixed. The paper explicitly acknowledges this departure from compute-optimal pretraining in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This follows the "LLaMA paradigm" (Touvron et al., 2023) of training on more tokens than Chinchilla-optimal, rather than the compute-optimal scaling where both data and parameters increase together (Hoffmann et al., 2022).

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs—scaling both parameters and training tokens equally—would likely achieve higher performance than a model that scales only parameters while holding data fixed. This means the pretraining baseline in Section 7 is weaker than it needs to be, and the reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R << 1, Figure 1 bar chart) may overstate the case for inference-time computation.

The direction of the bias is clear: a stronger pretraining baseline would shrink or potentially reverse the test-time compute advantage, particularly in the medium-difficulty bins where the margins are smaller. Additionally, the ~14× larger model is evaluated with only greedy decoding—no majority voting, no best-of-N, and no search. The paper does not consider giving the larger model even a modest test-time compute budget (e.g., best-of-8 with majority voting), which would create a much more competitive baseline. A larger model with some test-time compute might dominate the smaller model with extensive test-time compute across a wider range of difficulties and R values.

What evidence exists in the paper. The limitation is acknowledged in Section 7, and the choice is justified as "representative of a canonical approach." The paper does not include any ablation with a Chinchilla-optimal larger model or with a larger model augmented with test-time compute. The entire FLOPs comparison in Figure 9 and the Figure 1 bar charts depends on this baseline choice.

Mitigation status. Acknowledged but not addressed. The authors frame the compute-optimal pretraining comparison as future work. This is a reasonable acknowledgment, but the headline claim—"a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model"—should be interpreted with the caveat that the larger model is not compute-optimally trained and uses a weaker inference protocol.

The Findings Are Demonstrated on a Single Benchmark with a Single Model Family

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper defends this choice in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

and argues that MATH is appropriate because it requires multi-step logical deduction rather than factual recall. However, the paper provides no evidence that the difficulty-dependent scaling patterns, the over-optimization thresholds, or the optimal allocation strategies generalize to other model families, other reasoning domains, or other task formats.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific in ways that practitioners cannot assess from the reported results:

Model-specific factors. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different pass@1 distributions across difficulty levels would exhibit different optimal allocation strategies. For example, if a base model has higher pass@1 on MATH problems (e.g., a more capable model), the "easy" bin would be larger, the "hard" bin smaller, and the edge-of-competence transition would occur at different difficulty thresholds. The paper's difficulty quintiles are defined relative to PaLM 2-S*'s capabilities, not absolute problem difficulty, meaning the optimal per-bin strategies are model-specific.

Benchmark-specific factors. MATH consists of competition-level math problems with exact ground-truth answers, which enables both the Monte Carlo rollout PRM training (requiring automated correctness checking) and the difficulty estimation procedure (requiring pass@1 computation). Many real-world reasoning tasks—code generation, logical reasoning, scientific QA, planning—have different structure (less clean correctness signals, different error patterns, different composition of easy vs. hard problems). It is unknown whether the core findings (beam search degrades on easy problems, revisions help on easy problems, sequential-to-parallel ratio matters for medium problems) would replicate in these domains.

Limited test set size. The 500-question test set, split into five difficulty quintiles of ~100 each, then further divided by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether observed differences between strategies (e.g., beam search vs. best-of-N at a specific budget level) are statistically reliable given the small per-bin sample size.

What evidence exists in the paper. The limitation is not explicitly addressed as a generalization concern. The paper's experimental section (Section 4) describes the MATH benchmark and PaLM 2-S* as the experimental setting but does not discuss how findings might transfer. The cross-validation protocol (Section 3.2) acknowledges the small per-bin sample size implicitly by using two-fold cross-validation, but does not quantify statistical uncertainty.

Mitigation status. Not addressed within the paper. The authors do not claim generalizability beyond MATH and PaLM 2-S*, but neither do they caution readers about the specificity of their findings. Replication on additional benchmarks (code generation, logical reasoning) and model families (different scales, different architectures) would be required to establish whether the difficulty-conditioned compute-optimal approach is broadly applicable or MATH/PaLM-specific. The paper's framework is designed to enable such replication (the synthetic reasoning data generator can produce problems in different domains), but the replication itself is not performed.

The PRM and Revision Model Are Trained on Different Distributions, Preventing Their Combination

The assumption or constraint. The paper studies two complementary mechanisms—PRM-guided search (Section 5) and iterative revisions (Section 6)—as independent axes for improving test-time compute. However, they are never combined into a unified system. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The practical barrier to combination is a distribution shift problem: the PRM is trained on base model outputs, but the revision model's outputs come from a different distribution (fine-tuned on sequential revision chains). Appendix J (Figure 15a) shows that "the base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs," confirming that distribution shift is a measurable problem. The revision-specific ORM trained in Appendix J mitigates this for best-of-N weighted selection but does not provide the per-step scores needed for beam search or lookahead search against revision model outputs.

The consequence. The paper's results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary, difficulty-dependent strengths: revisions improve the proposal distribution (generating better candidates through iterative refinement), particularly on easy problems, while PRM search improves candidate selection (finding the best among generated candidates), particularly on medium-hard problems. A system that applies beam search to revision model outputs—using a PRM trained on revision model distributions to score intermediate steps—could potentially capture both benefits, pushing performance beyond what either mechanism achieves alone.

The distribution shift problem also has implications for the scalability of each approach individually. As revisions change the model's output distribution (the proposal distribution drifts during fine-tuning), any verifier trained on the original base model distribution becomes progressively less reliable. This creates a co-adaptation problem: the revision model and the verifier need to be retrained jointly or iteratively, but the paper does not explore this.

What evidence exists in the paper.

  • Section 8 acknowledges the non-combination explicitly as a direction for future work.
  • Appendix J, Figure 15a provides quantitative evidence of the distribution shift problem: the base-LM PRM achieves lower accuracy when scoring revision model outputs compared to the revision-specific ORM.
  • The paper's difficulty-dependent analysis (Figures 3 right, 7 right) demonstrates the complementary strengths that motivate combination but does not test it.

Mitigation status. Flagged as future work but not addressed. The paper identifies the combination as a natural next step and the distribution shift as a barrier, but does not propose or evaluate a training procedure for a unified PRM that works across both base and revision model outputs. Options include: fine-tuning the PRM on revision model outputs (similar to the revision-specific ORM but with per-step labels), using the revision model itself as a verifier (scoring its own confidence at each step), or jointly training the revision model and PRM in an alternating fashion.

Sequential Revision Latency Is Not Accounted for in the Cost Model

The assumption or constraint. The paper measures test-time compute in "generations"—one complete sampled answer from the model—and treats all generations as having equivalent cost. This is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial: each revision depends on the previous one, and the chain must be generated step-by-step. Parallel best-of-N sampling, by contrast, can be executed simultaneously with sufficient hardware (all N samples generated at once).

The consequence. The paper's compute-optimal policy favors sequential-heavy allocations for easy problems and balanced allocations for medium problems (Figures 7, 8). A strategy that allocates 128 generations as "64 sequential × 2 parallel" takes roughly 32× longer wall-clock time than one that runs 128 parallel samples simultaneously (assuming the sequential chain length is 64 and each generation takes roughly equal time). For latency-sensitive applications—interactive assistants, real-time decision-making, customer-facing systems where users wait for responses—the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be completely impractical regardless of their accuracy advantages.

This creates a FLOPs-to-latency tradeoff that the paper does not characterize. An organization optimizing for queries-per-second at fixed accuracy would make different allocation choices than one optimizing for accuracy at fixed FLOPs. The paper's budget allocation recommendations (e.g., "use sequential revisions on easy problems") assume FLOPs are the only scarce resource, which is not true in most deployment scenarios.

What evidence exists in the paper. The paper does not discuss latency, wall-clock time, or throughput. The generation budget is defined solely in terms of number of samples (Section 2), and the compute-equivalence formulas (Section 5.3, Equation 1) treat all generations as identical tokens for FLOP accounting. The sequential-to-parallel ratio experiments (Figure 7) sweep allocation strategies without reporting or controlling for latency implications.

Mitigation status. Not addressed. The paper's cost model is purely FLOP-based, consistent with the scaling laws literature it builds on, but this is a significant mismatch with deployment reality. A practical extension would need to introduce a latency budget constraint (e.g., "must produce an answer within 2 seconds") and re-optimize the allocation policy under that constraint. This would likely shift the optimal policy toward more parallel sampling—trading some FLOP efficiency for lower latency—on problems where the accuracy gain from sequential revisions is modest.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new algorithm, architecture, or training objective. Its contribution is conceptual and methodological: it provides the first controlled causal framework for disentangling how pre-training, mid-training, and RL post-training jointly produce reasoning capabilities in language models. The magnitude of the shift is best characterized not as a paradigm change—the basic training pipeline (pre-train → mid-train → RL) remains intact—but as a reframing of how the field should interpret and allocate investment across these stages, backed by the kind of causal evidence that opaque large-scale training pipelines have never provided.

Resolving the central contradiction in the RL-for-reasoning literature. The paper's most immediate impact is to reconcile the two dominant and seemingly contradictory narratives about RL's effectiveness. On one side, Yue et al. (2025), Wu et al. (2025), and Zhao et al. (2025) present evidence that RL does not extend reasoning beyond the base model's capabilities—it sharpens sampling efficiency (improving pass@1) but leaves pass@128 unchanged. On the other side, Yuan et al. (2025), Sun et al. (2025a), and Liu et al. (2025b) demonstrate substantial compositional generalization gains from RL on synthetic tasks. This paper shows that both findings are correct, but neither is universal—they describe different regions of a unified phase space defined by where the RL data sits relative to the base model's edge of competence. Figure 3 makes this point with causal clarity: the same base model, same RL algorithm, and same training budget (200K samples) can produce results supporting either narrative, depending solely on which operation-count range is used for RL. RL on in-distribution data (op=7-10) improves pass@1 but never surpasses the base model's pass@128 ceiling—replicating the Yue et al. finding. RL on edge-of-competence data (op=11-14) yields ~20 percentage point pass@128 gains on OOD-hard tasks—replicating the Yuan et al. finding. The debate is resolved not by declaring one side correct, but by showing that the operative variable was never whether RL works, but rather whether the RL data was calibrated to the model's learning frontier.

This reframing has a methodological consequence for the field: any future study claiming to evaluate "whether RL extends reasoning" must report (a) the base model's pass@k curve on the evaluation tasks, and (b) the position of the RL data relative to the model's edge of competence. Without these, the study's conclusions are uninterpretable—they could reflect genuine RL-driven generalization or mere sharpening of existing skills, with no way to distinguish between them. The paper effectively proposes this as a new reporting standard for RL-for-reasoning research, analogous to how the Chinchilla scaling laws standardized the reporting of model size and data quantity in pretraining studies.

Mid-training elevated from engineering detail to strategic design choice. Prior to this work, mid-training was acknowledged as a component of modern training pipelines—Wang et al. (2025) showed it improved RL responsiveness, Liu et al. (2025a) characterized it as a distributional bridge—but it was treated as a fixed upfront cost rather than a strategically allocated resource. The paper's compute-budget analysis in Figure 6 demonstrates that the optimal mid-training/RL split is not a universal constant but a task-dependent variable: Light RL (80% mid-training, 20% RL) achieves the best OOD-edge pass@1, while Heavy or Full RL achieves the best OOD-hard pass@k. This means that organizations training reasoning models face a genuine design choice—allocate compute toward prior installation (mid-training) or exploration (RL)—and the right answer depends on whether the deployment goal is reliability on similar tasks or generalization to harder ones. This reframing makes mid-training a first-class object of optimization in training pipeline design rather than an afterthought.

Process verification as a training signal, not just an evaluation tool. The paper extends process reward models (PRMs) from their traditional role as verifiers for search and evaluation (Lightman et al., 2023) into the RL reward function itself, demonstrating that process-aware rewards produce a causal shift in reasoning strategy—away from shortcut exploitation and toward structurally faithful reasoning—that yields measurable generalization improvements (Figure 7: +4.3% to +5.2% pass@1 on OOD-hard tasks across process-reward formulations). This suggests that reward design is not merely about providing a correct optimization target, but about shaping the trajectory of generalization, with process-level supervision producing models that generalize differently (better) than outcome-only supervision even when final-answer accuracy is comparable. This shifts the reward design conversation from "how do we prevent reward hacking?" toward "how do we design rewards to incentivize the reasoning patterns that generalize?"

The "minimal seed" principle and its implications for data strategy. The finding that RL-driven contextual transfer requires only ~1% pre-training exposure to atomic primitives in a long-tailed context—but fails completely at 0.1%—establishes a nonlinear threshold relationship between pre-training coverage and downstream RL effectiveness. This challenges the implicit assumption in most data curation strategies that more pre-training coverage is always better and that underrepresentation of long-tailed domains is a fundamental barrier. Instead, the paper suggests a breadth-over-depth strategy: pre-training should prioritize broad coverage of atomic primitives across many domains at low density (~1%) rather than deep coverage in any single domain, because RL can efficiently compose atomic primitives into complex reasoning once they are seeded. This has direct implications for how pre-training data budgets should be allocated across the long tail of domains, skills, and formats—spread thinly but broadly, trusting RL to handle the composition.

Research directions that become less attractive. The paper's negative results prune several branches of potential research investment:

  • Sophisticated search algorithms over verifiers become less attractive relative to improving verifier robustness. The paper's finding that lookahead search—the most powerful optimizer—paradoxically performs worst overall (Figure 3, left panel of the original paper Section 5.3) due to over-optimization suggests that the bottleneck is verifier reliability, not search algorithm sophistication. Researchers should invest in better PRM training (reduced over-optimization, better calibration, on-policy data) rather than more complex search procedures.

  • Naively maximizing RL data difficulty becomes clearly counterproductive. The paper shows that RL on far-OOD data (op=17-20) produces zero improvement—the model cannot discover correct solutions to learn from, so the reward signal is always zero and no learning occurs. The field should abandon the intuition that "harder RL data is always better" in favor of edge-of-competence curation.

  • Using pass@1 as the sole metric for RL improvement is revealed as potentially misleading. The paper shows that RL on in-distribution data improves pass@1 substantially without extending the capability ceiling (pass@128 unchanged), meaning pass@1 gains can reflect sharpening rather than genuine improvement. Studies that report only pass@1 without pass@k curves at higher k may be overclaiming RL's effectiveness.

Follow-Up Research This Work Enables

Developing cheap, prompt-only difficulty estimators to close the deployment gap. The paper's compute-optimal policy depends on knowing a prompt's difficulty before allocating compute, but the current estimation method (2048 samples + PRM scoring) is far more expensive than the test-time budgets being optimized. A natural follow-up would train a lightweight classifier—potentially a small distilled model or even a linear probe on the base model's representations—that takes only the prompt text as input and predicts the difficulty bin. The training signal exists: the paper has already computed oracle and predicted difficulty bins for 500 MATH test questions, providing a labeled dataset. A strong follow-up would (a) train a difficulty predictor that requires no sample generation, (b) measure its correlation with the oracle bins, (c) integrate it into the compute-optimal pipeline, and (d) report whether the efficiency gains survive when the difficulty estimation cost is amortized. The key metric is whether a system with a cheap predictor + compute-optimal allocation outperforms a system that spends the same total budget on uniform best-of-N. The paper already showed that predicted bins (using PRM scores) track oracle bins closely (Figures 4, 8 curves largely overlap), so the remaining question is whether the prediction can be made cheap enough.

Combining PRM tree search with the revision model under a unified verifier. The paper studies PRM-guided search and iterative revisions as independent mechanisms, but they have complementary, difficulty-dependent strengths: revisions improve the proposal distribution on easy problems, search improves candidate selection on medium problems. Section 8 acknowledges this combination as future work but identifies the barrier: the PRM is trained on base model outputs and suffers distribution shift when scoring revision model outputs (Appendix J, Figure 15a). A follow-up would address this by training a unified PRM on revision model outputs—generating step-level Monte Carlo rollouts from the revision model and training a PRM on that distribution—and then applying beam search or lookahead search with this PRM to revision model candidates. The key comparison would be: does PRM search over revision model outputs outperform (a) PRM search over base model outputs and (b) best-of-N weighted over revision model outputs, at equivalent generation budgets? The hypothesis from the paper's difficulty analysis is that this combination should excel on medium-difficulty problems (bins 3-4), where both mechanisms individually show their strongest relative advantages.

Adaptive, online difficulty estimation that amortizes probing into problem-solving. The paper flags the difficulty estimation cost as an exploration-exploitation tradeoff but does not implement an adaptive solution. A natural extension would develop a dynamic allocation policy that begins by generating a small number of parallel samples (say, 4-8), uses the PRM's score distribution on those samples as a real-time difficulty signal, and then decides whether to switch to beam search, continue parallel sampling, or initiate sequential revisions for the remaining budget. This connects to the multi-armed bandit and Bayesian optimization literatures. A strong follow-up would compare such an adaptive policy against the oracle compute-optimal policy (which knows difficulty in advance) to measure the cost of online estimation, and against uniform best-of-N as a baseline. The metric of interest is regret: how much performance is lost due to allocating some budget to probing rather than directly solving, compared to knowing difficulty from the start. If the regret is small (e.g., the adaptive policy achieves 95% of the oracle policy's accuracy at the same total budget), the framework becomes immediately practical.

Replicating the edge-of-competence finding across model families, scales, and domains. The paper's core causal claim—that RL produces pass@128 gains only when data targets the edge of competence—is demonstrated on a single model family (PaLM 2-S*) at a single scale, on a single benchmark (MATH). A critical stress test would replicate the controlled experiment on (a) a different architecture family (e.g., LLaMA-style models, which Gandhi et al., 2025 and Shao et al., 2025 showed respond differently to RL), (b) different model scales (the edge-of-competence region likely shifts with model capability—a 1B-parameter model's edge is at different operation counts than a 100M model's), and (c) different reasoning domains (code generation on HumanEval/MBPP, logical reasoning on ARC/FOLIO, where the relationship between operation count and difficulty may differ). The paper's synthetic data generation framework is designed to enable such replication—the GSM-Infinite generator can produce problems in new domains—but the replication itself is the essential next step. A negative result—finding that the edge-of-competence principle does not hold for a particular architecture or domain—would be as informative as a positive replication, because it would bound the principle's generality.

Disentangling whether the PRM's last-step aggregation benefit comes from representation learning or from the soft Monte Carlo training signal. The paper finds that PRMs with last-step aggregation outperform both ORMs and PRMs with min/prod aggregation (Appendix E, Figure 13), and hypothesizes that "the step-level PRM training acts as a form of beneficial representation learning even when the intermediate predictions aren't directly used at aggregation time." A follow-up could test this hypothesis directly by comparing: (a) a PRM trained with per-step Monte Carlo labels + last-step aggregation (the paper's best method), (b) an ORM trained with the same final-step labels but no intermediate supervision, and (c) a PRM trained with per-step binary labels (correct/incorrect) rather than soft Monte Carlo values, to isolate whether the benefit comes from the per-step training architecture (representation learning) or from the soft-label supervision signal. If (a) outperforms (b) but (c) does not, the benefit is architectural (step-level training forces useful intermediate representations). If (a) outperforms both (b) and (c), the benefit is in the soft-label signal (Monte Carlo rollouts provide better-calibrated targets than binary correctness labels). The authors note that their finding contradicts Lightman et al. (2023) and Wang et al. (2023), who found "min" to be best, and attribute this to the soft vs. binary label difference—an explicit ablation would resolve this discrepancy.

Testing whether the compute-optimal policy is actually the result of verifier over-optimization avoidance. The paper's compute-optimal policy routes easy problems away from aggressive search (using best-of-N instead of beam search) and routes medium problems toward search. The implicit mechanism is that search over-optimizes the verifier on easy problems (where the verifier is reliable but not perfect, so aggressive optimization amplifies residual errors) while providing genuine guidance on medium problems (where the verifier has more room to discriminate). A follow-up could test this mechanism explicitly by training PRMs of varying quality (e.g., by varying the amount of Monte Carlo rollout data, or by adding calibrated noise to labels) and measuring whether the optimal search strategy shifts predictably: with a weaker PRM, the over-optimization threshold should occur at lower budgets, and the compute-optimal policy should route more problems to best-of-N; with a stronger PRM, beam search should remain beneficial to higher budgets. If this relationship holds, it confirms that verifier quality improvement directly translates to compute-optimal scaling improvement, providing a clear research target.

Practical Applications and Downstream Use Cases

Cost-efficient batch inference with per-problem budget allocation. Organizations running large-scale batch inference on reasoning tasks—evaluating student answers, generating training data for self-improvement pipelines, or scoring candidate solutions—currently apply uniform compute budgets (e.g., best-of-256 for every problem) because they lack principled criteria for differential allocation. This paper's framework provides such a criterion: estimate each problem's difficulty relative to the base model's capability, then allocate a small budget (e.g., 4-8 sequential revisions) to easy problems, a medium budget (e.g., 32-64 generations of beam search) to medium problems, and the full budget or human escalation for hard problems. The paper's headline efficiency gain—matching best-of-256 performance with 64 generations through compute-optimal allocation (a 4× cost reduction, Figure 8)—translates directly to a 4× reduction in inference cost for organizations processing problems with a mix of difficulty levels. For a batch of 100,000 problems, the savings are substantial: 100,000 × 64 × (generation cost) versus 100,000 × 256 × (generation cost). The primary barrier to this deployment is the difficulty estimation cost, which the paper does not solve but which a learned difficulty predictor (see follow-up research above) could address.

Self-improvement data generation with edge-of-competence targeting. When using LLMs to generate training data for themselves—as in STaR (Zelikman et al., 2022), ReST^EM (Singh et al., 2024), or rejection sampling fine-tuning—the quality and diversity of generated solutions matter enormously. A naive approach applies best-of-N uniformly, wasting compute on problems the model already solves easily (generating redundant correct solutions) and problems it cannot solve at all (generating only incorrect solutions with no learning signal). The paper's edge-of-competence principle provides a principled filtering criterion: allocate the generation budget to problems where the base model's pass@1 is low but pass@k is non-zero—precisely the region where additional compute can flip incorrect attempts into correct solutions. These are the highest-value data points for fine-tuning because they represent problems the model can learn from (solvable with effort) but hasn't yet mastered. The paper's finding that RL on edge-competence data produces the largest pass@128 gains (Figure 3) directly supports this: the same principle that makes edge-competence data valuable for RL also makes it valuable for generating fine-tuning data. A self-improvement pipeline that uses difficulty estimation to identify edge-of-competence problems and then applies compute-optimal allocation to solve them would be more data-efficient than uniform sampling, producing a higher ratio of novel correct solutions to wasted compute.

On-device or edge deployment with adaptive compute. The paper's finding that a 100M-parameter model with compute-optimal test-time strategies can match a ~14× larger model on easy-to-medium problems (Section 7, Figure 9) has direct implications for deployment scenarios where running a large model is infeasible—mobile devices, embedded systems, latency-sensitive applications. In these settings, a small model with variable test-time compute can handle most routine queries locally, reserving cloud-based large-model inference only for genuinely hard problems that the small model cannot solve even with extensive test-time compute. The difficulty estimator serves double duty: it determines how much local compute to allocate AND whether to escalate to the cloud. For example, a math tutoring app running on a tablet could use a 100M-parameter model locally for most student questions (with adaptive budgets based on estimated difficulty), and only route problems where the difficulty estimator predicts bin 5 (hardest) to a cloud-hosted 7B-parameter model. The paper's numbers suggest this architecture would handle the majority of queries locally—difficulty bins 1-4, which represent ~80% of the MATH distribution—at substantially lower latency and zero cloud cost, while maintaining overall accuracy close to the cloud-only baseline. The primary engineering challenge is the cheap difficulty estimator, as above, but the conceptual framework is ready for implementation.

When to Prefer This Method

The paper does not propose a single "method" to prefer or reject—it provides a framework for analyzing when and why RL, mid-training, and pre-training investments pay off, and what conditions enable each to succeed. It does not position its framework against named alternatives (e.g., "use our compute-optimal policy instead of Chinchilla scaling"), nor does it claim that one training stage should be preferred over another in general. Instead, it establishes conditional guidance based on the deployment goal and the model's current capability profile. The paper's practical guidance (Section 3, Takeaway 1; Section 4, Practical Guidance 2; Section 5, Practical Guidance 3; Section 6, Practical Guidance 4) is already framed as decision rules keyed to specific conditions (e.g., "design RL data around the model's edge of competence," "seed long-tail primitives in pre-training at ~1% density," "allocate more mid-training for OOD-edge reliability, more RL for OOD-hard exploration"). Forcing these into a "Prefer X when Y" matrix would duplicate what the paper already states and risk implying a cleaner tradeoff than the evidence supports—the paper's own results show that the optimal mid-training/RL split shifts with compute budget (Appendix Figure 18), meaning there is no single fixed preference ordering. The conditional guidance is best left in the paper's own practical recommendation format, which accurately reflects the context-dependence of the findings.