rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

ArXiv: 2501.04519

🎯 Pitch

A 7B parameter model trained through self-play with Monte Carlo Tree Search scores 90% on MATH, beating OpenAI’s o1-preview without any distillation from larger models. The breakthrough comes from a process preference model trained on automatically discovered ranking pairs, plus code-augmented reasoning that filters out flawed intermediate steps through executable verification.

1. Executive Summary

rStar-Math introduces a self-evolved System 2 deep thinking approach that enables small language models (1.5B–7B parameters) to achieve frontier math reasoning without distillation from superior models, demonstrating that SLMs can rival or surpass OpenAI o1 on challenging competition benchmarks. The system uses Monte Carlo Tree Search (MCTS) where a math policy SLM generates step-by-step code-augmented chain-of-thought solutions — verified by Python execution to eliminate erroneous intermediate steps — guided by a process preference model (PPM) that provides step-level reward signals through pairwise ranking rather than direct score annotation, and both models are iteratively improved across four self-evolution rounds on 747k math problems. On MATH, rStar-Math boosts Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9% respectively, with the 7B model solving 53.3% (8/15) of AIME 2024 problems. The approach establishes that self-evolved MCTS deep thinking with process preference models enables SLMs to achieve o1-level math reasoning, with the PPM emerging as the dominant factor determining final System 2 performance once the policy model attains a reasonably strong capability level.

2. Context and Motivation

The Core Problem: Building Frontier Reasoners Without Frontier Models

The fundamental challenge rStar-Math addresses is deceptively simple to state but profound in its implications: can a small language model develop mathematical reasoning capabilities that match or exceed the best frontier models, using only its own efforts? The current paradigm for building advanced math reasoners relies on a straightforward but limiting recipe: take a powerful teacher model like GPT-4, generate large volumes of chain-of-thought solutions, and distill that knowledge into smaller models through supervised fine-tuning. This approach works — models fine-tuned on GPT-4 traces perform better than their base versions — but it imposes a hard ceiling: the student can never surpass its teacher. The reasoning capability is fundamentally bounded by what the frontier model can produce.

rStar-Math challenges this assumption. The paper asks: what if, instead of distilling from a superior model, an SLM could generate its own training data through systematic search and self-verification, progressively improving both its solution-generation ability and its solution-evaluation ability in a self-reinforcing cycle? This is not merely an academic curiosity. If self-evolution works, it means that reasoning capability is not something that must be "borrowed" from larger models via distillation — it can be grown from within, potentially even exceeding the capabilities of any single teacher model. This has direct implications for democratizing access to frontier reasoning: it suggests that organizations without the resources to train GPT-4-scale models could still build GPT-4-class reasoners by investing in smarter inference-time computation and self-training pipelines.

Why This Problem Matters Now

The urgency of this problem is driven by three converging trends that the paper engages with, implicitly or explicitly:

The diminishing returns of distillation-based data synthesis. The paper documents a worrying pattern in the math reasoning literature: scaling up GPT-4-distilled chain-of-thought data yields rapidly diminishing improvements. OpenMathInstruct-2, for example, saw only a 3.9% boost on MATH despite an 8× increase in dataset size. This saturation suggests that distillation alone cannot sustain progress toward frontier mathematical reasoning — more data from the same teacher model simply recycles the same patterns, the same blind spots, and the same error modes. Some genuinely new capability must come from somewhere, and that somewhere cannot be a static teacher model.

The test-time compute paradigm shift. A broader transformation is occurring in how the field thinks about LLM inference. The traditional approach — generate one complete answer in a single forward pass — is increasingly recognized as insufficient for complex reasoning tasks. This "System 1" style thinking is fast but error-prone, particularly for multi-step mathematical deduction where a single mistake anywhere in the chain propagates to produce a wrong answer. The emergence of test-time compute scaling, exemplified by OpenAI's o1 series, represents a shift toward "System 2" thinking: spend more computation at inference, systematically explore alternative solution paths, evaluate intermediate steps, and select the most promising trajectories. This paradigm creates new opportunities because the reasoning capability is no longer determined solely by the model's weights — it emerges from the interaction between the policy model and a reward model during search. rStar-Math positions itself squarely within this emerging paradigm, but with a crucial twist: both the policy and reward models are small, self-trained, and mutually reinforcing.

The unsolved process reward model problem. The System 2 paradigm requires reliable step-level evaluation — a process reward model (PRM) that can score intermediate reasoning steps to guide search. But training effective PRMs remains, as the paper puts it, "an open question." The fundamental difficulty is data: accurate step-by-step feedback requires knowing whether each intermediate step in a solution is correct, not just whether the final answer is right. This demands either expensive human annotation (as in PRM800k) or some automated method of assigning step-level quality scores. Existing automated approaches attempt to use Monte Carlo sampling or MCTS to derive step-level scores, but the resulting scores are inherently noisy — it's genuinely difficult to determine, even with extensive sampling, whether a particular step is "excellent" versus merely "good enough." The paper frames this imprecision as the primary bottleneck preventing effective PRM training, and it is the problem that their process preference model (PPM) is designed to circumvent.

Where Prior Approaches Fall Short

The paper identifies specific, consequential limitations in three categories of prior work. Understanding these limitations is essential because rStar-Math's design choices — code-augmented CoT, PPM instead of PRM, self-evolution — are direct responses to them.

GPT-distilled data synthesis (the dominant paradigm). The most common approach to building math-capable LLMs involves synthesizing chain-of-thought solutions using frontier models like GPT-4 and fine-tuning smaller models on these traces. Prominent examples include MetaMath and NuminaMath, which are themselves used as baselines in the paper's ablation studies. The paper identifies two fatal limitations with this approach:

First, and most obviously, the student cannot exceed the teacher. Hard problems that GPT-4 cannot solve are simply absent from the training data — the model never sees challenging examples that require novel reasoning strategies. This creates a capability ceiling that distillation cannot transcend. For Olympiad-level problems in particular, even GPT-4 solves only a fraction of the available problems, meaning large swaths of the difficulty spectrum remain unexplored in training.

Second, and more subtly, a correct final answer does not guarantee correct intermediate steps. The paper cites Lanham et al. (2023) on this point: LLMs frequently produce solutions that reach the right answer through reasoning that contains errors, irrelevant steps, or logical gaps. When these trajectories are used as training data, the model learns to reproduce not just correct reasoning patterns but also the spurious correlations and lucky guesses that happened to work on particular problems. This degrades data quality in ways that are difficult to detect, because conventional evaluation (checking the final answer) does not reveal intermediate errors.

Rejection sampling — generate many solutions and keep only those with correct final answers — partially addresses this by filtering out completely wrong solutions, but it does not guarantee step-level correctness. A solution can pass rejection sampling while containing flawed intermediate reasoning.

Automated process supervision (attempting to train PRMs). Recognizing the value of step-level feedback, several recent works have attempted to train process reward models using automatically generated step-level scores. The general approach, exemplified by Math-Shepherd and AlphaMath, uses MCTS or Monte Carlo rollouts to estimate the "goodness" of each step — roughly, what fraction of completions from that step reach the correct answer. These Q-values are then used as direct training targets for the PRM, typically with mean squared error loss.

The paper argues this approach is fundamentally flawed because these automatically generated Q-values are not precise enough to serve as direct training labels. Even with extensive MCTS rollouts, it is extremely difficult to distinguish fine gradations of step quality. Is a step with Q-value 0.7 meaningfully better than one with Q-value 0.65? The noise in these estimates means that forcing the PRM to regress to specific scalar values introduces training signal that is at best uninformative and at worst actively misleading. The paper's key insight here — which motivates the PPM design — is that while Q-values may not be precise enough for direct score prediction, they are reliable enough to establish preference ordering: steps with high Q-values are generally better than steps with low Q-values, even if the exact values are noisy. This shifts the learning problem from regression (predict the exact Q-value) to ranking (predict which of two steps is better), which is more robust to annotation noise.

The paper also notes that existing PRM training approaches using human-annotated data, such as PRM800k, face fundamental scalability limitations. Expert human annotation of step-level correctness for thousands of problems is expensive, slow, and suffers from inter-annotator inconsistency — even experts disagree on whether particular intermediate steps are correct or sufficient. This makes human annotation impractical for the scale of data needed to train a robust PRM.

Existing test-time compute scaling with open-source models. Several recent works have applied test-time compute scaling to open-source LLMs for math reasoning, including approaches using Best-of-N sampling with outcome reward models, self-consistency through majority voting, and MCTS-guided search. The paper acknowledges these efforts but argues they have shown "limited gains in math reasoning, often due to policy LLM or reward model limitations." The underlying problem is circular: to do effective System 2 reasoning, you need both a policy model capable of generating promising candidate steps and a reward model capable of evaluating them. But building either requires high-quality training data, and generating that data requires — you guessed it — a capable policy model and reliable reward model. Prior open-source efforts get stuck in this chicken-and-egg problem: their policy models are too weak to generate good training data, and their reward models are too noisy to guide effective search, so the System 2 gains are modest.

The paper's diagnosis is that this deadlock cannot be broken by better search algorithms alone — it requires a self-evolution process where both models improve iteratively, each round's improvements enabling higher-quality data for the next round.

How rStar-Math Positions Itself

rStar-Math positions itself not as yet another distillation method, nor as a better search algorithm for an existing policy model, but as a complete self-evolution framework that addresses the root causes of prior limitations. The positioning can be understood along four axes:

Against teacher distillation: self-generation, not distillation. Rather than depending on a superior model to provide training data, rStar-Math uses MCTS to have the SLM generate its own solutions and verify them through code execution. The paper explicitly demonstrates that, after self-evolution, rStar-Math's policy SLM can generate code-augmented CoT solutions that are comparable to or better than GPT-4-distilled datasets (MetaMath, NuminaMath) when used as SFT training data, even without verification. With verification, they substantially outperform these baselines. This establishes that self-generation can match or exceed distillation — a crucial claim because it means the capability ceiling imposed by teacher models can be broken.

Against direct PRM training: preferences, not scores. The paper introduces the process preference model as a deliberate departure from the direct score prediction paradigm. Instead of trying to train a model to output precise step-level reward values, the PPM learns to rank steps: given two candidate next steps with the same prefix, predict which is better. This is formalized through the Bradley-Terry pairwise ranking loss, the same approach used in RLHF for training reward models from human preferences. The crucial difference from prior automated PRM work is that the training signal — Q-value-derived preferences — is robust to the noise that makes direct Q-value regression fail. The paper shows this empirically: the PPM significantly outperforms a Q-value-based PRM (which they call PQM) on challenging benchmarks, with the gap widening on harder problems.

Against the chicken-and-egg problem: iterative self-evolution. The four-round self-evolution recipe is not just an optimization — it is the mechanism that escapes the circular dependency between policy and reward model quality. The process is carefully staged: Round 1 bootstraps an initial policy model using a larger model (DeepSeek-Coder-V2-Instruct, 236B) for MCTS data generation, providing a strong enough starting point. Round 2 uses this improved policy model to generate higher-quality data through more extensive MCTS rollouts, training the first reliable PPM. Round 3 integrates the PPM into MCTS itself (PPM-augmented MCTS), which generates significantly better trajectories, which train a better policy model and PPM. Round 4 pushes further with additional computational effort on the hardest problems. Each round's improvements are documented quantitatively: policy SLM pass@1 on MATH rises from 58.8% (base) → 69.6% (r1) → 73.6% (r2) → 75.8% (r3) → 78.4% (r4), while the fraction of Olympiad problems solved in the training set jumps from 20.99% (r1) to 80.58% (r4).

Within the System 2 paradigm: a practical realization for SLMs. The paper positions rStar-Math as demonstrating that effective System 2 reasoning is not the exclusive province of frontier models. The key design choices — using SLMs to enable extensive MCTS rollouts on accessible hardware (4×40GB A100 GPUs), code execution to mitigate hallucination in intermediate steps, preference-based reward modeling to handle annotation noise — are all oriented toward making self-evolved deep thinking practical at small scale. The explicit comparison to Best-of-N baselines that use 10× larger reward models (Qwen2.5-Math-RM-72B) highlights this: rStar-Math achieves superior results using a 7B PPM, suggesting that the architecture matters more than raw scale.

The paper also connects to a broader vision that extends beyond math. The self-evolution framework requires, at its core, a mechanism for determining whether a generated trajectory reaches a correct answer — which in math comes from ground-truth labels and code execution verification. For other domains, the paper suggests that analogous feedback could come from test cases (code), human labeling, or mutual verification between LLMs. This positions rStar-Math not as a math-specific solution but as an instance of a general methodology for self-improving reasoning systems, with mathematics serving as the proving ground because it offers the cleanest correctness signals.

3. Technical Approach

3.1 Reader Orientation

rStar-Math is a self-improving System 2 reasoning system where a small language model learns to solve math problems through a structured search process — generating step-by-step solutions, checking each step with code execution, evaluating progress with a learned reward model, and iteratively refining both the solver and the evaluator using their own outputs. The system addresses the fundamental chicken-and-egg problem of building strong reasoners without strong teacher models: to train a good policy model you need high-quality training data, but to generate high-quality training data you need a good policy model. The "shape" of the solution is a four-round self-evolution loop where each round uses Monte Carlo Tree Search to generate slightly better training data than the previous round, progressively bootstrapping both models from weak starting points to frontier-level capability.

3.2 Big-Picture Architecture (Diagram in Words)

The rStar-Math system consists of five major components organized into a cyclical training pipeline and a test-time inference procedure:

1. Math Policy SLM — a small language model (1.5B–7B parameters) that serves as the action generator in MCTS. At each step of problem-solving, it produces candidate next steps, each containing natural language reasoning embedded as Python comments alongside executable Python code. This is the model that ultimately solves problems at test time.

2. Process Preference Model (PPM) — a separately trained reward model (7B parameters) that scores partial solution trajectories. Given a problem and a sequence of solution steps up to some point, it outputs a scalar between -1 and 1 indicating how promising that partial solution is. Unlike traditional PRMs trained to predict exact Q-values, the PPM is trained with pairwise ranking to distinguish good steps from bad ones.

3. Monte Carlo Tree Search (MCTS) Engine — the search procedure that orchestrates problem-solving. It maintains a tree where nodes are reasoning steps, uses the policy SLM to expand candidate child nodes at each step, filters invalid generations via Python code execution, scores surviving candidates with the PPM, and uses the Upper Confidence Bound for Trees (UCT) formula to balance exploration and exploitation when selecting which node to expand next.

4. Code Execution Verifier — a Python interpreter that executes the concatenated code from all steps in a partial trajectory. Any candidate step whose code fails to execute (raising a SyntaxError, NameError, TypeError, etc.) is immediately discarded. This provides dense, automatic verification of intermediate step correctness without requiring any human labels or learned verifiers.

5. Self-Evolution Training Pipeline — the outer loop that runs four rounds. In each round, the current policy SLM and PPM are used to run MCTS on 747k math problems, generating step-by-step verified reasoning trajectories annotated with per-step Q-values. These trajectories are then filtered (by Q-value, by correctness, by code execution success) and used to fine-tune a stronger policy SLM and train a stronger PPM for the next round.

The information flow at inference time is: a math problem enters → MCTS initializes a root node → at each step, the policy SLM generates candidate next steps → Python execution filters invalid candidates → PPM scores surviving candidates → UCT selects the best node for expansion → back-propagation updates Q-values → the process repeats until terminal nodes are reached → the trajectory with the highest PPM score is selected as the final answer. During training, the flow is: math problems enter → MCTS generates trajectories with Q-values → Q-values are used to select high-quality trajectories (top-2 per problem by average Q-value among correct solutions) → these trajectories SFT the policy SLM → Q-values also construct per-step preference pairs (high-Q steps vs. low-Q steps) → these pairs train the PPM via pairwise ranking loss → the improved models are used in the next round's MCTS to generate better data.

3.3 Roadmap for the Deep Dive

• First, the code-augmented CoT generation mechanism (Section 3.2 of the paper), because this is the fundamental unit of all reasoning in rStar-Math — every candidate step in MCTS is a code-augmented CoT, and the verification it provides is what makes self-evolution possible without human supervision.
• Second, the MCTS procedure and Q-value annotation (final parts of Section 3.2), since MCTS is the search framework that generates all training data and performs test-time inference, and understanding how Q-values are computed and updated is prerequisite knowledge for the PPM training.
• Third, the Process Preference Model (Section 3.3), which is the paper's primary methodological innovation — substituting pairwise ranking for direct score regression, and why that matters given the noise properties of MCTS-derived Q-values.
• Fourth, the self-evolution recipe (Section 3.4), which ties everything together into the four-round cycle, including how training data is selected from MCTS output, how policy model SFT works, how PPM training data is constructed, and what specifically improves in each round.
• Fifth, a brief synthesis of design choices and their justifications, collecting the reasoning behind critical decisions (why code-augmented, why PPM over PRM, why four rounds, why 16 rollouts, etc.) that are distributed throughout the methodology sections.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that small language models can achieve frontier math reasoning through self-evolved MCTS deep thinking, where a code-augmented CoT synthesis method generates verified step-by-step reasoning trajectories, a process preference model provides reliable step-level reward signals through pairwise ranking rather than direct score prediction, and a four-round self-evolution cycle progressively bootstraps both models from an initial weak state to o1-competitive capability.

Code-Augmented Chain-of-Thought Generation

The paper introduces a novel code-augmented CoT generation method that is the atomic unit of all reasoning in rStar-Math. This method is motivated by a well-documented failure mode of language models: they frequently produce solutions that reach the correct final answer through reasoning that contains hallucinated, incorrect, or logically irrelevant intermediate steps. Detecting these flawed steps is challenging because the final answer alone does not reveal intermediate errors, and natural language verification (having another LLM check the reasoning) is itself unreliable.

The code-augmented CoT method addresses this by coupling every reasoning step with executable Python code. As shown in Figure 2 of the paper, for a given step $i$ of a solution to problem $x$, the policy model generates a candidate step $s_{i,j}$ (the $j$-th candidate at step $i$) that contains two integrated components: a natural language chain-of-thought explanation embedded as Python comments (lines beginning with #), and the corresponding Python code that implements the computation described in those comments. The natural language CoT is inside the code block, formatted as comments, so that the entire generation — reasoning and computation — is a single code block that can be executed.

For example, from Figure 2, a step for calculating total distance walked south might generate:

# Step 1: Calculate the total distance walked south
total_south = 1/2 + 1/2

The policy model is prompted (using templates shown in Appendix A.3) to follow this format consistently, with each step ending in a special <end_of_step> token that signals the boundary between steps.

Verification by concatenated execution. The critical design choice is that each candidate step $s_{i,j}$ is not executed in isolation — it is concatenated with the Python code from all previous steps to form $s_1 \oplus s_2 \oplus \dots \oplus s_{i-1} \oplus s_{i,j}$, where $\oplus$ denotes string concatenation, and this entire accumulated code block is executed in a Python interpreter. This means that step $s_{i,j}$ can reference variables defined in earlier steps (e.g., total_south from step 1 can be used in step 3), and any inconsistency — referencing an undefined variable, producing a type error, importing a package not yet imported — will cause the execution to fail.

Only candidate steps whose concatenated code executes successfully (without raising any exception) are retained as valid nodes in the MCTS tree. Steps that fail execution are discarded entirely. The paper states:

"Only generations with successfully executed Python code are retained as valid candidates."

This code execution filter serves as a dense, automatic verifier for intermediate reasoning quality. It cannot guarantee that the reasoning is mathematically sound — it is possible to write code that executes successfully but computes the wrong thing — but it catches a large class of errors: undefined variables, syntax mistakes, type mismatches, and logical inconsistencies that manifest as runtime errors. The paper's ablation study (Table 7) provides indirect evidence for the value of this verification: even randomly sampled code-augmented CoT solutions from the self-evolved policy model yield SFT training data that produces comparable or better downstream performance than carefully curated GPT-4-distilled datasets (MetaMath and NuminaMath-CoT), suggesting that the code execution filter itself elevates data quality substantially.

Why code-augmented CoT over pure natural language CoT? Prior MCTS approaches for math reasoning, such as rStar (Qi et al., 2024) and ReST-MCTS* (Zhang et al., 2024a), primarily generate natural language CoTs without executable verification. The paper argues that this leaves intermediate step errors largely undetectable. Natural language CoTs can contain plausible-sounding but incorrect reasoning that is difficult to automatically verify. Code execution provides an objective, deterministic filter: the code either runs or it doesn't. This is particularly important for the self-evolution setting, where the training data is generated by SLMs that are more prone to hallucination than large frontier models — without code execution filtering, the training data would contain substantially more erroneous intermediate steps, degrading the quality of both the policy model fine-tuning and the reward model training.

The paper also notes a practical benefit: the code-augmented format naturally produces step-by-step trajectories, since each <end_of_step> marker delineates a natural boundary for MCTS node expansion, Q-value annotation, and preference pair construction.

Monte Carlo Tree Search with Q-Value Annotation

MCTS is the search framework that generates all training data and performs test-time inference in rStar-Math. The paper adopts MCTS over alternatives like Best-of-N or self-consistency for two stated reasons:

1. Decomposition of difficulty: MCTS breaks complex math problems into simpler single-step generation tasks. Instead of requiring the policy model to generate a complete multi-step solution in one inference (which is challenging, especially for SLMs), the model only needs to generate one reasoned step at a time, conditioned on the partial solution so far. This incremental generation reduces the cognitive burden on the policy SLM.

2. Natural step-level annotation: The tree structure of MCTS, combined with back-propagation of terminal outcomes, automatically produces Q-values for each intermediate step based on that step's contribution to reaching correct answers. This eliminates the need for human-generated step-level labels.

The MCTS procedure. Starting from a root node representing the math problem $x$, rStar-Math runs multiple MCTS iterations, each consisting of four standard phases — selection, expansion, simulation/rollout, and back-propagation — adapted for the code-augmented setting:

Selection: At step $i$, the algorithm has a partial trajectory $x \oplus s_1 \oplus s_2 \oplus \dots \oplus s_{i-1}$ representing the path from the root to the current node. To select which child node to explore next, it uses the Upper Confidence Bound for Trees (UCT) formula:

$UCT(s) = Q(s) + c \sqrt{\frac{\ln N_{parent}(s)}{N(s)}}$

where $Q(s)$ is the average reward (Q-value) of node $s$, computed as the total accumulated Q-value of $s$ divided by $N(s)$ (the number of times node $s$ has been visited), $N_{parent}(s)$ is the visit count of $s$'s parent node, and $c$ is an exploration constant set to $2$ in all experiments (to promote greater exploration, as stated in Appendix A.1).

What this equation computes: UCT assigns a score to each candidate child node $s$ that balances two competing objectives — exploitation (favoring nodes with high average reward $Q(s)$, which have historically led to good outcomes) and exploration (favoring nodes with low relative visit counts $N(s)$ compared to their parent, represented by the second term which grows as $\ln N_{parent}(s) / N(s)$ increases). The child node with the highest UCT score is selected for expansion. This is the standard UCT formula from Kocsis and Szepesvári (2006), used without modification.

Why this form: UCT is the de facto standard for MCTS node selection because it provably balances exploration and exploitation in a way that converges to the optimal policy given infinite rollouts. Alternatives like pure exploitation (always selecting the highest-Q child) would risk missing novel solution paths, while pure exploration (uniform random selection) would waste compute on unpromising branches. The exploration constant $c = 2$ is larger than typical values (often $\sqrt{2} \approx 1.414$), chosen to compensate for the weaker policy model's tendency to generate low-diversity candidates — higher exploration prevents premature convergence to suboptimal solution paths.

Expansion: Once a node is selected for expansion, the policy SLM generates $n$ candidate next steps $s_{i,0}, s_{i,1}, \dots, s_{i,n-1}$ by prompting it with the problem and the partial trajectory so far. The number of candidates $n$ is configurable: in the bootstrap round it is 5 (due to the larger model size), in rounds 2–3 it is 8, and in round 4 it is increased to 16 for challenging problems. Each candidate is filtered through code execution — only those whose concatenated code executes successfully become new child nodes. The surviving candidates are then scored by the PPM (or, in rounds 1–2 when the PPM is unavailable, assigned an initial Q-value of 0 for terminal-guided annotation). The selected child node becomes the new current node for the rollout phase.

Rollout and back-propagation: From the newly expanded node, the process continues: more candidate steps are generated, filtered, scored, and selected, building out the tree until a terminal node $s_d$ is reached. A terminal node is defined as one where the policy model generates a final answer (indicated by the <answer> tag in the generation format). At this point, the terminal node receives a reward: $q(s_d) = 1$ if the final answer matches the ground-truth answer, and $q(s_d) = -1$ otherwise. This terminal reward is then back-propagated up the tree, updating the Q-values of all ancestor nodes along the path.

Q-value annotation — Terminal-guided (Rounds 1–2). In the first two self-evolution rounds, when the PPM is either unavailable (Round 1) or insufficiently accurate (Round 2), the paper uses terminal-guided annotation. The Q-value of each intermediate step is updated based on the terminal outcome:

$q(s_i)_k = q(s_i)_{k-1} + q(s_d)_k$

where $q(s_i)_k$ is the accumulated Q-value of step $s_i$ after the $k$-th rollout (back-propagation), $q(s_i)_{k-1}$ is its accumulated Q-value before this rollout, and $q(s_d)_k$ is the terminal node's reward ($+1$ for correct, $-1$ for incorrect) from this rollout. The initial Q-value $q(s_i)_0$ is 0 in the first rollout.

What this equation computes: Each time a complete trajectory is generated, the terminal outcome (correct or incorrect) is added to the running Q-value sum of every intermediate step in that trajectory. Steps that appear in many trajectories that ultimately reach correct answers accumulate large positive Q-values; steps that appear in trajectories leading to incorrect answers accumulate negative Q-values; steps that appear in a mix of correct and incorrect trajectories end up with intermediate Q-values reflecting their empirical reliability.

Why this form: This is the standard MCTS back-propagation rule, following AlphaGo (Silver et al., 2017) and rStar (Qi et al., 2024). The additive form means Q-values are linear in the number of correct completions, making them interpretable as a running score rather than a probability. The choice of $+1/-1$ for terminal rewards creates a symmetric penalty for incorrect answers, equally weighting the evidence from successes and failures. An alternative would be $+1/0$ (reward correct, no penalty for incorrect), but this would mean steps appearing in many incorrect trajectories could still accumulate positive Q-values if they occasionally lead to correct answers — the symmetric penalty ensures that only steps that reliably contribute to correct answers maintain positive Q-values.

The actual Q-value used in UCT selection (in Eq. 1) is the average: $Q(s) = q(s) / N(s)$, where $q(s)$ is the accumulated sum and $N(s)$ is the visit count.

Q-value annotation — PPM-augmented (Rounds 3–4). Starting from Round 3, when a reliable PPM (PPM-r2) is available, the paper switches to PPM-augmented annotation. Instead of initializing each new node's Q-value to 0, the PPM directly predicts an initial score:

$q(s_i)_0 = PPM(x \oplus s_1 \oplus s_2 \oplus \dots \oplus s_{i-1} \oplus s_i)$

where $PPM(\cdot)$ is the process preference model, which takes the problem and the partial trajectory up through step $s_i$ and outputs a scalar in $[-1, 1]$ representing the predicted quality of this partial solution.

What this equation computes: The PPM provides a learned prior estimate of step quality before any rollouts have explored completions from this step. This initial $q(s_i)_0$ is then updated through subsequent back-propagation using Eq. 2 (the terminal-guided update, $q(s_i)_k = q(s_i)_{k-1} + q(s_d)_k$), so the PPM's prediction serves as a starting point that is gradually refined by empirical evidence from rollouts.

Why this form: Terminal-guided annotation requires multiple rollouts before Q-values become informative — a step that appears in only one trajectory has a Q-value of $+1$ or $-1$, which is a noisy estimate. The PPM provides a non-zero initial value based on learned patterns, effectively giving the search a "warm start." This is particularly important for challenging problems where only a few trajectories reach correct answers — without the PPM's prior, UCT would struggle to distinguish promising from unpromising steps in the early rollouts. The combination of learned prior plus empirical updates means the search benefits from both the PPM's generalization (from training on many problems) and the specific evidence accumulated for the current problem.

Terminal node scoring during training data generation. An important detail: during training data generation (as opposed to test-time inference), terminal nodes $s_d$ are not scored by the PPM. Instead, they are scored based on ground-truth answer matching: $q(s_d) = +1$ for correct, $q(s_d) = -1$ for incorrect. The paper states:

"For terminal node $s_d$, we do not use PRM for scoring during training data generation. Instead, we assign a more accurate score based on ground truth labels as terminal-guided rewarding."

This is a deliberate choice: ground-truth labels provide a noise-free terminal reward signal, which is more reliable than the PPM's prediction for determining whether a complete solution is correct. The PPM is used for scoring intermediate steps (where no ground-truth exists and the PPM's generalization is needed), while the terminal reward comes from the objective answer check.

Extensive rollouts for Q-value reliability. The paper emphasizes the importance of running many MCTS rollouts to obtain reliable Q-values. The default is 16 rollouts per problem in rounds 2–4 (8 in the bootstrap round due to the larger model's higher computational cost). After 16 rollouts, each node in the tree has been visited multiple times, and its Q-value reflects a meaningful empirical estimate of how often paths through that node lead to correct answers. The paper draws an explicit analogy to Go players who "retrospectively evaluate the reward of each move based on game outcomes" — individual games provide noisy feedback, but repeated play refines evaluations.

In Round 4, for problems that remain unsolved after 16 rollouts, the paper performs additional rollouts (64 total, and if still unsolved, up to 128) and conducts multiple independent MCTS tree expansions with different random seeds. This targeted additional computation is key to increasing the Olympiad-level problem coverage from 62.16% (Round 3) to 80.58% (Round 4).

Process Preference Model (PPM)

The Process Preference Model is the paper's most significant methodological innovation — a departure from the standard process reward model (PRM) paradigm that addresses the fundamental difficulty of obtaining precise step-level reward annotations.

The problem with direct Q-value regression. In prior work on automated PRM training (e.g., Math-Shepherd, AlphaMath, ReST-MCTS*), MCTS-derived Q-values are used as direct training targets. The PRM is trained to predict, for each step $s_i$ in a trajectory, a scalar value matching the Q-value $Q(s_i)$, typically using mean squared error loss:

$\mathcal{L}_{PRM}(x) = \|Q - Q'\|^2$

where $Q = (Q(s_1), Q(s_2), \dots, Q(s_d))$ is the vector of annotated Q-values and $Q' = (Q'(s_1), Q'(s_2), \dots, Q'(s_d))$ is the PRM's predicted Q-values for all steps in the trajectory.

The paper argues this approach is fundamentally limited because the annotated Q-values are inherently imprecise. Even with 16 rollouts, the Q-value of a step is a noisy estimate — the paper gives the example that within a set of correct steps, it is extremely difficult to reliably rank them as "best," "second-best," or "average," and within a set of incorrect steps, distinguishing "worst" from "moderately poor" is equally challenging. Forcing the model to regress to these noisy scalar targets introduces training signal that is, at best, uninformative about fine-grained quality distinctions and, at worst, actively misleading.

The PPM solution: pairwise preference learning. Rather than predicting exact Q-values, the PPM learns to perform a simpler and more robust task: ranking. Given a problem $x$ and a partial trajectory up to step $i-1$, the PPM must predict which of two candidate next steps $s_i^{pos}$ and $s_i^{neg}$ is better. The training signal comes from the same MCTS Q-values, but instead of using the numerical values directly, the paper uses them to select positive and negative examples and trains with a pairwise ranking loss.

Construction of preference pairs. For each step position $i$ in a trajectory, the paper selects preference pairs from the MCTS tree as follows:

For intermediate steps (not the final answer step): for a given partial trajectory $x \oplus s_1 \oplus \dots \oplus s_{i-1}$, the MCTS tree contains multiple candidate next steps that were explored from this prefix. The paper selects two candidates with the highest Q-values as positive examples ($s_i^{pos}$) and two candidates with the lowest Q-values as negative examples ($s_i^{neg}$). Crucially, the selected positive steps must lead to trajectories that ultimately reach the correct final answer, and the selected negative steps must lead to trajectories that reach incorrect final answers. This ensures that the preference signal is grounded in objective outcomes, not just noisy Q-value estimates.

For the final answer step (where the trajectory is complete): it is rare for two trajectories sharing the exactly same prefix to diverge only at the final answer, so the paper relaxes the prefix-matching constraint. Instead, it selects two complete correct trajectories with the highest average Q-values (averaged across all their steps) as positive examples, and two complete incorrect trajectories with the lowest average Q-values as negative examples. The preference pair compares the full trajectories: $y_i^{pos} = s_1 \oplus \dots \oplus s_d^{pos}$ versus $y_i^{neg} = s_1 \oplus \dots \oplus s_d^{neg}$.

For each problem that has both correct and incorrect trajectories in its MCTS output (problems where all trajectories are correct or all are incorrect are excluded from PPM training), this construction yields multiple preference pairs — one per step in the trajectory.

The PPM training objective. The PPM is trained using the standard Bradley-Terry model with a pairwise ranking loss, following the same formulation used in RLHF (Ouyang et al., 2022):

$\mathcal{L}_{PPM}(\theta) = -\frac{1}{2 \times 2} \mathbb{E}_{(x, y_i^{pos}, y_i^{neg}) \in \mathcal{D}} \left[\log\left(\sigma\left(r_\theta(x, y_i^{pos}) - r_\theta(x, y_i^{neg})\right)\right)\right]$

where $\mathcal{D}$ is the dataset of preference pairs, $r_\theta(x, y)$ is the scalar output of the PPM with parameters $\theta$ for problem $x$ and partial trajectory $y$ (constrained to the range $[-1, 1]$ via a tanh activation), and $\sigma(\cdot)$ is the logistic sigmoid function. The positive trajectory $y_i^{pos}$ and negative trajectory $y_i^{neg}$ share the same prefix up to step $i-1$:

$y_i^{pos} = s_1 \oplus \dots \oplus s_{i-1} \oplus s_i^{pos}$ $y_i^{neg} = s_1 \oplus \dots \oplus s_{i-1} \oplus s_i^{neg}$

What this equation computes: For each preference pair $(y_i^{pos}, y_i^{neg})$, the PPM produces two scalar scores $r_\theta(x, y_i^{pos})$ and $r_\theta(x, y_i^{neg})$. The difference $r_\theta(x, y_i^{pos}) - r_\theta(x, y_i^{neg})$ is passed through the sigmoid function $\sigma(z) = 1/(1 + e^{-z})$, which maps the difference to $(0, 1)$. The log of this sigmoid-transformed difference is the log-likelihood that the positive example is preferred over the negative example under the Bradley-Terry model. The loss is the negative of this log-likelihood, averaged over all preference pairs (the $1/(2 \times 2)$ factor accounts for the fact that each step has two positive and two negative examples, yielding four pairwise comparisons per step). Minimizing this loss encourages the PPM to assign higher scores to positive examples than to negative examples.

Why this form: The Bradley-Terry loss is the natural objective for pairwise preference data — it models the probability that one item is preferred over another as a function of the difference in their latent scores. The key advantage over MSE regression on Q-values is robustness to annotation noise. The Q-values need only be reliable enough to correctly identify which of two steps is better, not to assign the "correct" numerical score to each. The paper argues that even with noisy Q-values from limited rollouts, the ordering (highest-Q steps vs. lowest-Q steps) is much more reliable than the exact values, because systematic biases in Q-value estimation tend to affect both candidates similarly when they share the same prefix. The tanh activation constraining outputs to $[-1, 1]$ provides a bounded score range that matches the terminal reward scale and prevents the model from producing unbounded scores that could destabilize training or UCT selection.

PPM architecture and initialization. The PPM is initialized from the fine-tuned policy SLM (not the base model), with its next-token prediction head (the language modeling head) replaced by a scalar-value head consisting of a linear layer followed by a tanh activation function. This initialization strategy means the PPM starts with the policy model's understanding of math reasoning and solution structure, which provides a strong prior for evaluating solution quality. The PPM is trained for 1 epoch with a batch size of 512 and an initial learning rate of $7 \times 10^{-6}$.

Why initialize from the policy model? This is a common design pattern in reward model training: the reward model benefits from the policy model's representation of the problem domain. Since the PPM needs to evaluate the quality of partial solutions generated by the policy model, starting from the policy model's weights ensures that the PPM's internal representations are aligned with the distribution of solutions it will evaluate. Training from scratch or from a different base model would require the PPM to learn both the domain representation and the evaluation function simultaneously, which would be less sample-efficient.

PPM at inference time. At test-time inference, the PPM is used within MCTS to score candidate nodes: each generated candidate step $s_{i,j}$ receives a score $r_\theta(x, s_1 \oplus \dots \oplus s_{i-1} \oplus s_{i,j})$ from the PPM, which serves as the initial Q-value for that node. The PPM also serves as the final answer selector: after MCTS completes (with 4 rollouts per step at test time, as stated in Appendix A.1), the trajectory with the highest PPM score is selected as the final answer. The paper notes that because the PPM "effectively provides step-level quality evaluations," MCTS is limited to just 4 rollouts per step at inference time — far fewer than the 16 used during training data generation — making inference significantly cheaper.

Empirical evidence for PPM over PQM. The paper includes a direct comparison (Table 8) between the PPM and a Q-value-based PRM (which they call PQM, for Process Q-value Model). The PQM is trained with MSE loss to predict the exact Q-values, using the same training data (Round 4 trajectories) and the same model architecture. On MATH, PPM achieves 89.4% vs. PQM's 88.2%; on AIME 2024, 50.0% vs. 46.7%; on Olympiad Bench, 65.3% vs. 62.9%. The gap is largest on the most challenging benchmarks, consistent with the paper's argument that Q-value imprecision is most damaging on hard problems where rollouts are sparse and Q-value estimates are noisiest.

Self-Evolution Recipe

The self-evolution recipe is the outer loop that organizes the four rounds of iterative improvement. Each round follows the same general pattern: use the current policy model and PPM to run MCTS on 747k math problems, generate step-by-step verified reasoning trajectories, select high-quality trajectories for supervised fine-tuning of a new policy model, construct preference pairs for training a new PPM, and deploy both improved models in the next round.

Math problem collection. The training set consists of 747k math word problems with ground-truth final answers, sourced primarily from two public datasets: NuminaMath (competition-level problems only — Olympiads, AIME, AMC) and MetaMath. The paper explicitly states that grade-school-level problems from NuminaMath are excluded because they "do not significantly improve LLM complex math reasoning." To augment the limited competition-level problems, the paper follows Li et al. (2024) in using GPT-4 to synthesize new problems based on seed problems from 7.5k MATH training problems and 3.6k AMC-AIME training problems. However, the paper acknowledges that "GPT-4 often generated unsolvable problems or incorrect solutions for challenging seed problems." To filter these, they prompt GPT-4 to generate 10 solutions per synthetic problem and retain only those with at least 3 consistent solutions. This filtering is applied to reduce noise in the training set, and further filtering during self-evolution removes synthetic problems whose generated trajectories achieve less than 50% accuracy.

Round 1: Bootstrapping an initial policy model (SLM-r1). The first round addresses the cold-start problem: the base model (Qwen2.5-Math-7B, with 58.8% on MATH) is too weak to self-generate high-quality training data through MCTS. The solution is to bootstrap using a larger, more capable model: DeepSeek-Coder-V2-Instruct (236B parameters). This larger model is used as the policy model within MCTS to generate trajectories on the 747k problems.

Key Round 1 parameters: 8 MCTS rollouts per problem (reduced from the standard 16 for computational efficiency with the 236B model), 5 candidate nodes per step, terminal-guided Q-value annotation (no PPM exists yet). Only trajectories leading to correct answers are retained. For each problem with at least one correct trajectory, the top-2 trajectories with the highest average Q-values (averaged across all steps in the trajectory) are selected as SFT training data for the policy model. The paper states:

"we run MCTS with DeepSeek-Coder-V2-Instruct (236B) to collect the SFT data."

The collected trajectories are used to fine-tune Qwen2.5-Math-7B (the base model), producing policy SLM-r1. The model is fine-tuned for 2 epochs with a sequence length of 4096 tokens, batch size 128, and AdamW optimizer with a linear learning rate scheduler starting at $7 \times 10^{-6}$. The paper notes that the policy model is fine-tuned from the initial base model in each round, rather than training incrementally on the model from the previous round — this is based on their "extensive experiments" showing better results from re-initializing.

A PPM (PPM-r1) is also trained in Round 1, but the paper reports that its effectiveness is limited due to the small number of rollouts (8), which yields unreliable Q-values for preference pair construction. This PPM is effectively discarded — the paper uses terminal-guided annotation for Round 2 MCTS.

Round 2: Training the first reliable PPM (PPM-r2). With policy SLM-r1 now available (69.6% on MATH, up from 58.8%), Round 2 uses this improved policy model to run MCTS with more extensive rollouts: 16 per problem, with 8 candidate nodes per step. The increased rollouts produce more reliable Q-values, enabling the training of the first effective PPM.

The training data construction in Round 2 follows the procedures described in Sections 3.2 and 3.3: for each problem, run MCTS with policy SLM-r1 and terminal-guided annotation → collect all trajectories and their per-step Q-values → for problems with both correct and incorrect trajectories, construct step-level preference pairs by selecting high-Q positive steps and low-Q negative steps → train PPM-r2 using the Bradley-Terry loss in Eq. 4.

For SFT data, the same selection rule applies: for each problem, select the top-2 trajectories with the highest average Q-values among correct solutions as training data for policy SLM-r2.

The results of Round 2 are substantial. PPM-r2 achieves 84.1% on MATH (up from 75.2% for PPM-r1, though note these numbers are measured with policy SLM-r1 fixed for fair comparison). Policy SLM-r2 improves to 73.6% on MATH (up from 69.6%). Critically, the Olympiad-level problem coverage in the training set jumps from 20.99% (Round 1) to 56.04% (Round 2), meaning many more challenging problems now have at least one correct trajectory and can be included in training.

Round 3: PPM-augmented MCTS for higher-quality data. This round introduces a qualitative change in the data generation process: instead of terminal-guided annotation, MCTS uses PPM-augmented annotation (Eq. 3). The PPM (PPM-r2) scores each candidate step when it is generated, providing an informed prior that guides the search toward promising solution paths.

The paper reports that this shift to PPM-augmented MCTS produces "significantly higher-quality trajectories." The quantitative evidence: MATH-level problem coverage rises from 67.40% (Round 2) to 88.69% (Round 3) in the training set, and Olympiad-level coverage rises from 56.04% to 62.16%. The generated trajectories and self-annotated Q-values (now combining PPM priors with terminal back-propagation updates) are used to train policy SLM-r3 and PPM-r3.

Policy SLM-r3 reaches 75.8% on MATH (up from 73.6%), and PPM-r3 reaches 85.2% (up from 84.1%). The most dramatic improvements are on competition benchmarks: AIME 2024 goes from 10.0% (r2) to 16.7% (r3) for the policy model, and from 26.7% to 33.3% for the PPM-guided System 2 performance.

Round 4: Solving the hardest problems. After Round 3, while grade school and MATH-level problems achieve high success rates (98.15% and 88.69% training set coverage respectively), only 62.16% of Olympiad-level problems are included in the training set. The paper identifies that this is "NOT solely due to weak reasoning abilities in our SLMs, as many Olympiad problems remain unsolved by GPT-4 or o1." To push coverage further, Round 4 employs an aggressive strategy:

For problems that remain unsolved after the standard 16 MCTS rollouts, the paper performs additional rollouts: 64 total, and if still unsolved, up to 128. Additionally, the paper conducts multiple MCTS tree expansions with different random seeds, effectively running independent search processes that explore different regions of the solution space. The number of candidate nodes per step is increased from 8 to 16, providing more options at each expansion step.

This targeted additional computation boosts Olympiad-level coverage to 80.58%. The improved trajectories are used to train policy SLM-r4 (78.4% on MATH, 26.7% on AIME 2024) and PPM-r4 (87.0% on MATH, 43.3% on AIME 2024, both measured with policy SLM-r1 fixed).

Self-evolution termination. After four rounds, 90.25% of the 747k problems are successfully covered in the training set. The paper manually reviews a random sample of 20 remaining unsolved problems and finds that 19 are synthetic problems with incorrectly labeled answers. Based on this finding, they conclude that "the remaining unsolved problems are of low quality and thus terminate the self-evolution at round 4." This is an important practical detail: the self-evolution loop naturally reaches a plateau not because the models stop improving, but because the remaining problems in the training set are themselves flawed and cannot provide useful training signal.

Training data filtering and selection details. Throughout all four rounds, several consistent filtering and selection rules are applied:

• Problem-level filtering: Synthetic math problems where all generated trajectories achieve less than 50% accuracy (i.e., fewer than half of MCTS rollouts produce correct answers) are removed from the training set. The paper notes this is to "reduce noise in synthetic math problems (e.g., incorrect ground-truth answers labeled by GPT-4)."
• Trajectory selection for SFT: For each problem with at least one correct trajectory, the top-2 trajectories with the highest average Q-values (averaged across all steps in the trajectory) among correct solutions are selected. This is a more stringent criterion than standard rejection sampling (which accepts any correct solution) or best-of-N (which takes the single highest-scoring solution). The paper argues this is "the key for fine-tuning a frontier math LLM" — "a more effective approach ensures that every step in the trajectory is of high quality."
• PPM training data selection: Only problems with mixed outcomes (both correct and incorrect trajectories) are used for PPM training. Problems where all trajectories are correct provide no negative examples; problems where all are incorrect provide no positive examples. For qualifying problems, two positive and two negative examples are selected per step based on Q-values.
• Policy model re-initialization: In each round, the policy SLM is fine-tuned from the initial base model (Qwen2.5-Math-7B), not from the previous round's policy model. The paper states this is based on "extensive experiments" but does not provide a detailed ablation. This is a notable design choice — it prevents error accumulation across rounds (if each round's model had some systematic errors, training on it could amplify those errors) but means each round starts from scratch and must re-learn the base model's knowledge plus the new training data.

Computational scale of self-evolution. The paper provides detailed resource usage for the self-evolution process (Appendix A.1):

• Round 1 (bootstrap): 10 nodes of 8×80GB H100 GPUs, using DeepSeek-Coder-V2-Instruct (236B) with 8 MCTS rollouts. This required approximately two weeks to complete data generation for the 747k problems.
• Rounds 2–4: 15 nodes of 4×40GB A100 GPUs, using the fine-tuned 7B policy SLM. Each round completed in approximately three days.
• Round 4 additional computation: increased to 64 rollouts for unsolved Olympiad problems, extending data generation to one week.

The total training data generated across all rounds consists of "millions of synthesized solutions for 747k math problems." The inference costs per problem at test time vary significantly by benchmark (Table 9): MATH problems average 5,453 generated tokens per trajectory, while AIME 2024 problems average 15,693 tokens — reflecting the increased number of steps and candidate explorations needed for harder problems.

Synthesis of Design Choices and Their Justifications

Why code-augmented CoT instead of pure natural language? The paper argues that natural language CoT generation is inherently vulnerable to hallucination — models can produce convincing-sounding but incorrect reasoning that is difficult to automatically verify. Code execution provides an objective, deterministic filter that catches a large class of errors (undefined variables, type mismatches, import errors, logical inconsistencies in computation) without requiring any human labels or learned verifiers. This is particularly important in the self-evolution setting, where training data is generated by SLMs that hallucinate more frequently than large frontier models. The ablation study (Table 7) supports this: even randomly sampled code-augmented CoT solutions from the self-evolved policy model yield SFT data comparable in quality to carefully curated GPT-4-distilled datasets.

Why PPM instead of PRM? The paper argues that MCTS-derived Q-values, even with 16 rollouts, are too noisy to serve as direct regression targets. Forcing a model to predict exact Q-values introduces training signal that is at best uninformative about fine-grained step quality and at worst misleading. The PPM avoids this by learning only the ordering between steps (which is better?), which is much more robust to Q-value noise — systematic biases in Q-value estimation tend to affect both candidates in a preference pair similarly. The Bradley-Terry ranking loss is the same objective used in RLHF for training reward models from human preferences, adapted here for automated preference label generation. Empirical comparison (Table 8) confirms that PPM outperforms PQM (Q-value-based PRM) on all benchmarks, with the gap largest on the most challenging problems.

Why four rounds of self-evolution? The paper does not claim that four is optimal — rather, the process naturally terminated at four rounds because 90.25% of training problems were covered, and the remaining uncovered problems were found to be low-quality (incorrectly labeled). Each round targets a specific improvement: Round 1 bootstraps from a larger model to escape the cold-start problem; Round 2 uses the improved policy model with more rollouts to train the first reliable PPM; Round 3 integrates the PPM into MCTS for higher-quality data generation; Round 4 applies additional computation to solve the hardest remaining problems. The round-by-round improvements in policy model accuracy (Table 3) and training set coverage (Table 2) show consistent gains, with the largest jumps occurring at Round 2 (when PPM is first effectively used) and Round 4 (when additional rollouts tackle Olympiad problems).

Why 16 rollouts? The paper does not provide an ablation on this number, but the motivation is clear from the design: too few rollouts produce noisy Q-values (Round 1 with 8 rollouts produced an ineffective PPM); too many rollouts are computationally prohibitive at scale (747k problems). Sixteen rollouts provides enough visits per node for Q-values to be meaningfully informative while keeping data generation tractable (3 days per round on 15×4×A100 GPUs). In Round 4, additional rollouts (up to 128) are used selectively on the hardest problems, reflecting a practical tradeoff between uniform and targeted computation.

Why re-initialize from the base model each round? The paper states this is based on "extensive experiments" but provides limited detail. The likely rationale is that cumulative fine-tuning across rounds could lead to catastrophic forgetting or error amplification — if each round's model has systematic biases or errors, training on its outputs could reinforce those errors. Starting fresh from the base model each round ensures that each policy model learns from the highest-quality data available at that stage, without inheriting the parameter-level artifacts of previous fine-tuning rounds. The tradeoff is that each round must re-learn the base model's knowledge, which is amortized by the improving quality of the training data.

Why MCTS with UCT rather than simpler search methods? The paper argues that MCTS naturally decomposes complex multi-step problems into simpler single-step generation tasks, which is better suited to SLMs than generating complete solutions in one shot (as in Best-of-N or self-consistency). Additionally, the tree structure naturally produces step-level Q-values through back-propagation, enabling both trajectory selection for SFT and preference pair construction for PPM training. Alternative search methods like beam search or Best-of-N do not provide this granular per-step annotation. The UCT formula is used without modification (it is "the well-known Upper Confidence bounds for Trees") because it is the standard and provably convergent MCTS node selection algorithm — the paper's contribution is not in the search algorithm but in the overall self-evolution framework.

4. Key Insights and Innovations

Innovation 1: Self-Evolved Deep Thinking as an Escape from the Distillation Ceiling

The paper's most fundamental conceptual contribution is the demonstration that small models can generate their own frontier-level training data through systematic search and self-verification, breaking through the capability ceiling imposed by teacher-model distillation. This is not merely an empirical finding — it represents a reframing of what is possible in the relationship between model scale and reasoning capability.

Prior to this work, the dominant paradigm for building math-capable LLMs was straightforward: use a frontier model (GPT-4) to generate chain-of-thought solutions on training problems, filter for correctness via final-answer matching or rejection sampling, and fine-tune smaller models on these traces. This approach has a hard, structural limitation that is not addressable by scaling up data volume — the student model's reasoning is bounded by what the teacher model can produce. The paper's documentation of diminishing returns from distillation (citing OpenMathInstruct-2's mere 3.9% MATH improvement despite an 8× data increase) is not incidental — it is evidence that this paradigm has reached its natural ceiling.

rStar-Math's self-evolution framework, embodied in the four-round process documented in Table 2 and Table 3, constitutes a fundamental shift from the distillation paradigm to what might be called an emergence paradigm: reasoning capability is not transferred from a superior model but grown through iterative cycles of search, verification, and self-training. The mechanism is straightforward — MCTS generates training data, the data trains better models, the better models generate better data — but the conceptual breakthrough is recognizing that this cycle can, starting from a bootstrap model that is itself no stronger than the target SLM eventually becomes, produce capability that exceeds any individual model in the loop.

The evidence for this is stark. In Round 1, the bootstrap model (DeepSeek-Coder-V2-Instruct) solves only 20.99% of Olympiad-level problems in the training set. By Round 4, the self-evolved 7B policy model solves 80.58% — problems that the bootstrap model could not handle are now generating training data. The policy SLM's pass@1 on MATH rises from 58.8% (base Qwen2.5-Math-7B) to 78.4% (SLM-r4), and when augmented with PPM-guided MCTS, reaches 90.0% — exceeding the bootstrap model's own MCTS performance (75.2% in Round 1, Table 6) by a substantial margin. This means the self-evolved system has surpassed the capability of the model that bootstrapped it, something that distillation fundamentally cannot achieve.

This finding has theoretical significance beyond math reasoning: it suggests that reasoning capability is not solely determined by model scale or training data quality in a static sense, but can emerge from the interaction between search and learning. The "intelligence" resides not in any single model's weights but in the dynamic between a generator, a verifier, and a search procedure — and this dynamic can self-improve.

Innovation 2: Process Preference Modeling — Replacing Regression with Ranking for Step-Level Reward Learning

The Process Preference Model (PPM) is the paper's most significant methodological innovation, and it represents a diagnostic move rather than an incremental architectural tweak. The paper identifies that the fundamental bottleneck in training process reward models is not the model architecture or the scale of training data, but the nature of the training signal itself: MCTS-derived Q-values are inherently too noisy to serve as direct regression targets, and forcing models to predict exact scalar values introduces training signal that is at best uninformative and at worst actively misleading.

This diagnosis challenges a widely-held assumption in the automated PRM literature. Works like Math-Shepherd, AlphaMath, and ReST-MCTS* all operate on the premise that if you can estimate a Q-value for each step (via Monte Carlo rollouts or MCTS back-propagation), you should train a model to predict that Q-value directly. The standard loss is mean squared error between predicted and annotated Q-values. The implicit assumption is that the Q-values are accurate enough to serve as ground-truth labels — that the noise is small enough to be averaged away during training.

rStar-Math's counterargument is empirically grounded but has theoretical depth: the difficulty of assigning precise scalar values to reasoning steps is not merely a problem of insufficient rollouts, but a fundamental property of the task. Even expert humans struggle to assign fine-grained numerical scores to intermediate reasoning steps with consistency, and the MCTS process — which estimates step quality through the outcome statistics of completions from that step — conflates several distinct sources of variation (quality of the step itself, quality of subsequent steps, stochasticity in the policy model's completions) into a single number. The resulting Q-values can reliably distinguish good steps from bad ones (because good steps lead to consistently correct answers and bad steps do not), but they cannot reliably distinguish "excellent" from "merely good" or "bad" from "terrible."

The PPM's innovation is to match the training objective to the reliability properties of the training signal. By switching from regression (predict the exact Q-value) to ranking (predict which of two steps is better), the PPM only requires the training data to provide reliable ordinal information, which the paper argues is much more robust to the noise inherent in automated Q-value estimation. The Bradley-Terry pairwise ranking loss — the same objective used in RLHF for training reward models from human preferences — is the natural formalization of this insight.

This is not merely an incremental improvement. It is a reframing of what process reward modeling should try to accomplish. The field has been asking: "Can we train a model to predict the exact quality score of each reasoning step?" rStar-Math asks instead: "Can we train a model to distinguish better steps from worse ones?" The second question is both easier to answer affirmatively (because the training signal is more reliable for ordering than for scoring) and sufficient for guiding search (because MCTS with UCT only needs to compare candidate steps, not know their absolute values).

The empirical comparison in Table 8 validates this reframing. The PPM outperforms the PQM (a Q-value regression PRM trained on the same data) across all benchmarks, with the gap largest on the most challenging problems — Olympiad Bench (65.3% vs. 62.9%) and College Math (59.0% vs. 57.6%) — where the paper would predict Q-value noise is most severe due to sparser successful rollouts. The PPM even enables the 7B policy model to approach or match o1-mini (90.0% on MATH for o1-mini vs. 89.4% for rStar-Math with PPM; 65.3% on Olympiad Bench for o1-mini vs. 65.3% for rStar-Math), suggesting that the preference-based reward signal is competitive with whatever proprietary reward modeling OpenAI uses.

Innovation 3: Code Execution as Dense Intermediate Verification for Self-Training

The code-augmented chain-of-thought generation method introduces a verification mechanism that is qualitatively different from both learned verifiers (reward models) and outcome-based verification (checking final answers). Python code execution provides dense, deterministic, and cost-free verification of intermediate reasoning steps — a property that none of the existing verification paradigms possess.

To appreciate why this is a conceptual innovation rather than just a practical trick, consider the verification landscape before rStar-Math. Outcome reward models (ORMs) verify only the final answer — they provide no signal about intermediate step quality, making them unsuitable for guiding step-by-step search or for filtering flawed intermediate reasoning from training data. Process reward models (PRMs) provide step-level verification, but their training requires either expensive human annotation or noisy automated methods, and even well-trained PRMs are imperfect — they can assign high scores to plausible-sounding but incorrect reasoning. Natural language self-verification ("check your work") is unreliable, as the paper notes in citing Huang et al. (2023).

Code execution is different in kind. It does not estimate or predict correctness — it enforces a specific, well-defined constraint: the code must execute without raising an exception. This constraint catches a broad class of reasoning errors: undefined variables (indicating the step references something from a previous step that doesn't exist), type mismatches (indicating a conceptual confusion about what kind of mathematical object is being manipulated), import errors (indicating the model forgot necessary library functions), and various runtime errors that manifest from logical inconsistencies in the computation.

Crucially, this verification is available during training data generation, not just at inference time. When the SLM generates candidate steps during MCTS rollouts, every candidate is immediately filtered through code execution. Steps that fail are discarded before they can become nodes in the MCTS tree, meaning the Q-values that are back-propagated and ultimately used to train the PPM and select SFT trajectories are computed over a pre-filtered set of steps that have already passed a basic correctness check. This makes the Q-values cleaner and the preference pairs more informative than they would be if the PPM had to learn to distinguish executable from non-executable steps (a task that code execution handles perfectly and for free).

The ablation in Table 7 provides indirect but compelling evidence for this innovation's importance. The baseline "Random sample" uses code-augmented CoT solutions from the self-evolved policy model (SLM-r3) without any MCTS verification or Q-value filtering — just randomly sampled trajectories. Fine-tuning Qwen2.5-Math-7B on this data yields 72.4% on MATH, which is comparable to or better than the carefully curated GPT-4-distilled NuminaMath-CoT dataset (69.6%). This means that code execution alone — without any search, without any reward model guidance, without any trajectory selection — elevates the quality of SLM-generated training data to the level of GPT-4-distilled data. When combined with MCTS search and Q-value-based trajectory selection ("Step-by-step verified" in Table 7), the same base model reaches 78.4% on MATH — substantially outperforming all distillation baselines.

This finding has implications beyond math reasoning. It suggests that for any domain where intermediate reasoning steps can be coupled with executable verification (code generation with test cases, formal proofs with proof checkers, structured planning with constraint solvers), the self-evolution paradigm may be applicable. The code execution verifier provides the "ground truth at intermediate resolution" that makes self-training without human supervision viable.

Innovation 4: The Reward Model, Not the Policy Model, Determines System 2 Performance Ceilings

The paper's fourth conceptual contribution is an empirical finding with significant implications for how the field should allocate research effort in System 2 reasoning: once the policy model reaches a threshold of reasonable competence, the reward model — not the policy model — becomes the dominant factor determining final reasoning performance. This is not an obvious result; one might reasonably expect that a stronger policy model (higher pass@1) would consistently produce better System 2 results, since search can only find solutions that the policy model is capable of generating somewhere in its distribution.

The evidence for this claim comes from Figure 5, which plots the System 2 accuracy of different policy models (varying in size from 1.5B to 72B, and varying in training from base models to instruct-tuned to rStar-Math fine-tuned) when paired with different reward models. The pattern is striking: policy models with significantly different pass@1 accuracies converge to similar System 2 performance when paired with the same reward model. For instance, rStar-Math-7B has lower SFT accuracy than Qwen2.5-Math-72B-Instruct (78.4% vs. 82.6% on MATH), yet when both are paired with rStar-Math's 7B PPM, the smaller model with the weaker policy actually achieves higher System 2 accuracy (89.4% vs. the 72B model's System 2 performance with its own 72B ORM, which is 85.8%).

The paper's explicit statement captures this finding:

"the reward model proves to be the dominant factor in System 2 reasoning"

and later:

"despite varying Pass@1 accuracy across our three policy SLM sizes, the final reasoning accuracy converges after applying the PPM."

This finding has practical and theoretical implications. Practically, it suggests that research investment in better reward models yields higher returns than investment in larger policy models for System 2 reasoning, at least once the policy model crosses a minimum capability threshold. A 7B policy model with an excellent 7B reward model can outperform a 72B policy model with a 72B reward model — the reward model quality swamps the policy model scale difference. This is a direct argument for the paper's approach of spending substantial training compute on evolving the PPM across rounds, rather than focusing solely on improving the policy model's standalone accuracy.

Theoretically, this finding reframes System 2 reasoning as primarily a verification problem rather than a generation problem. The policy model's job is to generate a sufficiently diverse set of candidate steps so that the correct solution path exists somewhere in the search tree; the reward model's job is to identify that path among the many incorrect alternatives. If the reward model is good enough, even a mediocre policy model can succeed — because MCTS with a strong reward signal will navigate toward the correct solution even if the policy model generates many low-quality candidates along the way. Conversely, if the reward model is weak, even an excellent policy model will struggle — because search will be misled into pursuing plausible-sounding but incorrect branches.

This insight also explains why rStar-Math's self-evolution focuses so heavily on PPM improvement across rounds (Table 4 documents PPM quality rising from 75.2% to 87.0% on MATH) rather than solely on policy model improvement. The policy model does improve (58.8% → 78.4% on MATH), but the System 2 gains (from 75.2% in Round 1 to 89.4% in Round 4, Table 6) are driven substantially by the PPM's improving ability to distinguish good steps from bad ones in the MCTS tree.

These four innovations are not independent — they form a coherent stack. The code execution verifier (Innovation 3) provides clean intermediate-step filtering that makes MCTS Q-values more reliable. The PPM's preference-based training (Innovation 2) extracts robust signal from those Q-values despite their remaining noise. The resulting strong reward model, combined with the improving policy model, enables System 2 performance that exceeds the bootstrap teacher (Innovation 1). And the finding that reward model quality dominates policy model quality (Innovation 4) explains why the self-evolution recipe works: each round's primary value is not just in generating more training data for the policy model, but in producing a better PPM that more effectively guides the next round's MCTS toward higher-quality trajectories.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on seven diverse mathematical benchmarks. The primary challenge benchmarks are MATH-500 (the test split from Lightman et al., 2023, consisting of 500 high-school competition problems), AIME 2024 (30 problems from the American Invitational Mathematics Examination I and II, designed for top high school math students), AMC 2023 (from the American Mathematics Competition validation set), and Olympiad Bench (He et al., 2024, Olympiad-level problems). Additional benchmarks include College Math (Tang et al., 2024, college-level problems), GSM8K (Cobbe et al., 2021, grade-school math), and GaoKao En 2023 (Liao et al., 2024, Chinese College Entrance Exam English version, serving as an out-of-domain test). Full details appear in Section 4.1.

• Base model(s). rStar-Math is evaluated on four base models of varying sizes and specializations: Qwen2.5-Math-1.5B (a math-specialized 1.5B model), Phi3-mini-Instruct (a general-purpose 3.8B model without math specialization, from Microsoft/Abdin et al., 2024), Qwen2-Math-7B (math-specialized 7B), and Qwen2.5-Math-7B (the latest math-specialized 7B). The 7B models were chosen to demonstrate that SLMs can rival frontier models; the 1.5B and 3.8B models test generalization to smaller scales and non-math-specialized architectures. Only Qwen2.5-Math-7B underwent the full four-round self-evolution; the other three models were fine-tuned using trajectories generated from Qwen2.5-Math-7B's fourth round and use the final PPM from that round.

• Metrics. The primary metric throughout is Pass@1 accuracy — the fraction of test problems for which the single selected final answer matches the ground truth. For System 2 methods (including rStar-Math), Pass@1 refers to the accuracy after running MCTS and selecting the best trajectory via the PPM. For System 1 baselines, it is the greedy decoding accuracy. For AIME 2024 specifically, results are also reported as raw number of problems solved out of 15 (for AIME I and II respectively). The paper notes that accuracy numbers for some frontier models on some benchmarks are taken from public technical reports rather than re-evaluated.

• Baselines. The paper compares against three categories. Frontier LLMs: GPT-4o, Claude 3.5 Sonnet, OpenAI o1-preview, and o1-mini — representing the state-of-the-art in both System 1 and System 2 reasoning. Open-sourced reasoning LLMs (System 1): DeepSeek-Coder-V2-Instruct (236B), Mathstral-7B-v0.1, NuminaMath-72B-CoT (Jia LI and Polu, 2024a), LLaMA3.1-8B-Instruct and LLaMA3.1-70B-Instruct (Dubey et al., 2024), and Qwen2.5-Math-72B-Instruct. System 2 baselines: Qwen2.5-Math-72B-Instruct paired with a 72B outcome reward model (Qwen2.5-Math-RM-72B) for Best-of-N selection, and similarly Qwen2.5-Math-1.5B-Instruct and Qwen2-Math-7B-Instruct each paired with the same 72B ORM. Notably, these baselines use a reward model ~10× larger than rStar-Math's 7B PPM.

• Generation budget / compute accounting. The primary unit of test-time computation is the number of MCTS trajectories (complete solution paths from root to terminal node). For the main results (Table 5), rStar-Math generates 16 trajectories for AIME/AMC and 8 trajectories for other benchmarks, using PPM to select the best one — matching the number of samples used in the Qwen Best-of-N baselines. A separate rStar-Math64 configuration uses 64 trajectories to study scaling behavior. Figure 3 systematically varies the number of trajectories to show how accuracy scales with compute. The paper does not equate trajectories across methods in terms of FLOPs or wall-clock time — a trajectory in rStar-Math's MCTS involves multiple single-step generations and PPM evaluations, while a trajectory in Best-of-N is one complete solution generation — so the comparison slightly favors rStar-Math in terms of total computation per "trajectory."

• Cross-validation / statistical protocol. The paper does not describe explicit cross-validation or statistical significance testing for the main evaluation results. The difficulty-based analysis from the prior sections (cross-validation within difficulty bins) is not applied to the main benchmark evaluations. For PPM quality measurement across rounds (Table 4), the policy model is fixed to policy SLM-r1 for fair comparison. For the SFT data ablation (Table 7), all fine-tuning uses the same hyperparameters and training recipe (2 epochs, batch size 128, learning rate 7e-6 for Qwen models). Error bars or confidence intervals are not reported for any results.

Main Quantitative Results

Overall Benchmark Performance (Table 5)

The headline result is that rStar-Math with Qwen2.5-Math-7B achieves 89.4% on MATH-500 (90.0% with 64 trajectories), which surpasses o1-preview at 85.5% (+4.5 percentage points with 64 trajectories) and matches o1-mini at 90.0%. On AIME 2024, rStar-Math reaches 50.0% (53.3% with 64 trajectories), exceeding o1-preview's 44.6% (+8.7 points) but trailing o1-mini's 56.7%. On Olympiad Bench, rStar-Math scores 65.3% (65.6% with 64 trajectories), compared to o1-mini's 65.3% — an exact match. On College Math, rStar-Math achieves 59.0% (60.5% with 64 trajectories), exceeding o1-mini's 57.8% (+2.7 points with 64 trajectories).

The results demonstrate that rStar-Math generalizes across model scales and types. With the 1.5B Qwen2.5-Math model, rStar-Math reaches 87.8% on MATH (88.6% with 64 trajectories) and 46.7% on AIME 2024 — substantially outperforming the base model (51.2% MATH, 0.0% AIME) and the instruct-tuned version (60.0% MATH, 10.0% AIME). With the general-purpose Phi3-mini-3.8B (not math-specialized), rStar-Math achieves 85.4% on MATH (86.4% with 64 trajectories) and 40.0% on AIME (43.3% with 64 trajectories) — from a base of 41.4% MATH and 3.33% AIME. The consistency across model types is notable: even a non-math-specialized 3.8B model reaches o1-preview-competitive performance.

Against open-source System 1 baselines, the gap is dramatic. Qwen2.5-Math-72B-Instruct (a model ~10× larger than rStar-Math's policy model) achieves 85.6% on MATH — rStar-Math's 7B version (89.4%) exceeds this by 3.8 points. On AIME, the 72B instruct model reaches 30.0%, while rStar-Math-7B reaches 50.0%. On Olympiad Bench, 49.0% for the 72B instruct vs. 65.3% for rStar-Math-7B.

Against System 2 baselines with larger reward models, the advantage is consistent. Qwen2.5-Math-7B-Instruct paired with a 72B ORM (Best-of-N) achieves 88.4% on MATH, 26.7% on AIME, 49.9% on Olympiad Bench, and 49.6% on College Math. rStar-Math with the same base model (non-instruct) and a 7B PPM achieves 89.4%, 50.0%, 65.3%, and 59.0% respectively — better on all benchmarks despite using a reward model 1/10 the size and a policy model without instruct tuning. Even Qwen2.5-Math-72B-Instruct + 72B ORM reaches only 36.7% on AIME, compared to rStar-Math-7B's 50.0%.

On out-of-domain evaluation (GaoKao En 2023), rStar-Math-7B achieves 80.5% (81.3% with 64 trajectories), compared to its base model's 51.7% and Qwen2.5-Math-72B-Instruct's 71.9%. This suggests the self-evolution process does not overfit to the training distribution — the improvements transfer to an unseen math examination format.

Test-Time Compute Scaling (Figure 3)

Figure 3 plots Pass@1 accuracy as a function of the number of sampled trajectories (1, 2, 4, 8, 16, 32, 64) for rStar-Math compared to Qwen Best-of-N baselines, across four benchmarks (MATH, AIME 2024, Olympiad Bench, College Math). Three key patterns emerge:

rStar-Math dominates at low budgets. With only 4 trajectories, rStar-Math on MATH reaches approximately 87% accuracy — exceeding the Best-of-N baseline with 64 trajectories (approximately 86%) and approaching o1-mini performance. On AIME, 4 trajectories achieve roughly 40%, compared to Best-of-N's roughly 18% at the same budget. The gap is widest at low trajectory counts, indicating that PPM-guided MCTS is substantially more sample-efficient than Best-of-N with outcome reward models.

Scaling behavior varies by benchmark. On MATH, rStar-Math shows rapid initial improvement (roughly 78% at 1 trajectory → 87% at 4 → 88.5% at 8 → 89.4% at 16) with saturation by 16–32 trajectories, reaching 90.0% at 64. On AIME, improvement is more gradual and continues through 64 trajectories (roughly 30% at 1 → 38% at 4 → 42% at 8 → 50% at 16 → 53.3% at 64), suggesting headroom for further scaling. On Olympiad Bench, the curve rises from roughly 53% (1 trajectory) to 63% (8) to 65.6% (64). On College Math, the curve rises steadily from roughly 47% (1) to 59% (16) to 60.5% (64) — unlike MATH, College Math shows continuing improvement at 64 trajectories with no clear plateau.

The gap between rStar-Math and Best-of-N widens on harder benchmarks. On MATH, rStar-Math at 1 trajectory (roughly 78%) already exceeds Best-of-N at 64 trajectories (roughly 86%? — actually the paper's Figure 3 shows Best-of-N with the Qwen2.5-Math-7B-Instruct model reaching approximately 88.4% at an unspecified number of samples, but the scaling curves show Best-of-N falling substantially below rStar-Math at all budget levels). On AIME, the gap is enormous — Best-of-N with 64 trajectories achieves roughly 26%, while rStar-Math with 4 trajectories reaches roughly 40%. This is consistent with the paper's earlier finding that reward model quality is the dominant factor: the PPM's step-level guidance provides more value on problems where outcome-level signals are sparse or misleading.

A surprising detail in Figure 3: at 1 trajectory (which corresponds to the policy model's greedy pass@1 in the Best-of-N baseline, but in rStar-Math still involves MCTS with PPM guidance), rStar-Math substantially outperforms the policy model's standalone pass@1 (Table 10). For Qwen2.5-Math-7B, the policy model's greedy pass@1 on MATH is 78.4% (Table 3), but 1-trajectory rStar-Math in Figure 3 appears to achieve roughly 78% as well — suggesting that even with a single trajectory, the PPM-guided MCTS process (which generates multiple candidate nodes per step and selects the best via UCT) improves over simple greedy decoding only on harder benchmarks. On AIME, the policy model's pass@1 is 26.7% (Table 3), while 1-trajectory rStar-Math achieves roughly 30%.

Self-Evolution Progress (Tables 2, 3, 4, 6)

The four-round self-evolution process is documented across several tables measuring different aspects of progress.

Policy model pass@1 accuracy (Table 3). The base Qwen2.5-Math-7B starts at 58.8% on MATH and 0.0% on AIME 2024. After Round 1 (bootstrap SFT): 69.6% MATH, 3.3% AIME. Round 2: 73.6% MATH, 10.0% AIME. Round 3: 75.8% MATH, 16.7% AIME. Round 4: 78.4% MATH, 26.7% AIME. The largest absolute jumps occur at Round 1 (initial bootstrap, +10.8 points on MATH) and Round 4 on AIME (+10.0 points from Round 3). On Olympiad Bench, the progression is 21.8% → 34.7% → 39.0% → 44.1% → 47.1%. On College Math: 41.6% → 44.5% → 45.7% → 49.6% → 52.5%.

Training set coverage (Table 2). The fraction of the 747k problems for which MCTS generates at least one correct trajectory, broken down by difficulty level: GSM-level problems reach 96.61% in Round 1 and plateau at 98.15% by Round 3. MATH-level coverage rises from 67.36% (Round 1) to 94.53% (Round 4). Olympiad-level coverage shows the most dramatic improvement: 20.99% (Round 1) → 56.04% (Round 2) → 62.16% (Round 3) → 80.58% (Round 4). The Round 2 jump (+35.05 points in Olympiad coverage) coincides with the introduction of the first reliable PPM and increased rollouts (16 vs. 8). The Round 4 jump (+18.42 points) comes from the targeted additional computation (up to 128 rollouts, multiple tree expansions with different seeds). Overall coverage across all problems rises from 60.17% to 90.25%.

PPM quality (Table 4). Measured with the policy model fixed to SLM-r1 for fair comparison, PPM-guided System 2 accuracy improves consistently: PPM-r1 (Round 1): 75.2% MATH, 10.0% AIME. PPM-r2 (Round 2): 84.1% MATH, 26.7% AIME. PPM-r3 (Round 3): 85.2% MATH, 33.3% AIME. PPM-r4 (Round 4): 87.0% MATH, 43.3% AIME. The largest PPM improvement occurs at Round 2 (+8.9 points on MATH, +16.7 on AIME), corresponding to the switch from 8 rollouts (Round 1, which produced an ineffective PPM) to 16 rollouts with the improved policy SLM-r1. The continued improvements in Rounds 3–4 are more modest but consistent, showing that PPM-augmented MCTS data (Round 3) and harder-problem coverage expansion (Round 4) produce incrementally better reward models.

End-to-end System 2 performance across rounds (Table 6). This table shows the full rStar-Math System 2 accuracy (policy model + PPM) in each round, providing the most direct measure of self-evolution impact. Round 1: 75.2% MATH, 10.0% AIME, 35.7% Olympiad Bench — already competitive with GPT-4o (76.6% MATH, 9.3% AIME, 43.3% Olympiad Bench). Round 2: 86.6% MATH, 43.3% AIME, 59.4% Olympiad Bench — clearly surpassing GPT-4o across the board. Round 3: 87.0% MATH, 46.7% AIME, 61.6% Olympiad Bench. Round 4: 89.4% MATH, 50.0% AIME, 65.3% Olympiad Bench. Starting from Round 2, rStar-Math outperforms GPT-4o on all seven benchmarks evaluated.

System 2 vs. System 1 Gap Across Model Scales (Figure 5, Figure 6, Figure 7, Table 10)

The paper provides extensive characterization of the relationship between policy model pass@1 and System 2 performance.

Policy model pass@1 comparison (Table 10). The fine-tuned policy models produced by rStar-Math do not universally outperform the official instruct versions. On MATH: rStar-Math's Qwen2.5-Math-7B policy (78.4%) is below Qwen2.5-Math-7B-Instruct (82.6%). On AIME: rStar-Math policy (26.7%) substantially exceeds the instruct version (6.0%). On Olympiad Bench: 47.1% vs. 41.6%. The pattern is that rStar-Math's policy models are stronger on competition-level benchmarks but may underperform on MATH relative to extensively instruct-tuned versions.

Pass@N with random sampling (Figure 6). This figure shows what fraction of problems have at least one correct solution among N randomly sampled trajectories. At N=64, rStar-Math's Qwen2.5-Math-7B policy model achieves pass@64 comparable to Qwen2.5-Math-7B-Instruct on MATH (both near 95%) and significantly higher on other benchmarks — consistent with the pattern in Table 10 where rStar-Math's models are optimized for competition problems.

Pass@N with PPM-augmented MCTS (Figure 7). This figure shows what fraction of problems yield at least one correct trajectory in PPM-guided MCTS. The key finding: "after generating 64 trajectories, the pass@N accuracy of different policy models becomes comparable." The four policy models of different sizes (1.5B, 3.8B, and two 7B variants) converge to similar pass@64 rates when guided by the same PPM. This is direct evidence for the claim that the reward model — not the policy model — determines the upper bound of System 2 capability.

Reward model as the dominant factor (Figure 5). This figure directly visualizes the central claim: policy models with different pass@1 accuracies (shown as the first bar in each cluster) converge to similar System 2 accuracy when paired with rStar-Math's PPM (shown as the third bar). For example, Qwen2.5-Math-1.5B (pass@1 of 74.8% on MATH) and Qwen2.5-Math-7B (pass@1 of 78.4%) both reach approximately 89% with PPM-guided MCTS. The Qwen2.5-Math-72B-Instruct policy model (pass@1 of 82.6%) paired with its own 72B ORM achieves only 85.8% — the PPM's quality more than compensates for the policy model's smaller size.

Comparison to Prior SFT Data Generation Methods (Table 7)

This ablation compares fine-tuning Qwen2.5-Math-7B on different training datasets, using the same SFT recipe, to isolate the value of rStar-Math's step-by-step verified trajectories.

The baselines: (1) GPT-4 distillation — open-sourced datasets MetaMath and NuminaMath-CoT, both synthesized using GPT-4. (2) Random sampling from self-generation — the policy SLM-r3 randomly generates code-augmented CoT solutions without MCTS search or verification. (3) Rejection sampling — 32 trajectories randomly sampled from SLM-r3, with high-quality solutions ranked by a trained outcome reward model (ORM), keeping two correct trajectories per problem.

Results (Table 7): MetaMath fine-tuning achieves 55.2% MATH, 3.33% AIME, 19.1% Olympiad Bench. NuminaMath-CoT achieves 69.6% MATH, 10.0% AIME, 37.2% Olympiad Bench. Random sampling from SLM-r3 achieves 72.4% MATH, 10.0% AIME, 41.0% Olympiad Bench — outperforming NuminaMath-CoT despite being generated by a 7B model rather than GPT-4. Rejection sampling with ORM achieves 73.4% MATH, 13.3% AIME, 44.7% Olympiad Bench — better than random sampling but still below the step-by-step verified method. rStar-Math's step-by-step verified trajectories achieve 78.4% MATH, 26.7% AIME, 47.1% Olympiad Bench — substantially outperforming all baselines, with the AIME gap being particularly notable (+13.4 points over rejection sampling).

Two observations are highlighted by the paper: (1) The step-by-step verified trajectories significantly outperform both random sampling (which lacks verification) and rejection sampling (which has only sparse, outcome-level verification via ORM), demonstrating the value of dense, step-level verification through code execution and MCTS Q-value filtering. (2) Even random sampling from the self-evolved policy model yields data comparable to or better than GPT-4-distilled datasets, showing that the policy SLMs, after self-evolution, can generate high-quality math solutions without relying on advanced teacher models.

Process Reward Model Comparison: PPM vs. ORM vs. PQM (Table 8)

This ablation fixes the policy model (policy SLM-r4 from Round 4) and compares three reward model types for System 2 reasoning.

ORM (Outcome Reward Model): Trained on Round 4 trajectories using pairwise ranking loss (same as PPM but only on complete trajectories). Used for Best-of-N selection (since ORMs provide only final-answer scores). Achieves 82.6% MATH, 26.7% AIME, 55.1% Olympiad Bench, 55.5% College Math.

PQM (Process Q-value Model): Trained on Round 4 trajectories using Q-values as direct targets with MSE loss (following the standard PRM training approach of Chen et al., 2024, Zhang et al., 2024a). Used within MCTS. Achieves 88.2% MATH, 46.7% AIME, 62.9% Olympiad Bench, 57.6% College Math.

PPM (Process Preference Model): rStar-Math's method, trained on Round 4 trajectories using pairwise ranking loss. Used within MCTS. Achieves 89.4% MATH, 50.0% AIME, 65.3% Olympiad Bench, 59.0% College Math.

The key findings: (1) Both process-level reward models (PQM and PPM) substantially outperform the outcome-level ORM, confirming the value of dense step-level signals for complex math reasoning — the gap on AIME is especially stark (46.7% and 50.0% vs. 26.7%). (2) PPM outperforms PQM across all benchmarks, with the gap being largest on the most challenging problems: AIME (+3.3 points), Olympiad Bench (+2.4 points), College Math (+1.4 points). This supports the paper's central claim that preference-based training is more robust to Q-value noise than direct regression, particularly for hard problems where Q-value estimates are least reliable. (3) PPM with the 7B policy model achieves accuracy comparable to o1-mini across the board (90.0% vs. 89.4% on MATH, 56.7% vs. 50.0% on AIME, 65.3% vs. 65.3% on Olympiad Bench — actually matching on Olympiad Bench), demonstrating that a well-trained 7B process reward model can rival whatever proprietary reward modeling OpenAI uses.

Ablation Studies and Robustness Checks

Self-evolution round progression (Tables 2, 3, 4, 6): Already detailed in the main results section, but the key finding is monotonic improvement across all four rounds on all benchmarks, with the largest jumps at Round 2 (when the first reliable PPM is introduced). The rounds demonstrate that each component — better policy model, better PPM, PPM-augmented MCTS, additional rollouts — contributes incremental gains.

Code-augmented CoT vs. GPT-4 distillation (Table 7): As detailed above, even randomly sampled code-augmented CoT from the self-evolved 7B model matches or exceeds carefully curated GPT-4-distilled datasets, and step-by-step verified trajectories substantially outperform all distillation baselines. This is the key ablation validating the code-augmented CoT synthesis method.

PPM vs. PQM vs. ORM (Table 8): As detailed above, PPM outperforms both Q-value regression PRM (PQM) and outcome-level ORM, validating the preference-based training approach.

Base model generalization (Table 5): rStar-Math is evaluated on four base models of varying sizes (1.5B, 3.8B, 7B) and types (math-specialized vs. general-purpose). The consistent improvements across all models — including the general-purpose Phi3-mini, which is not math-specialized — demonstrate that the method generalizes across model families and scales. This is not a formal ablation but serves as a robustness check against model-specific effects.

Pass@N convergence under shared PPM (Figure 7): The finding that policy models of different sizes converge to similar pass@N when guided by the same PPM serves as evidence that the PPM, not the policy model architecture or scale, is the binding constraint on System 2 capability. The implicit ablation is: varying policy model quality while holding PPM fixed yields similar final performance; varying PPM quality while holding policy model fixed (Table 4) yields substantial performance differences.

Out-of-domain evaluation (Table 5, GaoKao column): rStar-Math's strong performance on GaoKao En 2023 (81.3% with 64 trajectories for the 7B model, compared to 51.7% base and 71.9% for the 72B instruct model) serves as a robustness check against benchmark overfitting. The training data (NuminaMath competition problems, MetaMath, GPT-4-synthesized MATH/AMC-AIME problems) does not include GaoKao-style problems, yet the improvements transfer.

Pass@1 of fine-tuned policy models (Table 10): The finding that rStar-Math's policy models do not always match the official instruct versions on pass@1 (e.g., 78.4% vs. 82.6% on MATH for Qwen2.5-Math-7B) is itself an informative negative result — it shows that the self-evolution SFT data is optimized for producing models that work well within MCTS search (generating diverse, verifiable candidate steps) rather than for standalone greedy decoding. This is a tradeoff the paper acknowledges implicitly but does not ablate directly.

Missing ablations: Several ablations that would strengthen the paper's claims are absent. There is no ablation on the number of MCTS rollouts for Q-value reliability (8 vs. 16 vs. 32) — Round 1 uses 8 and produces an ineffective PPM, Rounds 2+ use 16 and produce effective PPMs, but it's unclear whether 32 would yield further gains or whether 16 is sufficient. There is no ablation on the exploration constant c (set to 2 throughout) or the number of candidate nodes per step. There is no comparison to alternative search methods within rStar-Math's framework (e.g., beam search vs. MCTS given the same policy model and PPM). There is no ablation on the number of self-evolution rounds — the process terminates because training set coverage saturates (90.25%) and remaining problems appear incorrectly labeled, not because further rounds stop improving. It is possible that additional rounds on a cleaner dataset would continue to yield gains. Finally, there is no head-to-head comparison between rStar-Math's self-evolved 7B models and a 7B model fine-tuned on o1-distilled data (since o1 outputs were not available) — the comparison to o1 is always rStar-Math's System 2 vs. o1's System 1/2, not a comparison of underlying model quality.

Critical Assessment

The experimental evaluation is extensive and generally well-aligned with the paper's central claims, but several important qualifications and gaps should be noted.

Claim: Small models can rival or surpass o1 without distillation. The experiments strongly support this claim for the specific models and benchmarks tested. rStar-Math with Qwen2.5-Math-7B achieves 90.0% on MATH vs. o1-mini's 90.0% and o1-preview's 85.5%; 53.3% on AIME vs. o1-mini's 56.7% and o1-preview's 44.6%. The four-model evaluation (1.5B, 3.8B, two 7B variants) shows the result is not specific to one model or scale.

However, the word "rival" papers over important nuances. On AIME — the most challenging benchmark and the one most directly testing Olympiad-level mathematics — rStar-Math's 53.3% trails o1-mini's 56.7% by 3.4 points, and the paper notes that 8 of the 15 AIME problems are "geometry-based, requiring visual understanding, a capability rStar-Math currently does not support." This means the comparison on AIME is not like-for-like: o1-mini can process visual geometry diagrams while rStar-Math cannot, making the 3.4-point gap potentially an underestimate of the true capability difference on text-only problems. The claim that rStar-Math "surpasses" o1-preview on AIME (+8.7 points) is accurate but less informative than the o1-mini comparison since o1-mini is the stronger math model.

A deeper concern: the paper compares rStar-Math's System 2 performance (MCTS with 8–64 trajectories) against o1's default "thinking time" setting, but o1's thinking budget is not publicly documented. If o1-mini with increased thinking time could achieve substantially higher accuracy, the "rival" claim would need qualification — rStar-Math rivals o1 at its default budget, not o1 at arbitrary scale. The paper does not report how o1's accuracy varies with thinking time, nor does it compare rStar-Math at maximum budget (64 trajectories) against o1 at maximum budget.

Claim: Self-evolution progressively improves both models. The monotonic improvements in Tables 2, 3, 4, and 6 provide clean evidence for this claim. However, the self-evolution loop depends on a bootstrap round using a 236B external model (DeepSeek-Coder-V2-Instruct). The paper does not ablate whether self-evolution could succeed starting from a weaker bootstrap (e.g., using the base Qwen2.5-Math-7B itself with more rollouts in Round 1) or no bootstrap at all. The claim that SLMs can self-improve "without distillation from superior models" is technically true for Rounds 2–4 (which use the self-evolved 7B model), but the initial Round 1 does distill from a superior model. This is a reasonable practical choice, but it means the system as a whole is not fully free of external supervision — it requires one round of large-model bootstrapping to escape the cold-start problem.

Additionally, the improvements in Rounds 3 and 4 are relatively modest compared to Rounds 1–2. Policy SLM pass@1 on MATH: Round 1 (+10.8), Round 2 (+4.0), Round 3 (+2.2), Round 4 (+2.6). PPM quality on MATH: Round 2 (+8.9 from Round 1), Round 3 (+1.1), Round 4 (+1.8). System 2 MATH accuracy: Round 2 (+11.4), Round 3 (+0.4), Round 4 (+2.4). After Round 2, the gains are incremental, and most of the Olympiad-level coverage improvement in Round 4 comes from brute-force additional rollouts (up to 128) rather than from improved models generating better data. This pattern — rapid initial gains followed by saturation — is consistent with the paper's own argument about diminishing returns and the difficulty ceiling being determined by verifier quality, but it raises the question of whether four rounds is genuinely a self-evolution success story or whether two rounds (plus targeted extra computation) would have sufficed.

Claim: The PPM, not the policy model, determines System 2 performance. Figure 5 provides compelling evidence, but several caveats apply. First, this finding is demonstrated for policy models that are all "reasonably strong" — the worst policy model in the comparison (Qwen2.5-Math-1.5B) still achieves 74.8% pass@1 on MATH. It is unclear whether the finding would hold for genuinely weak policy models (e.g., sub-50% pass@1), where the policy model might be the bottleneck because correct solutions exist too rarely in the search tree for even a perfect reward model to find them. The paper does not explore the crossover point where policy model quality begins to dominate.

Second, the finding that different policy models "converge" to similar System 2 performance with the same PPM is based on pass@N (Figure 7), which measures whether at least one correct solution exists among N trajectories. This is not the same as Pass@1 after PPM selection — a PPM could theoretically select the wrong trajectory even when a correct one exists. The convergence in Figure 7 shows that PPM-guided MCTS is equally good at generating correct solutions across policy models, but the actual Pass@1 results (Table 5) do show some variation: rStar-Math-7B achieves 89.4% on MATH while rStar-Math-1.5B achieves 87.8% — a 1.6-point gap that, while small, suggests policy model quality does still matter somewhat.

Third, the comparison that shows the policy model matters less than the reward model (Figure 5) uses different base models with different training histories. The rStar-Math fine-tuned policy models may have different generation characteristics (more diverse, more verifiable, better step-by-step structure) than the instruct versions, making the comparison not purely about pass@1. A cleaner ablation would be: take the same policy model, artificially degrade or improve its pass@1 (e.g., by training on less/more data), and measure System 2 performance with a fixed PPM.

Claim: Code execution verification is essential for data quality. Table 7 provides strong evidence, but the comparison is between code-augmented CoT (with implicit execution filtering) and natural language CoT from GPT-4 distillation. There is no ablation comparing code-augmented CoT with execution filtering against code-augmented CoT without execution filtering. Such an ablation would isolate the marginal value of the execution step from the marginal value of the code-augmented format itself. It is possible that simply generating code-augmented CoT (even without executing it) improves data quality because the structured format enforces more rigorous reasoning — the paper cannot distinguish this from the execution filtering effect.

Weaknesses in experimental design. Several limitations warrant attention:

• Single model family for self-evolution. The full four-round self-evolution is only performed on Qwen2.5-Math-7B. The other three models use trajectories and PPM from this run. It is unknown whether self-evolution from scratch would work comparably on Phi3-mini or Qwen2-Math-7B — the paper argues the method is general but does not demonstrate this for the most computationally expensive part.

• No statistical significance reporting. All results are point estimates without confidence intervals, error bars, or significance tests. With MATH-500 having only 500 test questions, and AIME 2024 having only 30, small absolute differences (e.g., 89.4% vs. 88.2% on MATH between PPM and PQM in Table 8) may not be statistically significant. The paper does not report test-retest variance or provide any measure of uncertainty.

• Compute not equalized across methods. rStar-Math's "trajectory" involves MCTS with multiple single-step generations, code executions, and PPM evaluations per step, while Best-of-N's "trajectory" is one complete solution generation. rStar-Math at 8 trajectories almost certainly uses more total FLOPs than Best-of-N at 8 trajectories. The paper does not provide a FLOPs-matched comparison or wall-clock time comparison between rStar-Math and baselines, making the efficiency claims (e.g., "with only 4 trajectories, rStar-Math significantly outperforms Best-of-N baselines") difficult to interpret in practical terms.

• Oracle terminal scoring during training. During training data generation, terminal nodes are scored using ground-truth answer labels ($+1$ for correct, $-1$ for incorrect). This means the Q-values used to train the PPM and select SFT trajectories rely on oracle information that is not available at test time (where the PPM scores terminal nodes). The paper does not ablate whether training with PPM-scored terminal nodes (matching the test-time setup) would degrade results.

• Small test set for AIME. With only 30 problems in AIME 2024, a single problem represents 3.3 percentage points. The difference between rStar-Math-7B (53.3%, 8/15 problems per exam) and o1-mini (56.7%) is one problem per exam. On a different set of 30 AIME problems, the ordering could easily reverse.

• No comparison to o1 with matched compute. The paper does not know (and cannot control) o1's test-time compute budget. If o1-mini uses significantly less compute than rStar-Math's 64-trajectory MCTS, the comparison is unfair to o1; if it uses more, it is unfair to rStar-Math. The paper acknowledges this implicitly by reporting results at multiple trajectory counts (Table 5, Figure 3), allowing readers to compare at different budgets, but the o1 datapoint remains a single unknown budget.

• Potential benchmark contamination. The training set includes MATH training problems (used for GPT-4 synthesis) and NuminaMath competition problems. While the paper evaluates on MATH-500 (the test set), the AIME 2024 and AMC 2023 problems may overlap with the NuminaMath training data, which includes Olympiad and competition problems. The paper does not explicitly verify that AIME 2024 and AMC 2023 are not in the training set, though these are recent exams (2023 and 2024) that may postdate the NuminaMath dataset collection.

Missing experiments that would strengthen the paper:

1. Scaling beyond 64 trajectories (Figure 3). The paper stops at 64, but AIME and College Math show no saturation at that point. Understanding where these curves plateau would clarify the ultimate capability ceiling of the approach.

2. FLOPs-matched comparison to Best-of-N. How many Best-of-N samples can be generated for the cost of one rStar-Math MCTS trajectory? A FLOPs-equalized comparison would reveal whether rStar-Math's apparent superiority is due to better search or simply more computation.

3. Ablation on the bootstrap model. Can self-evolution succeed if Round 1 uses the base 7B model (Qwen2.5-Math-7B) instead of DeepSeek-Coder-V2-Instruct, perhaps with more rollouts? This would test whether the bootstrap is truly necessary or merely convenient.

4. Ablation on code execution filtering. What is the marginal value of code execution over simply generating code-augmented CoT without execution? Train a policy model on unfiltered code-augmented trajectories and compare.

5. Direct comparison of PPM-generated Q-values to human step-level judgments. The paper claims Q-values are noisy and imprecise but does not validate this against human annotations (e.g., on a subset of PRM800k problems). Such a comparison would strengthen the argument for preference-based training.

6. Self-evolution on a non-math-specialized model from scratch. Performing all four rounds on Phi3-mini rather than Qwen2.5-Math-7B would test whether the method works without a math-specialized initialization.

7. Evaluation on additional math reasoning benchmarks. The paper uses 7 benchmarks, which is substantial, but adding benchmarks like miniF2F (formal theorem proving) or GSM-Plus (harder GSM variants) would test generalization more thoroughly.

8. Ablation on the number of fine-tuning epochs. The policy model is fine-tuned for 2 epochs in all rounds. Would more epochs (on the same data) improve or degrade performance? Is there a risk of overfitting to the self-generated trajectories?

Despite these limitations, the experimental evaluation is thorough by the standards of the field. The consistent improvements across four model scales, seven benchmarks, and four self-evolution rounds, combined with careful ablation of the PPM design and data synthesis method, provide substantial evidence for the paper's main claims — while also leaving clear directions for the follow-up work that would address the remaining questions.

6. Limitations and Trade-offs

6.1 The Cold-Start Problem: Self-Evolution Requires a Superior Bootstrap Model

The assumption or constraint. The four-round self-evolution pipeline depends critically on Round 1 using DeepSeek-Coder-V2-Instruct (236B) — a model more than 30× larger than the target 7B model — to generate the initial training data. The paper acknowledges this explicitly:

"To enable SLMs to self-generate reasonably good training data, we perform a bootstrap round to fine-tune an initial strong policy model"

and further notes:

"With no available reward model in this round, we use terminal-guided annotation for Q-values and limit MCTS to 8 rollouts for efficiency."

Without this bootstrap, the base Qwen2.5-Math-7B starts at 58.8% on MATH and 0% on AIME — likely too weak to generate training data of sufficient quality to initiate the self-improvement cycle.

The consequence. The paper's central narrative — that SLMs can achieve frontier reasoning "without distillation from superior models" — is true only for Rounds 2–4. Round 1 is distillation from a superior model (DeepSeek-Coder-V2-Instruct), and without it, the self-evolution loop has no starting point. The open question is whether self-evolution could bootstrap from the SLM itself (with more rollouts, more problems, or a different initialization strategy), or whether a large external model is strictly necessary. If the bootstrap is necessary, then rStar-Math is better understood as a distillation amplifier — it takes one round of large-model supervision and amplifies it through self-play to exceed the teacher — rather than a fully self-contained self-improvement system. This matters for organizations that lack access to 236B-scale models: the method cannot be replicated from scratch using only small models.

What evidence exists in the paper. Table 2 shows that Round 1 (using DeepSeek-Coder) achieves 60.17% overall problem coverage vs. Round 2 (using the fine-tuned 7B SLM-r1) at 66.60%. The Round 1 policy SLM-r1 achieves 69.6% on MATH (Table 3), a substantial jump from the base 58.8%. No ablation tests whether starting from the base 7B model with more rollouts (e.g., 64 or 128) could achieve comparable Round 1 data quality. The paper does not report what happens if the bootstrap round is skipped entirely or replaced with a weaker teacher.

Mitigation status. Not addressed. The paper does not ablate the bootstrap model choice, test alternative cold-start strategies, or discuss whether the bootstrap requirement could be eliminated. Future work on "training models to directly predict difficulty" or cheaper initialization is suggested in other contexts, but the bootstrap dependency is not flagged as a limitation requiring resolution.

6.2 Difficulty Estimation and Oracle Terminal Scoring During Training Are Not Available at Deployment

The assumption or constraint. During training data generation (all four self-evolution rounds), terminal nodes in MCTS are scored using ground-truth answer labels: $q(s_d) = +1$ for correct final answers, $q(s_d) = -1$ otherwise. The paper states:

"For terminal node $s_d$, we do not use PRM for scoring during training data generation. Instead, we assign a more accurate score based on ground truth labels."

This means the Q-values that guide MCTS node selection, trajectory filtering for SFT, and preference pair construction for PPM training all depend on oracle information — knowing whether a generated solution is correct. At test time, the paper generates only 4 MCTS rollouts per step, using the PPM (not ground truth) for terminal scoring, since ground-truth answers are unavailable.

The consequence. There is a fundamental distribution shift between training and inference in how terminal nodes are evaluated. During training, the MCTS that generates the data used to train both the policy model and the PPM benefits from perfect terminal feedback, which means the Q-values are more accurate and the search is more effective than would be possible with PPM-based terminal scoring. This introduces an unknown gap between the data quality achievable during training and the reasoning quality achievable at test time. The PPM is trained on trajectories whose Q-values were computed with oracle terminal rewards, but at test time it must guide search using only its own (imperfect) predictions — including at terminal nodes. How much does this gap reduce performance? The paper provides no measurement.

Additionally, the terminal-guided annotation used in Rounds 1–2 and the back-propagation update in Eq. 2 ($q(s_i)_k = q(s_i)_{k-1} + q(s_d)_k$) both assume access to the ground-truth terminal reward. This means the PPM's training signal — the preference pairs constructed from Q-values — is itself derived from oracle-informed search. If the PPM had to be trained using data from search guided only by an earlier PPM (with no oracle terminal access), the data quality would likely be lower, and the improvements across rounds might be smaller or require more rounds.

What evidence exists in the paper. None. The paper does not report any ablation comparing training with oracle terminal scoring vs. PPM-based terminal scoring, nor does it measure the gap between MCTS performance with oracle terminals and MCTS performance with PPM-only terminals on the training set. The test-time inference procedure (Appendix A.1) uses PPM scoring for all nodes including terminals, but there is no comparison showing how much accuracy degrades relative to using oracle terminal scoring at test time.

Mitigation status. Not addressed. The paper treats the use of oracle terminal labels during training as an obvious design choice ("a more accurate score") without discussing the train-test discrepancy it creates. This is a significant gap: it means the headline results (90.0% on MATH, 53.3% on AIME) were achieved using a system trained with oracle information that is not available in any real deployment, and the paper provides no estimate of how much performance would drop without it.

6.3 Geometry and Multimodal Problems Are Out of Scope — A Hard Capability Boundary

The assumption or constraint. rStar-Math operates exclusively on text-based math problems and has no mechanism for processing visual information. The paper explicitly notes this limitation in the context of AIME results:

"8 of the unsolved problems were geometry-based, requiring visual understanding, a capability rStar-Math currently does not support."

AIME 2024 consists of 30 problems (15 from AIME I and 15 from AIME II). The paper reports solving 53.3% (8/15) of problems — meaning it solves approximately 8 of the 15 text-based problems (53.3%) and 0 of the geometry problems. This is not an incidental limitation; it represents a categorical exclusion of an entire class of math problems that frequently appear in competition settings.

The consequence. The comparison to o1 on AIME is not like-for-like. o1-preview and o1-mini can process images and therefore have access to geometry diagrams; rStar-Math cannot. The reported gap on AIME (rStar-Math 53.3% vs. o1-mini 56.7%) understates the true capability difference on the text-only subset, since rStar-Math's score includes zeros on all geometry problems while o1-mini's score includes successes on geometry. If rStar-Math solved approximately 8/15 text problems and 0/8 geometry problems (assuming 8 of the 15 problems per exam are geometry-based), its text-only accuracy would be roughly 53.3% overall but would be approximately 8/7 ≈ 100%+ — actually, the math doesn't work out neatly. The paper reports solving 8/15 problems total for AIME I and II (30 problems total, with 16 solved — 8 per exam). The 8 geometry problems are spread across both exams, meaning rStar-Math likely solves roughly 8 out of 14–15 text problems per exam, or approximately 53–57% of text-only problems. The exact breakdown isn't provided.

More broadly, this limitation means rStar-Math is not applicable to any math benchmark or deployment scenario involving figures, diagrams, graphs, or geometric constructions — a substantial fraction of competition mathematics and real-world quantitative reasoning. The paper's code-augmented CoT approach fundamentally assumes text-only input.

What evidence exists in the paper. The paper explicitly acknowledges the geometry limitation in the AIME discussion (Section 4.2) and provides the count of 8 geometry problems. No experiments test whether rStar-Math's approach could be extended to multimodal inputs, and no ablation measures performance specifically on the text-only AIME subset to enable clean comparison with text-only baselines.

Mitigation status. Not addressed as a research direction. The paper mentions that "rStar-Math can achieve further improvements by collecting more challenging math problems" but does not discuss multimodal extensions or geometry. This is a hard boundary on the method's current applicability that constrains the "frontier reasoning" claim — rStar-Math is a frontier text-only math reasoner, and its comparison to multimodal models like o1 is favorable only on the text subset.

6.4 Self-Evolution Saturation: Gains Diminish After Round 2, and the Final Round Relies on Brute-Force Computation

The assumption or constraint. The four-round self-evolution recipe is presented as a key innovation, with each round producing "progressive refinement." However, the quantitative improvements across rounds reveal a pattern of rapid early gains followed by diminishing returns, with the gains in Rounds 3 and 4 being substantially smaller than in Rounds 1–2, and Round 4's improvements coming largely from increased MCTS rollouts rather than better models.

Specifically:

• Policy SLM pass@1 on MATH: Round 1 (+10.8), Round 2 (+4.0), Round 3 (+2.2), Round 4 (+2.6). The Round 3 and 4 gains are 2–3 points each.
• PPM quality on MATH: Round 2 (+8.9 from Round 1), Round 3 (+1.1), Round 4 (+1.8). After Round 2, the PPM improvement is marginal.
• System 2 MATH accuracy: Round 2 (+11.4), Round 3 (+0.4), Round 4 (+2.4).
• Olympiad coverage (Table 2): 20.99% → 56.04% (Round 2, +35.05) → 62.16% (Round 3, +6.12) → 80.58% (Round 4, +18.42). The Round 4 jump comes from additional rollouts (64–128) and multiple tree expansions — brute-force computation, not model improvement.

The paper acknowledges the Round 4 strategy explicitly:

"For unsolved problems after 16 MCTS rollouts, we perform an additional 64 rollouts, and if needed, increase to 128. We also conduct multiple MCTS tree expansions with different random seeds."

The consequence. The self-evolution process shows diminishing returns from model improvement alone. After Round 2 — which produces the first effective PPM and accounts for the majority of the total improvement — further rounds yield incremental gains that may not justify their computational cost. Round 3 requires running PPM-augmented MCTS on all 747k problems (3 days on 15×4×A100 GPUs) for only a 0.4-point System 2 improvement on MATH. The paper does not establish that Rounds 3 and 4 are necessary rather than simply running Round 2 with more computation. If Round 2 trajectories were generated with 64 rollouts instead of 16, would the resulting policy model and PPM match or exceed Round 4 performance without the additional rounds? This is not tested.

The saturation pattern also suggests that rStar-Math may be approaching a fundamental ceiling imposed by the base model's capacity after fine-tuning and the PPM's ability to distinguish correct from incorrect steps. With policy SLM-r4 at 78.4% pass@1 on MATH and PPM-r4 at 87.0% accuracy, and System 2 reaching 90.0%, the remaining headroom for self-evolution on MATH-level problems is small (the base model's potential ceiling may be near 90–92% with this approach). On harder benchmarks (AIME, Olympiad Bench), there may be more headroom, but the Round 3–4 gains are still modest compared to Round 2.

What evidence exists in the paper. Tables 2, 3, 4, and 6 all show the diminishing returns pattern. The paper does not provide an ablation comparing (a) Round 2 with increased rollouts vs. (b) Round 4, which would test whether the additional rounds contribute beyond compute scaling. The per-round computational cost is reported (3 days per round for Rounds 2–3, 1 week for Round 4), enabling a cost-benefit calculation that the paper does not perform.

Mitigation status. Partially addressed. The paper notes that self-evolution terminates at Round 4 because 90.25% of training problems are covered and the remaining unsolved problems appear mislabeled. This is a natural termination condition, but it does not explain why gains from model improvement (as opposed to increased rollouts) diminished so sharply after Round 2. The paper does not analyze why PPM-augmented MCTS (Round 3) produced only a 1.1-point PPM improvement, nor does it discuss strategies for escaping the saturation regime beyond the brute-force Round 4 approach.

6.5 Computational Cost of Training Data Generation Is Not Factored into Headline Comparisons

The assumption or constraint. The headline results (Table 5) compare rStar-Math's test-time inference against baselines using a fixed number of trajectories (8, 16, or 64). However, this comparison excludes the enormous computational cost of generating the training data that makes rStar-Math's policy model and PPM effective. As documented in Appendix A.1:

• Round 1: 10 nodes of 8×80GB H100 GPUs for ~2 weeks
• Rounds 2–3: 15 nodes of 4×40GB A100 GPUs for ~3 days each
• Round 4: same hardware for ~1 week

The total training computation is on the order of hundreds of GPU-days, all invested before the first test-time inference. Baselines like Qwen2.5-Math-7B-Instruct also required substantial training, but their training cost is amortized across all users and tasks (the instruct model is general-purpose), while rStar-Math's training is specific to the MCTS + PPM inference procedure — if the inference method changes, the training would need to be redone.

The consequence. The paper's implicit argument is that investing compute in self-evolution training is preferable to investing compute in a larger base model, because the self-evolved 7B model can rival or surpass much larger models. But the total compute for self-evolution is not quantified in FLOPs and is not compared to the pretraining compute of the larger baseline models (Qwen2.5-Math-72B, DeepSeek-V3, GPT-4o). A practitioner deciding whether to adopt rStar-Math needs to know: is the 4-round self-evolution cheaper or more expensive than simply training a 72B model? If self-evolution costs as much as training the larger model, the headline claim that "small models can rival large models" is misleading — the fair comparison is total compute in vs. capability out, and the "small" model's total compute may not be small at all.

Furthermore, the training data generation process (MCTS on 747k problems with 16–128 rollouts each) generates and discards enormous numbers of token sequences. Table 9 shows that a single AIME trajectory averages 15,693 tokens. With 16 rollouts per problem on 747k problems across 4 rounds, and additional rollouts in Round 4, the total tokens generated during training likely exceed the pretraining tokens of the base model itself. This computational scale is not put in context.

What evidence exists in the paper. Appendix A.1 provides hardware specifications and wall-clock times for training data generation. The paper does not provide total FLOP estimates, does not compare training FLOPs to pretraining FLOPs of baseline models, and does not report total tokens generated during self-evolution. Table 9 provides average inference tokens per problem, which enables a rough estimate but only for test-time inference, not for the 4-round training pipeline.

Mitigation status. Not addressed. The paper frames rStar-Math as demonstrating that SLMs can achieve frontier reasoning, but the cost of getting them there — the self-evolution training pipeline — is treated as infrastructure rather than as part of the comparison. This is a significant omission for any practitioner evaluating total cost of ownership: the headline 90.0% MATH accuracy is achieved by a system whose training cost is unquantified and potentially comparable to training a much larger model that might achieve similar accuracy with simpler inference.

6.6 Limited Evidence for Generalization Beyond the Training Distribution and Model Family

The assumption or constraint. The full four-round self-evolution pipeline is demonstrated on exactly one base model (Qwen2.5-Math-7B), with the resulting trajectories and PPM transferred to three other models (Qwen2.5-Math-1.5B, Phi3-mini-3.8B, Qwen2-Math-7B). The paper does not demonstrate:

1. Self-evolution from scratch on a non-Qwen model (e.g., Phi3-mini, LLaMA)
2. Self-evolution from scratch on a non-math-specialized model
3. The method applied to any domain other than math word problems
4. Performance on math benchmarks substantially different in style from the training distribution

The discussion section (Section 5) gestures at generalization to "code and commonsense reasoning" but provides no experiments, stating that such extensions would require "a mechanism to provide feedback on whether a given trajectory reaches the desired output" — effectively acknowledging that the current method is tightly coupled to math-specific correctness signals.

The consequence. The paper's claim that "rStar-Math offers a general methodology for improving LLM reasoning applicable to various domains" is asserted but not demonstrated. The tight coupling between the method and math-specific properties — ground-truth answer verification, Python code execution for intermediate step checking, Q-value annotation via terminal outcome matching — means that extending rStar-Math to code generation (where correctness is determined by test cases, not a single answer), commonsense reasoning (where correctness is ambiguous or multi-dimensional), or theorem proving (where proofs, not answers, are the output) would require significant methodological adaptation. The paper provides no roadmap for these adaptations and no experiments even on math-adjacent tasks like formal theorem proving (where correctness can be verified by a proof assistant).

Additionally, the train-test distribution overlap is not rigorously analyzed. The training set draws from NuminaMath (competition problems), MetaMath, and GPT-4-synthesized problems based on MATH and AMC-AIME training splits. The evaluation includes MATH-500 (test split, likely non-overlapping), AIME 2024, AMC 2023, and Olympiad Bench. While the paper performs an out-of-domain evaluation on GaoKao En 2023 and shows strong performance (81.3%), this is a single benchmark. The consistent gap between policy model and System 2 performance on MATH (78.4% → 89.4%) compared to the smaller gap on GaoKao (65.7% → 80.5%) suggests that the System 2 gains are larger on in-distribution problems, but this is not analyzed.

What evidence exists in the paper. The main generalization evidence is: (a) rStar-Math works on four models of different sizes and types (Table 5), and (b) it transfers to GaoKao En 2023 (Table 5). No cross-domain experiments are performed. The self-evolution process is only demonstrated on Qwen2.5-Math-7B. The paper does not characterize the similarity between training and test distributions, does not report per-benchmark training set overlap analysis, and does not test on held-out math competitions from different sources or styles.

Mitigation status. Partially addressed through discussion. Section 5 sketches how the method could extend to code reasoning (via test cases) and general reasoning (via human labeling or mutual LLM verification), but these are suggestions for future work rather than validated extensions. The paper acknowledges the geometry limitation (discussed above) but does not acknowledge the broader generalization question as a limitation — it frames the method as general while providing only math-specific evidence. A more conservative framing would present rStar-Math as a math reasoning method with potential for extension, rather than as a general reasoning methodology already demonstrated to work across domains.

7. Implications and Future Directions

How This Work Changes the Landscape

rStar-Math establishes a new regime of possibility for small language models: that frontier mathematical reasoning is not the exclusive province of massive proprietary models or their distillations, but can emerge from self-evolved search and verification processes running on models as small as 1.5B parameters. This is not an incremental improvement over existing distillation approaches — it represents a qualitative shift in what the field should believe is possible without large teacher models. Prior to this work, the dominant assumption was that advanced math reasoning capability must be transferred from a frontier model via distillation, imposing a hard ceiling at the teacher's performance. rStar-Math demonstrates that this assumption is false: a 7B model can self-generate training data that, after iterative refinement, produces reasoning surpassing not only the model that bootstrapped it but also OpenAI's o1-preview on MATH and Olympiad Bench.

The magnitude of this shift is best appreciated by comparing it against the prior state of the art in open-source math reasoning. The best System 2 approaches before rStar-Math — Qwen2.5-Math-72B-Instruct paired with a 72B outcome reward model — achieved 85.8% on MATH, 36.7% on AIME, and 54.5% on Olympiad Bench. rStar-Math with a 7B policy model and a 7B reward model reaches 89.4%, 50.0%, and 65.3% respectively. That a model roughly 10× smaller, using a reward model also 10× smaller, can outperform the best open-source System 2 by 3.6 points on MATH and 13.3 points on AIME reframes the scaling narrative: intelligence per parameter, at least for reasoning within a model's reach, is not fixed but can be multiplied by investing compute in systematic self-improvement rather than in additional pretraining.

The paper also reconciles a tension in the process reward model literature that has been simmering since automated annotation methods were first proposed. Works like Math-Shepherd and AlphaMath attempted to train PRMs by regressing on MCTS-derived Q-values, with limited success on challenging benchmarks. The implicit assumption was that with enough rollouts, Q-values become accurate enough to serve as regression targets. rStar-Math provides evidence that this assumption is fundamentally flawed — the noise in automated Q-value estimation is not merely a matter of insufficient rollouts, but reflects a deeper difficulty in assigning precise scalar scores to reasoning steps — and that switching from regression to ranking is both necessary and sufficient to unlock the value of automated step-level annotation. The PPM's consistent outperformance of the PQM (Table 8), particularly on the hardest benchmarks, provides a clean empirical resolution to the debate: the bottleneck was never the quantity of search or the model architecture, but the mismatch between the training objective and the reliability properties of the training signal.

This finding redirects research attention in a concrete way. The paper implicitly argues that further progress on process reward modeling should focus on improving preference signal quality and ranking objectives, not on scaling up Q-value regression with more rollouts or larger models. This makes the extensive literature on preference learning, pairwise ranking, and RLHF directly relevant to automated math reasoning in a way that was not obvious before. Conversely, research directions focused on improving the precision of automated Q-value estimation — through better search algorithms, more rollouts, or more sophisticated aggregation — become less attractive, because the PPM results suggest that even if perfect Q-values were achievable, the marginal benefit over preference-based training might be small.

Perhaps the most consequential shift is the paper's demonstration — through the finding that reward model quality dominates policy model quality in determining System 2 performance (Figure 5) — that the economics of building reasoning systems favor investment in verification over investment in generation. This is not an obvious result; the natural intuition is that a stronger policy model (higher pass@1) should consistently yield better search results, since search can only find solutions the policy model can generate. The paper shows this intuition is incorrect once the policy model passes a relatively modest capability threshold. A 1.5B policy model with a strong PPM can nearly match a 7B policy model with the same PPM on System 2 accuracy, and both can surpass a 72B policy model with a 72B outcome reward model. This finding, if it generalizes to other domains, would reshape how organizations allocate research and engineering resources: the highest-ROI investment is not in scaling policy models but in building better reward models.

Finally, the emergence of intrinsic self-reflection documented in Figure 4 — where the policy model recognizes its own low-quality steps during MCTS and spontaneously backtracks to try a different approach, without any explicit self-reflection training — suggests that structured search plus step-level verification can elicit metacognitive behaviors that have been largely absent from open-source models. This finding, while anecdotal (the paper provides two examples but no systematic quantification of how often self-reflection occurs or whether it causally improves accuracy), hints at a deeper principle: that "thinking about one's own thinking" may not require explicit training on self-correction data, but can emerge naturally when a model is embedded in a search process that provides granular feedback on step quality. This aligns with and extends the observations from o1 about emergent self-correction, and it suggests that architectural innovations in search and verification may be more promising paths to metacognitive capability than behavioral training approaches like self-reflection prompting or explicit self-correction fine-tuning.

Follow-Up Research This Work Enables

Quantifying and inducing self-reflection in MCTS reasoning. The paper reports (Section 5) two qualitative examples of spontaneous self-reflection during rStar-Math's deep thinking — the model recognizes low-quality steps, abandons its current approach, and tries a new one. But the paper provides no quantification: how often does this occur? On what types of problems? Does it actually cause improved accuracy, or is it merely correlated with problems where the model would find the correct answer through other paths? A strong follow-up would instrument the MCTS tree to detect backtracking events (nodes where the PPM score drops sharply and the search switches to a qualitatively different branch), measure their frequency across difficulty levels, and ablate whether removing self-reflection trajectories (by forcing the model to continue along its original path) degrades accuracy. If self-reflection is causally important, this opens the door to training interventions that explicitly encourage it — for instance, adding synthetic self-reflection examples to SFT data, or modifying the MCTS selection criterion to favor exploration when PPM scores are declining.

Bootstrapping self-evolution without a large teacher model. The paper's four-round self-evolution depends on a Round 1 bootstrap using DeepSeek-Coder-V2-Instruct (236B), a model ~34× larger than the target 7B model. The paper does not test whether this bootstrap is necessary or merely convenient. A critical follow-up experiment would attempt self-evolution starting from the base Qwen2.5-Math-7B alone, varying the number of MCTS rollouts in Round 1 — 16, 64, 128, 256 — to see whether sufficient compute can compensate for the weaker initial policy model. If self-evolution succeeds without the large bootstrap, it would mean the method is fully self-contained and replicable by researchers without access to 236B-scale models. If it fails even at 256 rollouts, it would establish a fundamental threshold: there exists a minimum policy model capability below which self-generated training data is too noisy to initiate improvement. Either outcome would be scientifically informative, and the experiment is directly feasible using the paper's released code and data.

Process preference modeling beyond pairwise Bradley-Terry ranking. The PPM uses the standard Bradley-Terry pairwise ranking loss, which models the probability of preferring one step over another as a sigmoid of the score difference. But this loss treats all preference pairs equally, regardless of how confident the Q-value-based preference is. A natural extension would be to incorporate confidence weighting: preference pairs where the Q-value gap between positive and negative examples is large (e.g., +0.8 vs. -0.7) should carry more training signal than pairs where the gap is small (e.g., +0.2 vs. -0.1). This could be implemented through a margin-aware ranking loss or by weighting each pair in the expectation by the Q-value gap. A strong experiment would compare standard Bradley-Terry against confidence-weighted variants on the same training data, measuring PPM quality via the Table 4 protocol (fixed policy model, varying PPM). Additionally, extending the PPM from pairwise to listwise ranking — predicting the ordering of 4–8 candidates simultaneously, rather than decomposing into pairwise comparisons — could provide richer training signal from the same MCTS trees, since each node expansion typically generates 8–16 candidates that can be partially ordered by Q-values.

Combining PPM-guided MCTS search with iterative revision. The paper notes (Section 5, Generalization discussions) that the method could be extended to other domains, but a more immediate and tractable extension would be within math reasoning itself: use the PPM to guide iterative revision of incorrect solutions. In the current system, MCTS generates a tree of solution attempts from scratch for each problem. An alternative architecture would be: generate an initial solution, use the PPM to identify low-scoring steps, prompt the policy model to revise those specific steps, and re-evaluate with the PPM — essentially, local search in the neighborhood of an existing solution rather than global tree search from the root. This would be particularly valuable for problems where the model's initial approach is roughly correct but contains a specific error (e.g., an algebraic mistake or a misapplied theorem), which the self-reflection examples in Figure 4 suggest is a common failure mode. A concrete experiment: on problems where rStar-Math's first correct trajectory appears only after several MCTS rollouts (indicating the initial attempts were flawed), measure whether targeted revision of the lowest-PPM-score step in the first incorrect trajectory can recover the correct answer with fewer total tokens than full MCTS.

Scaling laws for self-evolution: how many rounds, how many rollouts, how much data? The paper performs exactly four rounds of self-evolution and terminates because 90.25% of training problems are covered. But this termination criterion is accidental — it depends on the quality of the training set, not on any inherent limit of the self-evolution process. A systematic study of self-evolution scaling laws would vary: (a) the number of rounds (2, 4, 6, 8), (b) the number of MCTS rollouts per problem per round (8, 16, 32, 64), (c) the number of problems in the training set (100k, 250k, 500k, 747k), and (d) the difficulty distribution of training problems (all levels vs. competition-only). The goal would be to establish whether there exist predictable scaling relationships — analogous to the Chinchilla scaling laws for pretraining — that describe how downstream reasoning accuracy improves as a function of self-evolution compute. The paper provides exactly one datapoint on this curve (the four-round results in Tables 3, 4, 6), which is insufficient to fit any scaling law. Such a study would be expensive (the 4-round process already requires hundreds of GPU-days) but would provide foundational knowledge for the field, analogous to what Hoffmann et al. (2022) provided for pretraining.

Stress-testing the PPM against adversarial steps and distribution shift. The PPM is trained on MCTS-generated trajectories from rounds 2–4, which means it sees only steps that (a) pass the code execution filter and (b) are generated by policy models from its own self-evolution lineage. At test time, the policy model continues to improve across rounds, meaning the PPM must evaluate steps from a model that is slightly different from the one that generated its training data — a mild distribution shift. A stress-test experiment would deliberately expose the PPM to out-of-distribution steps: (1) steps generated by a different model family (e.g., LLaMA-3.1 or DeepSeek) fine-tuned on the same math data, (2) adversarial steps designed to score highly under the PPM while being mathematically incorrect (e.g., steps that exploit PPM blind spots by using theorem names and formal notation without valid reasoning), and (3) steps from problems in a different domain (e.g., physics or chemistry word problems) where mathematical reasoning is required but the problem framing differs. Degradation under these conditions would reveal whether the PPM has learned robust mathematical reasoning evaluation or merely learned to recognize surface patterns in its training distribution. Conversely, robustness would suggest the preference-based training objective naturally produces more generalizable evaluation than Q-value regression.

Practical Applications and Downstream Use Cases

Cost-efficient mathematical tutoring and assessment systems. An immediate application is in educational technology, where automated math reasoning at frontier levels has been prohibitively expensive due to reliance on large proprietary models. rStar-Math's 7B policy model and 7B PPM can run on 4×40GB A100 GPUs — hardware that is accessible to mid-size organizations and academic institutions — and achieve o1-competitive accuracy on competition math problems. A tutoring system built on rStar-Math could generate step-by-step solutions with code verification for student problem sets, provide targeted feedback by identifying which steps have low PPM scores, and adapt difficulty based on the PPM's confidence. The paper's test-time inference costs (5,453 tokens per MATH trajectory, 15,693 per AIME trajectory, Table 9) are modest enough for interactive use, and the ability to scale test-time compute (Figure 3) means the system can trade latency for accuracy depending on the stakes of the assessment. For high-stakes exam preparation (AIME, AMC, Olympiads), the 64-trajectory configuration achieves 53.3% on AIME and 87.5% on AMC — sufficient to provide useful guidance to students, if not to replace expert human tutors.

Synthetic data generation for training specialized math models. Organizations building math-capable LLMs currently rely on GPT-4 distillation (NuminaMath, MetaMath) or expensive human annotation (PRM800k). rStar-Math provides a third path: use a self-evolved 7B model with MCTS to generate high-quality, step-verified training data at scale. The paper's ablation (Table 7) shows that even randomly sampled trajectories from the self-evolved policy model produce SFT data comparable to NuminaMath-CoT, and step-by-step verified trajectories substantially outperform it. For a team training a new math-specialized model (e.g., based on LLaMA or a custom architecture), the most efficient pipeline would be: (1) bootstrap rStar-Math on their target base model using available large models (or the released Qwen2.5-Math-7B trajectories), (2) run one or two self-evolution rounds to produce a strong policy model and PPM, (3) use PPM-augmented MCTS to generate millions of verified solution trajectories, (4) SFT the target model on these trajectories. This pipeline could produce training data that exceeds GPT-4 distillation quality for the specific model family, since the MCTS data is generated by and verified for that family's output distribution. The paper's released code and data at github.com/microsoft/rStar make this immediately actionable.

Automated problem curation and difficulty assessment for math competitions. The self-evolution process naturally produces per-problem difficulty estimates via the training set coverage statistics (Table 2): problems that are solved in early rounds with few rollouts are easy, problems requiring later rounds or many rollouts are hard, and problems that remain unsolved after Round 4 are very hard (or mislabeled). This difficulty signal is more informative than static human-assigned difficulty labels (e.g., MATH levels 1–5) because it reflects the actual capability frontier of the model. Math competition organizers or curriculum designers could use rStar-Math to automatically bin problems by difficulty relative to a target model, identify problems where human labels disagree with model-empirical difficulty (which might indicate mislabeling or problems that are surprisingly hard/easy for LLMs), and validate new problem proposals by checking whether they are solvable by the current self-evolved model. The paper's finding that 19 of 20 randomly sampled unsolved problems were mislabeled with wrong answers demonstrates the curation value: self-evolution can automatically flag likely mislabeled problems in large datasets, a task that would be prohibitively expensive to do manually at scale.

When to Prefer This Method

The paper does not articulate an explicit decision framework positioning rStar-Math against named alternatives, and the method is not presented as one option in a tradeoff space — it is presented as a self-contained approach that achieves state-of-the-art results. The natural comparison points (distillation from frontier models, Best-of-N with large reward models, direct PRM training with Q-value regression) are evaluated as baselines rather than as alternatives a practitioner would choose between. The paper's contribution is demonstrating that self-evolved deep thinking works at all for SLMs, not establishing the boundary conditions under which it is preferable to other methods. Forcing a decision matrix here would require inferring tradeoffs that the paper does not empirically characterize (e.g., rStar-Math vs. o1-distillation at equivalent total compute). The paper's ablation studies (Tables 7, 8) show rStar-Math's components outperforming specific alternatives (step-by-step verified trajectories > GPT-4 distillation; PPM > PQM > ORM), but these are within-method comparisons, not deployment-level tradeoffs between competing systems.