Let's Verify Step by Step

ArXiv: 2305.20050

🎯 Pitch

For training reward models on multi-step math reasoning, process supervision crushes outcome supervision—not marginally, but by enabling accuracy to keep climbing as you scale test-time compute, while outcome-based models hit a hard ceiling. Active learning on convincing wrong answers makes collecting step-level human feedback 2.6× more efficient, proving you can get dramatically better alignment with far fewer labels.

1. Executive Summary

This paper studies how to train more reliable reward models for mathematical reasoning by comparing two supervision paradigms — process supervision (human feedback on each intermediate step) versus outcome supervision (feedback only on the final answer) — using a GPT-4 base model on the challenging MATH dataset. The core contribution is a state-of-the-art process-supervised reward model (PRM) trained on 800,000 step-level human feedback labels that achieves 78.2% best-of-1860 accuracy, substantially outperforming both outcome-supervised reward models and majority voting, with the performance gap widening as the number of sampled solutions increases. The paper further demonstrates that active learning — strategically surfacing convincing wrong-answer solutions for human labeling — yields a 2.6× improvement in data efficiency for process supervision, establishing that process supervision significantly outperforms outcome supervision when scaled to sufficient data volumes and that a large PRM can serve as a reliable synthetic supervisor for training smaller models.

2. Context and Motivation

The Core Problem: Reward Models Are Only as Good as Their Training Signal

The fundamental problem this paper addresses is deceptively simple: how do you train a reward model that can reliably distinguish correct reasoning from incorrect reasoning? This is not merely an academic exercise. Reward models serve as the evaluative backbone for an entire class of techniques that make language models more capable — reinforcement learning from human feedback (RLHF), rejection sampling, and best-of-N search all depend on a reward model to score candidate outputs. As the paper bluntly states in Section 1:

"While these techniques are useful, the resulting system is only as reliable as the reward model itself. It is therefore important that we study how to most effectively train reliable reward models."

A reward model that cannot reliably spot errors produces a cascade of downstream failures: an RLHF-trained policy will optimize a flawed signal, a best-of-N search will select nonsense that happens to score highly, and a safety-critical system will fail to flag dangerous but superficially plausible outputs. In this sense, the reward model is the linchpin of alignment — get it wrong, and everything built on top is compromised.

The specific setting the paper studies — multi-step mathematical reasoning — sharpens this problem because errors are compositional. A single logical mistake in step 3 of a 10-step proof renders the entire solution incorrect, regardless of how coherent the remaining steps appear. This means the reward model faces an unusually demanding credit assignment problem: it must not only detect that an error exists somewhere in a long chain-of-thought, but ideally identify where the reasoning derailed. The difficulty of this task depends directly on the kind of supervision used to train the reward model.

Outcome Supervision: Simple but Impoverished

The standard approach at the time of this paper's writing was outcome supervision — a reward model is trained to predict whether a complete solution reaches the correct final answer. This is appealing for two reasons. First, it requires only final-answer labels, which for math problems can be generated automatically by comparing the model's answer to a ground-truth solution. No human annotators are needed to label intermediate steps. Second, the training setup is straightforward: treat the final token's prediction as the solution-level score, and train with binary cross-entropy against correctness labels (Section 2.5, Appendix E).

However, outcome supervision has a fundamental weakness that the paper identifies: it provides no information about where a solution went wrong. When an outcome-supervised reward model (ORM) is trained on a dataset of solutions labeled "correct" or "incorrect," it must learn to infer which steps are likely to be erroneous — a challenging credit assignment problem that the model must solve implicitly. As Section 6.1 explains:

"A reward model trained with outcome supervision faces a difficult credit-assignment task — to generalize well, it must determine where an incorrect solution went wrong. This is particularly difficult for hard problems: most model-generated solutions contain an error somewhere, so the marginal value of a negative label from outcome supervision is low."

This has concrete consequences. On hard problems where the generator's pass rate is low, most solutions in the ORM's training set are incorrect. A negative label tells the ORM that somewhere in the solution there is a mistake, but not which step. The ORM must essentially learn to diagnose errors without ever being told where they are — a form of unsupervised error detection within a supervised learning framework.

A second, subtler problem is false positives from automatic grading. The MATH dataset (Hendrycks et al., 2021) provides ground-truth final answers that can be automatically checked, but a solution can reach the correct final answer through incorrect reasoning. These "spurious" solutions receive a positive label under automatic outcome supervision, teaching the ORM that flawed reasoning is acceptable as long as the final number matches. This is not merely a theoretical concern — the paper notes in Section 2.5 that "false positives solutions that reach the correct answer with incorrect reasoning will be misgraded," and Section 4 explicitly investigates how much this damages ORM performance.

This weakness is particularly relevant from an AI alignment perspective. If an RLHF-trained model learns that it can reach correct answers through incorrect reasoning, it may develop reasoning patterns that are superficially successful but uninterpretable or unverifiable to human overseers. The paper cites prior work (Zelikman et al., 2022; Creswell et al., 2022) showing that "models trained with outcome supervision regularly use incorrect reasoning to reach the correct final answer." This is an alignment failure: the model learns to optimize the outcome proxy rather than developing genuinely sound reasoning.

Process Supervision: Richer Signal, Higher Cost

The alternative is process supervision: providing feedback on the correctness of each individual step in a chain-of-thought solution. A process-supervised reward model (PRM) receives a positive, negative, or neutral label for every reasoning step, allowing it to learn what correct intermediate reasoning looks like — not just whether the final answer matches. As Section 2.6 and Figure 2 illustrate, the PRM scores each step and aggregates these into a solution-level score by taking the product of per-step correctness probabilities.

Process supervision has several compelling theoretical advantages:

Precise error localization. The reward model is explicitly told which step contains an error, removing the credit assignment problem that plagues ORMs.
Richer per-sample information. An incorrect solution with 10 steps, only one of which is wrong, provides the PRM with 9 positive step-level labels and 1 negative label — vastly more information than a single "incorrect" label. Section 6.1 frames this as the key explanatory mechanism: "Process supervision makes credit assignment easier, and we believe that this explains its strong performance."
Alignment benefits. Process supervision "directly rewards models for following a human-endorsed chain-of-thought" (Section 1), incentivizing interpretable reasoning rather than optimizing an outcome proxy. The paper explicitly argues in Section 6.2 that this constitutes a negative alignment tax — safer behavior that improves rather than sacrifices performance.
Resistance to false positives. A solution with incorrect reasoning that accidentally reaches the right answer will have at least one step labeled negative, preventing the PRM from learning to reward spurious reasoning patterns.

The cost, of course, is data collection. There is no automated way to provide step-level correctness labels — a human must read each step of each solution and judge its validity. This makes process supervision data expensive, slow, and difficult to scale. The paper's release of PRM800K (800,000 step-level labels across 75,000 solutions) represents a substantial investment in human annotation effort, and a central question the paper must answer is whether this investment pays off relative to the essentially free outcome supervision signal.

The Uesato et al. (2022) Precedent: A Conflicting Result

The paper does not enter this comparison in a vacuum. A closely related prior study by Uesato et al. (2022) compared process and outcome supervision for training reward models on grade school math (the GSM8K dataset). Their finding was notably negative for process supervision:

"Uesato et al. (2022) found that outcome supervision and process supervision led to similar final performance in the domain of grade school math."

This is a striking result because it contradicts the theoretical advantages of process supervision outlined above. If process supervision provides strictly more information per solution (error localization in addition to error detection), why doesn't it outperform outcome supervision?

The paper identifies three key differences between its own work and Uesato et al. (2022) that may explain the discrepancy (Section 7.1):

Base model capability. The present work uses GPT-4, a significantly more capable model than those available at the time of Uesato et al. (2022). A stronger generator produces more subtle errors — mistakes that are harder for an ORM to detect without step-level guidance but also more valuable to label precisely.
Dataset difficulty. MATH is substantially harder than GSM8K. On harder problems, the credit assignment problem is more severe (most solutions contain errors, so outcome labels are less informative), making process supervision's advantage more pronounced.
Quantity of supervision. The present work collects an order of magnitude more process supervision data (PRM800K vs. the scale used in Uesato et al.). If process supervision shows a steeper data scaling curve than outcome supervision, the small-scale results in Uesato et al. might reflect a regime where both methods perform similarly, while the large-scale regime reveals process supervision's asymptotic advantage.

The paper explicitly reconciles these findings in Section 7.1 by appealing to a data scaling argument:

"The data scaling trend in Figure 4a suggests that a small amount of process supervision and a large amount of outcome supervision do in fact lead to similar performance, consistent with the results from Uesato et al. (2022). The trend also shows that process supervision beats outcome supervision when scaled up."

This is a crucial reframing: the conflict with prior work is not a contradiction but a scale effect. At low data volumes, outcome supervision (being essentially unlimited, since it's automatic) can match process supervision (which is expensive and limited). But as process supervision data accumulates, its richer per-sample signal yields a steeper improvement curve that eventually overtakes outcome supervision.

Where Existing Approaches Fall Short

Beyond the specific Uesato et al. (2022) comparison, the paper identifies several broader gaps in the existing literature:

No systematic large-scale comparison on challenging reasoning. Prior work had not conducted a head-to-head comparison of ORMs and PRMs at the scale and difficulty level that might reveal process supervision's advantages. GSM8K, the domain studied by Uesato et al., involves arithmetic word problems that most capable models solve with high accuracy — a regime where the distinction between detecting errors and localizing them may be less consequential.

Active learning for process supervision is underexplored. Given the high cost of human step-level labels, it is critical to maximize the value extracted from each annotation. The paper notes that prior work had not systematically investigated how to select which solutions to surface for human labeling. Uniform random sampling — showing labelers whatever the generator produces — is wasteful because many solutions will be obviously correct or obviously wrong, providing little new information. The paper's active learning strategy (Section 2.4, Section 4.2) addresses this gap by targeting "convincing wrong-answer solutions" — solutions that score highly under the current PRM but reach an incorrect final answer, and are therefore most likely to reveal the PRM's blind spots.

No demonstration that large PRMs can serve as synthetic supervisors. Collecting human process supervision at scale is expensive and requires ongoing annotator management (quality control, instruction refinement, retraining). If a large, carefully trained PRM could reliably replace human labelers for training smaller models, the cost barrier to process supervision would drop dramatically. The paper investigates this possibility in Section 4, using the large-scale PRM (PRMlarge) to provide synthetic process and outcome supervision for smaller reward models. Prior to this work, it was unclear whether synthetic process supervision could match the quality of human labels.

Alignment implications of supervision method are underappreciated. The paper argues in Section 6.2 that the choice of supervision method has direct consequences for AI alignment that extend beyond raw performance. Models trained with outcome supervision learn to exploit the outcome proxy — reaching correct answers through flawed reasoning — which is a microcosm of the broader specification gaming problem in AI safety. Process supervision, by directly rewarding faithful intermediate reasoning, produces models whose internal decision-making is more aligned with human expectations. The paper frames this as an underappreciated dimension of the ORM vs. PRM comparison: even if both performed identically on final-answer accuracy, process supervision would be preferable on alignment grounds.

How This Paper Positions Itself

The paper positions itself not as proposing a fundamentally new method — both process and outcome supervision were established by prior work — but as providing the definitive large-scale comparison that the field lacked, with several carefully designed innovations that strengthen the comparison:

Scale as a deliberate experimental choice. The paper makes a conscious decision to operate at a scale where the differences between supervision methods become apparent. This includes using GPT-4 as the base model, training on the challenging MATH dataset (not the easier GSM8K), and collecting 800,000 human step-level labels. The implicit argument is that prior null results (Uesato et al., 2022) were artifacts of insufficient scale, and that the true advantages of process supervision only emerge in the large-data regime.

Careful isolation of supervision method from other variables. A recurring theme is the paper's attention to confounding factors. The ORM and PRM training sets are not directly comparable at large scale (the ORM is trained on 100 uniform samples per problem — an order of magnitude more data — while the PRM uses actively-selected, biased data). The paper is explicit about this limitation (Section 3) and addresses it through the small-scale synthetic supervision experiments (Section 4), where training sets are held identical and only the supervision type varies. This two-regime design (large-scale for SOTA results, small-scale for controlled comparisons) is a methodological strength.

Active learning as a force multiplier for human data. The paper does not simply compare ORMs and PRMs with a fixed labeling budget — it asks whether the labeling budget can be spent more intelligently. The active learning results (Section 4.2) show a 2.6× improvement in data efficiency, meaning that process supervision can be made substantially cheaper without sacrificing quality. This directly addresses the primary objection to process supervision (cost) and makes the overall case for PRMs more practically compelling.

Process supervision as an alignment intervention with negative tax. Rather than treating alignment and capability as competing objectives (where safer methods typically sacrifice performance), the paper frames process supervision as a rare case where alignment and capability are aligned. Section 6.2 explicitly argues that process supervision incurs a "negative alignment tax" — it improves both safety properties (interpretable reasoning, human-endorsed chain-of-thought) and raw performance. This reframes the adoption question: there is no tradeoff to agonize over; process supervision is simply the better method on both axes.

Open release to catalyze further research. By releasing PRM800K — the complete dataset of human step-level labels — the paper lowers the barrier to entry for other researchers. This is significant because the high cost of human process supervision data had previously limited exploration of PRMs to well-resourced labs. The release explicitly aims to "catalyze related research on the alignment of large language models" (Section 8), positioning the paper as an enabler of future work rather than merely a one-off result.

In summary, the paper addresses a gap that is simultaneously technical (how to train the most reliable reward model), economic (how to maximize the value of expensive human feedback), and safety-relevant (how to incentivize interpretable reasoning). It positions process supervision as the superior paradigm — not by inventing it, but by demonstrating its advantages at a scale and rigor that prior work had not achieved, and by providing concrete mechanisms (active learning, synthetic supervision) to make it practical.

3. Technical Approach

3.1 Reader Orientation

This paper builds and evaluates reward models — neural networks that assign quality scores to solutions generated by a language model — using two fundamentally different forms of human supervision: feedback on final answers only (outcome supervision) versus feedback on every intermediate reasoning step (process supervision). The core problem is that reward models serve as the evaluative backbone for techniques like reinforcement learning and best-of-N search, but they are only as reliable as their training signal; the paper's solution is to demonstrate that process supervision produces substantially more reliable reward models than outcome supervision when scaled to sufficient data volumes, and that active learning can dramatically reduce the human labeling cost required to achieve this reliability.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a pipeline:

Generator (GPT-4 fine-tuned for step-by-step output): A language model that produces multi-step mathematical solutions in a structured, parseable format. It is the "proposal distribution" from which candidate solutions are drawn.
Human Data-Labelers (the supervision source): Trained annotators who examine individual reasoning steps and assign labels of positive, negative, or neutral, producing the PRM800K dataset of 800,000 step-level labels across 75,000 solutions.
Active Learning Selection Strategy: A meta-process that chooses which solutions to surface to human labelers — specifically targeting "convincing wrong-answer solutions" that score highly under the current best PRM but reach incorrect final answers — to maximize the information gained per human annotation.
Reward Model (ORM or PRM, fine-tuned from GPT-4 or smaller base models): A classifier trained to predict solution quality. ORMs predict correctness from the final token of a complete solution; PRMs predict correctness at every intermediate step. The PRM aggregates per-step scores via their product to produce a single solution-level score.
Best-of-N Evaluation Harness: At test time, the generator produces N candidate solutions per problem; the reward model scores each; the top-ranked solution is automatically graded against the ground-truth answer; the fraction correct measures reward model quality.

Information flows as follows: a MATH problem enters the generator → the generator produces N candidate solutions → the active learning strategy selects which solutions to send to human labelers → labelers assign per-step labels → the PRM or ORM is trained on these labels → at evaluation time, the trained reward model scores unseen test solutions → the top-ranked solution's final answer is compared to ground truth → accuracy is reported.

3.3 Roadmap for the Deep Dive

First, the generator — how it is trained, why its output format matters, and its role as the fixed proposal distribution — since all solutions that the reward model evaluates originate from this component.
Second, the large-scale data collection process, including the human labeling interface, the active learning strategy for selecting which solutions to surface, and the iterative retraining loop — since these decisions determine the composition and quality of PRM800K.
Third, the outcome-supervised reward model (ORM) — its training objective, architecture, and scoring procedure — as the baseline against which the PRM is compared.
Fourth, the process-supervised reward model (PRM) — its training, the per-step prediction mechanism, the aggregation of step scores into solution scores, and the critical design choices (neutral label handling, product vs. minimum reduction) — since this is the paper's central technical contribution.
Fifth, the small-scale synthetic supervision framework, which uses the large-scale PRM (PRMlarge) as a labeling oracle to conduct controlled experiments comparing process and outcome supervision on identical datasets — since this is where the paper's most rigorous ablation evidence comes from.
Sixth, the active learning experiment within the synthetic supervision framework, including the PRMselector model and the 80/20 sampling strategy — since this quantifies the 2.6× data efficiency improvement.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods and comparison paper whose core idea is that training reward models with step-level human feedback (process supervision) produces substantially more reliable evaluators than training with final-answer feedback alone (outcome supervision), and that active learning can amplify the value of expensive human labels by surfacing the most informative solutions for annotation.

The Generator: Producing Parseable Step-by-Step Solutions

The generator is the language model that produces candidate solutions for both training data and evaluation. Critically, the paper does not attempt to improve the generator with reinforcement learning — the generator is fixed, and the paper focuses exclusively on training a better reward model that can select among the generator's outputs. This scope limitation is stated explicitly in Section 2.1:

"We do not attempt to improve the generator with reinforcement learning (RL). When we discuss outcome and process supervision, we are specifically referring to the supervision given to the reward model."

The base model for all large-scale experiments is GPT-4 (OpenAI, 2023), pretrained solely to predict the next token without any Reinforcement Learning from Human Feedback (RLHF). The small-scale base models are "similar in design to GPT-4, but they were pretrained with roughly 200 times less compute" (Section 2.2). As an additional pretraining step before any reward model training, all models are fine-tuned on MathMix, a dataset of approximately 1.5 billion math-relevant tokens, which the paper notes improves mathematical reasoning capabilities — an approach similar to the Minerva model from Lewkowycz et al. (2022).

The generator undergoes a specific formatting fine-tuning step that is essential for the entire pipeline. The paper needs individual reasoning steps to be clearly demarcated so that human labelers can evaluate each step independently and so that the PRM can make per-step predictions. The procedure is:

Few-shot generate solutions to MATH training problems from the base GPT-4 model.
Filter to only those solutions that reach the correct final answer.
Fine-tune the base model on this filtered dataset for a single epoch, producing solutions in a "newline delimited step-by-step format."

This step is explicitly described as not teaching the generator new skills — it is purely a formatting intervention: "This step is not intended to teach the generator new skills; it is intended only to teach the generator to produce solutions in the desired format" (Section 2.3). The generator's underlying mathematical capability comes from GPT-4 pretraining plus MathMix fine-tuning; the formatting step simply makes its outputs parseable by downstream components.

A crucial design choice: the generator operates at temperature 1.0 when sampling solutions (Appendix E). This relatively high temperature ensures diversity in the candidate pool — essential because best-of-N search can only find a correct solution if at least one exists in the N samples. The generator's pass rate (the fraction of sampled solutions that are correct) varies dramatically across problems, from near 0% for the hardest MATH problems to over 90% for the easiest, a distribution that the paper later uses to bin problems by difficulty quintile for analysis (Appendix G).

The generator is shared across all experiments — the same model produces solutions for PRM training data, ORM training data, and evaluation. This is important for fairness: differences in reward model performance cannot be attributed to differences in the underlying solution quality.

Data Collection: PRM800K and the Human Labeling Process

The central dataset of this paper is PRM800K, containing approximately 800,000 step-level human feedback labels across 75,000 solutions to 12,000 MATH problems. The data collection process is elaborate and contains several design decisions that significantly affect the resulting PRM's quality.

Human labeling interface and task design. Human data-labelers are presented with step-by-step solutions (produced by the large-scale generator) and asked to assign each step one of three labels:

Positive: the step is "correct and reasonable" and progresses toward the solution.
Negative: the step is "either incorrect or unreasonable."
Neutral: the step is ambiguous — it may be "subtly misleading" or "a poor suggestion that is technically still valid" (Section 2.4).

The neutral category is a practical concession to the reality that mathematical reasoning is not always cleanly classifiable. A step might use a valid but suboptimal approach, or make a statement that is technically true but could mislead a reader. Rather than forcing labelers into a binary choice that would introduce noise, the paper permits neutrality and defers the handling decision to test time: neutral labels can be treated as either positive or negative depending on which yields better performance (Appendix F, Table 4).

Labelers are given the ground-truth final answer but not a reference solution. The paper explains this choice: "We chose not to provide reference solutions to avoid biasing them towards one particular path to the solution. We chose to provide ground truth final answers since this information can sometimes help labelers resolve their own misunderstandings" (Appendix D). This is a deliberate tension: providing the answer helps labelers catch errors (if a step's implication contradicts the known answer, it's likely wrong), but the paper avoids providing the full solution path to prevent labelers from penalizing valid alternative approaches.

Quality control and labeler screening. The data collection process includes rigorous quality control. Before starting the main labeling task, all labelers must complete 30 quality control questions where researchers have pre-specified which steps should be marked as incorrect. Labelers must agree with these gold labels at least 75% of the time to be admitted. During the main task, 10-20 problems per generation are designated as additional quality control questions and randomly surfaced to labelers. Labelers whose accuracy slips are removed, and common mistakes are used to create educational materials. This continuous quality monitoring is essential because step-level correctness judgments require mathematical sophistication — the task cannot be crowdsourced to untrained annotators.

Solution selection strategy: active learning via convincing wrong-answer solutions. The most strategically important aspect of data collection is which solutions get surfaced to human labelers. The simplest approach — uniform random sampling from the generator — is inefficient because many solutions will be obviously correct (providing little new information beyond what the generator already knows) or obviously wrong (providing little information beyond what a simple outcome check would reveal). The valuable annotations are those where the current best PRM is confident but wrong — solutions that look convincing but contain subtle errors.

The paper formalizes this intuition with a specific selection criterion. They surface convincing wrong-answer solutions, defined as solutions that are:

Convincing: rated highly by the current best PRM (the model believes the solution is likely correct).
Wrong-answer: reach an incorrect final answer (we know the solution contains at least one mistake because the answer doesn't match ground truth).

The rationale is explicit: "We expect to gain more information from labeling convincing wrong-answer solutions, since we know the PRM is mistaken about at least one step in each such solution" (Section 2.4). These are precisely the solutions that reveal the PRM's blind spots — by annotating them, the training data targets the regions of solution space where the model's judgment is most unreliable.

Iterative retraining loop. The data collection is not a single batch process but an iterative cycle with multiple generations (Section 2.4, Appendix B):

Start with an initial PRM (trained on early data from phase 1, described below).
For each MATH problem, sample N solutions from the generator.
Score all N solutions with the current PRM.
Surface the top K most convincing wrong-answer solutions to human labelers (either K per problem or K globally across the dataset).
Train a new PRM on all accumulated labels.
Repeat from step 2.

This loop means the PRM is continually improving as more data is collected, and each iteration's selection strategy is informed by the best available model. The selection therefore becomes progressively more targeted: as the PRM improves, the solutions it finds "convincing" become increasingly subtle, and the errors that labelers annotate become increasingly non-obvious — precisely the kind of hard cases that improve model robustness.

The paper experimented with both problem-level top-K filtering (K solutions surfaced per problem) and global top-K filtering (K solutions total across all problems, unequally distributed). The exact values of N and K, and the balance between problem-level and global filtering, were not ablated at scale due to the cost of human data collection; the paper instead performs surrogate ablations in the synthetic supervision experiments (Section 4).

Phase 1 vs. Phase 2 data. The data collection was split into two phases with different methodologies (Appendix B):

Phase 1 (approximately 5% of PRM800K, ~40,000 step-level labels): The initial phase collected labels for multiple alternative completions at each step of a solution. For each step, labelers could see several candidate next steps and label each. If all candidates were negative, labelers could enter their own step. This seeded the dataset but was inefficient — "for many steps the alternatives were repetitive, and we found labelers spent a lot of time supervising long uninteresting solutions."
Phase 2 (approximately 95% of PRM800K): The streamlined phase pre-generated complete solutions and ended the labeling task as soon as the first negative step was encountered. This is faster for labelers (they don't waste time labeling steps after an error is found) and more closely matches the paper's supervision philosophy: "we deliberately choose to supervise only up to the first incorrect step" (Section 2.6). The iterative retraining and convincing wrong-answer selection strategy were applied during this phase.

The distributional consequence of these methodological choices is shown in Table 3: Phase 1 data is 85.1% correct solutions (because labelers could correct mistakes), while Phase 2 data is only 13.2% correct solutions (because the convincing wrong-answer strategy deliberately targets incorrect solutions). However, even in Phase 2, 74.1% of individual steps are labeled positive — most steps in incorrect solutions are still correct; only one step (the first error) needs to be wrong for the whole solution to be incorrect. This is important because it means the PRM sees many more positive step examples than negative ones, despite the solution-level imbalance.

Supervision philosophy: stopping at the first error. A critically important design choice governs what information process supervision provides. The paper specifies:

"When we provide process supervision, we deliberately choose to supervise only up to the first incorrect step."

For a correct solution, this means every step is labeled positive — identical to the information provided by outcome supervision (which would simply say "correct"). For an incorrect solution, process supervision provides two pieces of information: (a) the solution contains an error (same as outcome supervision), and (b) the precise location of the first error. The paper explicitly does not provide supervision on steps after the first error.

This choice has several motivations:

Fair comparison with outcome supervision. If process supervision labeled every step (including those after the first error), it would have an even greater information advantage over outcome supervision. By stopping at the first error, the comparison is more conservative: process supervision reveals which step went wrong, but not what happens after.
Labeling cost efficiency. Determining whether a solution is correct is equivalent to finding its first mistake. Stopping at the first error means labelers don't spend time evaluating steps that are downstream of an already-identified error — steps that are likely to be nonsensical or irrelevant.
Realism for domains without checkable answers. The paper notes that "while most MATH problems do have easy-to-check final answers, we expect this to not remain true in more complex domains." In those domains, determining correctness would require finding the first mistake anyway, so this protocol simulates the more general case.

Outcome-Supervised Reward Model (ORM) Training

The ORM serves as the primary baseline against which the PRM is compared. Its training follows the methodology of Cobbe et al. (2021) with minor hyperparameter adjustments (Section 2.5, Appendix E).

Training data generation. For the large-scale ORM, the paper uniformly samples 100 solutions per problem from the generator at temperature 1.0, without any rebalancing of correct and incorrect solutions. Each solution is automatically labeled as correct or incorrect by comparing its final answer to the ground-truth answer from the MATH dataset. This produces a training set that is "an order of magnitude larger" than PRM800K (Section 3).

Why uniform sampling rather than the active learning strategy used for PRM800K? The paper argues that training the ORM on PRM800K solutions would be problematic because "our active learning strategy has heavily biased the dataset towards wrong-answer solutions" (Section 3). An ORM trained on such a dataset might learn that most solutions are incorrect and become overly pessimistic. Uniform sampling preserves the natural distribution of correct and incorrect solutions, which is important for ORM calibration. The paper did explore "training the ORM on a superset of PRM800K solutions, by mixing in uniformly sampled solutions, but we found that this did not improve ORM performance."

Training objective. The ORM is trained as a token-level classifier: it makes a prediction at every token in the solution, but the target label is the same for all tokens — namely, whether the complete solution is correct or incorrect. The loss is standard binary cross-entropy between the model's predicted probability and the binary correctness label.

Formally, if a solution has tokens $t_1, t_2, ..., t_T$ and correctness label $y \in \{0, 1\}$ (1 if correct, 0 if incorrect), the ORM produces a probability $\hat{y}_i$ at each token position $i$ . The loss for a single solution is:

$\mathcal{L}_{\text{ORM}} = -\frac{1}{T}\sum_{i=1}^{T} \left(y \log(\hat{y}_i) + (1 - y) \log(1 - \hat{y}_i)\right)$

where $T$ is the number of tokens in the solution, $y \in \{0, 1\}$ is the binary correctness label for the entire solution, and $\hat{y}_i \in [0, 1]$ is the model's predicted correctness probability at token position $i$ .

What it computes: the average binary cross-entropy across all token positions in the solution. At every token, the model is asked to predict whether the solution it is currently generating will ultimately be correct. The target is the same for every token — correct solutions get a target of 1 at every position; incorrect solutions get a target of 0 at every position. The model must learn to estimate solution quality from partial information: early tokens see only the problem statement and the first few reasoning steps; later tokens see nearly the complete solution.

Why this form: this token-level objective is identical to the "token-level verifier" approach from Cobbe et al. (2021). The key design choice is that the prediction target is constant across tokens — the model does not receive step-specific supervision. This means the ORM must learn implicitly that certain reasoning patterns (even early in a solution) are predictive of eventual correctness or incorrectness, without ever being told which steps contain errors. The averaging across tokens means the model gets more training signal from longer solutions (more token positions to predict at), which may or may not be desirable — longer solutions tend to be harder problems where the ORM's task is more difficult.

Training hyperparameters (Appendix E). The ORM is trained for a single epoch on the dataset of model samples, without dropout, and without jointly learning a language modeling objective (unlike some prior work that augments the classification loss with a next-token prediction loss). The paper states that "performance is not sensitive to most other hyperparameters, within a reasonable range," suggesting that the ORM's behavior is largely determined by the supervision signal and data quantity rather than optimization details.

At test time, the ORM score for a solution is simply the model's prediction at the final token. The paper does not average across tokens or use any other aggregation — the last token's prediction is taken as the model's best estimate of solution correctness, since it has seen the entire chain-of-thought. This is the natural choice given the training objective (all tokens predict the same target, but the final token has the most information).

Key weakness: false positives from automatic grading. A critical limitation of the ORM training procedure is that the correctness labels come from automatic final-answer checking, which is imperfect. A solution that reaches the correct final answer through incorrect reasoning will be labeled "correct" and provide misleading supervision. The paper acknowledges this explicitly: "false positives solutions that reach the correct answer with incorrect reasoning will be misgraded" (Section 2.5). This weakness is one of the factors that the small-scale synthetic supervision experiments (Section 4) are designed to isolate and measure.

Scale comparison with PRM800K. The large-scale ORM training set is substantially larger than PRM800K — 100 samples per problem uniformly sampled versus PRM800K's actively selected, biased dataset. The paper is explicit that "these two training sets are not directly comparable" (Section 3). The ORM benefits from more data and a natural label distribution; the PRM benefits from richer per-sample supervision and a strategically selected dataset. The large-scale comparison therefore reflects each method at its best, not a controlled experiment — which is why the small-scale synthetic supervision experiments in Section 4 are essential for establishing causality.

Process-Supervised Reward Model (PRM) Training and Usage

The PRM is the paper's central technical artifact. Unlike the ORM, which makes a single correctness prediction per solution, the PRM makes a correctness prediction at every reasoning step, trained on the human step-level labels in PRM800K.

Training as a language model fine-tuning task. The PRM is trained by fine-tuning the MathMix-pretrained base model to predict step-level labels. This is framed as a standard language modeling task: after the last token of each reasoning step, the model predicts a special token indicating the step's label (positive, negative, or neutral). The paper states:

"This prediction takes the form of a single token, and we maximize the log-likelihood of these target tokens during training. The PRM can therefore be trained in a standard language model pipeline without any special accommodations." (Section 2.6)

This design choice is important for practical reasons — it means no custom architecture, loss function, or training loop is needed; the PRM is simply a language model fine-tuned to output label tokens at step boundaries. The classification head is the standard language modeling head (a linear layer over the vocabulary followed by softmax), not a separate binary classifier.

Training objective. At each step boundary, the PRM produces a probability distribution over the label vocabulary {positive, negative, neutral}. The loss is the standard negative log-likelihood of the human-assigned label:

$\mathcal{L}_{\text{PRM}} = -\log p(\text{label} \mid \text{context up to and including this step})$

where the context includes the problem statement and all preceding steps. The model is trained to predict the label for each step given everything that has come before it.

What it computes: for each step in a solution, the model assigns a probability to each possible label (positive, negative, neutral), and the loss penalizes deviations from the human-assigned label. The model learns to evaluate reasoning incrementally: after reading the problem and the first step, it predicts whether that step is correct; after reading the first two steps, it predicts whether the second step is correct (conditioned on the first step being correct); and so on. This is fundamentally different from the ORM, which makes the same prediction at every token.

Why this form: this is the natural maximum-likelihood objective for a multi-class classification task embedded in a language model. The key property is that the model receives localized feedback — it learns what correct reasoning looks like at each step, not just what correct final answers look like. This is what the paper means by "process supervision makes credit assignment easier" (Section 6.1): the model doesn't have to infer which step went wrong; it is explicitly told.

Training hyperparameters. The paper notes that fine-tuning from a language modeling objective to a classification objective is a "large distribution shift" and that low learning rates were important to stable PRM training (Appendix F.1). All PRMs are trained for 2 epochs. On smaller early datasets, 2 epochs improve performance over 1 epoch; on larger later datasets, the benefit diminishes but 2-epoch training is maintained for consistency.

Step-level score interpretation. At test time, the PRM produces a probability distribution over {positive, negative, neutral} for each step. The step-level score is defined as the probability that the step's label is positive. If the PRM outputs probabilities $p_{\text{pos}}, p_{\text{neg}}, p_{\text{neut}}$ , the step score is simply $p_{\text{pos}}$ .

However, the neutral label introduces a design choice: should a neutral step be treated as positive or negative for scoring purposes? The paper experiments with both options (Appendix F.2, Table 4). Treating neutral as positive means neutral steps don't penalize the solution; treating neutral as negative means they do. The best-performing strategy is to treat neutral as positive, which makes sense: neutral steps are, by definition, technically valid if suboptimal — they shouldn't count against a solution's correctness.

Solution-level score aggregation. To compare multiple solutions (as required for best-of-N selection), the per-step scores must be reduced to a single solution-level score. The paper considers two reduction operations:

Product: the solution score is the product of all step-level scores: $\prod_{i=1}^{S} p_{\text{pos}}^{(i)}$ where $S$ is the number of steps.
Minimum: the solution score is the minimum step-level score: $\min_{i=1}^{S} p_{\text{pos}}^{(i)}$ .

The product reduction has a natural probabilistic interpretation: it is the probability that every step is correct, assuming step correctness is independent given the preceding context. This is the interpretation the paper explicitly adopts: "we define the PRM score for a solution to be the probability that every step is correct under the PRM. We implement this as the product of the correctness probabilities for each step" (Section 2.6).

The minimum reduction has a different interpretation: it is the probability of the least likely correct step, effectively grading the solution by its weakest link.

Table 4 shows the results: the product reduction achieves 78.2% best-of-1860 accuracy versus 77.6% for the minimum reduction (both treating neutral as positive). The difference is minor — less than one percentage point — but the product reduction consistently edges ahead. The paper uses the product reduction throughout the main experiments, noting that it "does create a slight bias against solutions with a larger number of steps" (since multiplying more numbers between 0 and 1 produces a smaller product, all else being equal).

Why product over minimum? The product considers all steps cumulatively, meaning a solution with many highly confident correct steps and one slightly uncertain step gets a moderate score, while a solution with uniformly moderate confidence across all steps gets a lower score. The minimum, by contrast, ignores everything except the worst step. For mathematical reasoning, where errors are typically localized (one mistake derails the solution) but correctness elsewhere is still informative, the product captures more signal. A solution where every step is 95% confident should score higher than one where every step is 60% confident, even if both have a single 50% step — the minimum would score them identically.

Efficient test-time computation. A practical advantage of the PRM architecture: "to determine the step-level predictions at test time, it suffices to perform a single PRM forward pass over the whole solution" (Section 2.6). Because the model predicts a label token at each step boundary, and the step boundaries are marked by newline delimiters in the generator's formatted output, a single autoregressive forward pass produces all step-level predictions sequentially. There is no need for separate forward passes per step.

Large-scale PRM training set composition. The PRM is trained on PRM800K, which has the distributional properties shown in Table 3: 14.2% of solutions are correct, but 73.1% of individual steps are correct. This asymmetry — mostly incorrect solutions but mostly correct steps — is a direct consequence of the "stop at first error" supervision philosophy and the convincing wrong-answer selection strategy. The PRM sees many examples of correct individual steps (from the early parts of incorrect solutions) and fewer examples of incorrect steps (the first errors). This may actually be beneficial: learning what correct reasoning looks like is arguably more important than cataloging every possible failure mode.

Visualization of PRM scores (Figure 2). The paper provides a compelling qualitative example: two solutions to the same problem, one correct and one incorrect, with step-level PRM scores visualized as green (high score) or red (low score) backgrounds. The PRM correctly identifies the mistake in the incorrect solution — early steps are green, then the problematic step is red. This visualization demonstrates that the PRM's step-level predictions are not just statistically useful but interpretable — a human can read the solution, look at the color coding, and immediately see where the model thinks the reasoning broke down.

The Active Learning Strategy in Detail

The active learning component is not an afterthought but a central methodological contribution. The paper quantifies its impact at 2.6× data efficiency (Section 4.2), meaning that the same PRM performance can be achieved with roughly 2.6 times fewer human labels when using active selection versus uniform sampling.

Intuition and formalization. The core insight is that not all solutions are equally informative for training. A solution that the current PRM already scores correctly (high score for correct, low score for incorrect) provides little new information — the model already "knows" the right answer. A solution that the PRM scores incorrectly (high score for incorrect, low score for correct) is a valuable training example because it reveals a blind spot in the model's current knowledge.

The paper operationalizes this as convincing wrong-answer solutions — solutions that are:

Wrong-answer: the final answer does not match ground truth, so we know the solution contains at least one error (no ambiguity).
Convincing: the current PRM assigns a high score, meaning the model is confident the solution is correct — and is therefore wrong.

By focusing human labeling effort on these solutions, the active learning strategy ensures that each annotation addresses a specific weakness in the current model. A uniform sampling strategy would waste many annotations on solutions the model already evaluates correctly.

Iterative retraining and the PRMselector model (Section 4.2). In the small-scale synthetic supervision experiments, the paper implements active learning using a specific protocol:

Train a small reward model, PRMselector, on a single sample per problem (a minimal dataset).
Use PRMselector to score 1000 samples per problem from the small-scale generator.
For each problem, select N samples to label, where 80% are the most convincing wrong-answer samples (according to PRMselector) and 20% are the most convincing samples that remain (which may be right-answer or wrong-answer).
Score the selected samples with PRMlarge (the large-scale PRM serving as a synthetic oracle) and train on those scores.

The 80/20 split is important: 80% wrong-answer ensures the training data is rich in informative errors, but 20% mixed-answer prevents the dataset from becoming too heavily biased toward incorrect solutions. If the PRM were trained exclusively on incorrect solutions, it might learn that all solutions are likely incorrect and lose calibration on correct solutions. The 20% buffer provides correct examples to maintain balance.

The paper estimates the 2.6× data efficiency improvement by comparing the slopes of the best-fit lines in Figure 4a for the active learning condition versus the uniform sampling condition. A steeper slope means each additional labeled sample contributes more to performance; the ratio of slopes quantifies the relative efficiency. This is a simple but effective way to measure the value of active selection without requiring a matched-budget comparison at every data scale.

A negative result on iterative retraining of PRMselector. The paper also attempted to iteratively retrain PRMselector throughout data collection — after each generation of labeling, retrain PRMselector on all accumulated labels, then use the improved selector for the next generation's sample selection. However, the paper reports:

"Unfortunately, we observed instability in this process which we were unable to diagnose. The resulting reward models performed no better than the models described above." (Section 4.2)

This is an honest negative result: the intuitive idea of iteratively improving the selector did not work in practice. The instability may stem from distribution shift — each retrained PRMselector sees data selected by its predecessor, creating a feedback loop that can amplify biases. The paper leaves this as "a compelling direction for future research."

Relation to the large-scale data collection. The large-scale PRM800K data collection also used active learning in Phase 2, but the selection was done by the current best large-scale PRM (not a separate selector model) and the retraining was done periodically between generations. The small-scale experiment with PRMselector is a surrogate that allows controlled measurement of the active learning benefit — something that would be prohibitively expensive to ablate with human labelers at scale.

Synthetic Supervision Framework: Using `PRMlarge` as an Oracle

The small-scale experiments in Section 4 are designed to address confounding factors in the large-scale comparison. The key innovation is using the large-scale PRM (PRMlarge) as a synthetic labeling oracle for training smaller reward models. This enables experiments that would be impossible with human labelers: generating arbitrary amounts of process or outcome supervision, varying the supervision type while holding the training data identical, and measuring the impact of active learning in a controlled setting.

How PRMlarge provides process supervision. For each step in a solution, PRMlarge outputs a probability distribution over {positive, negative, neutral}. To convert these probabilities into discrete labels, the paper sets a threshold:

"Any step that PRMlarge assigns a negative label with greater than 20% probability is considered incorrect." (Appendix H)

This threshold is chosen based on the observation that "PRMlarge is slightly miscalibrated in the direction of favoring positive labels." In other words, the model tends to be overconfident that steps are correct, so a modest threshold (20% negative probability) is sufficient to catch errors without generating excessive false negatives.

The process supervision procedure then mirrors the human data collection: label steps as positive or negative up to the first step marked negative, then stop. Steps after the first error are not labeled.

How PRMlarge provides outcome supervision. For outcome supervision, PRMlarge evaluates the entire solution. The solution is labeled correct if and only if PRMlarge considers every step to be correct (using the same 20% threshold). If any step exceeds the 20% negative probability threshold, the solution is labeled incorrect. This is a cleaner outcome signal than final-answer checking because it doesn't suffer from false positives — a solution that reaches the right answer through incorrect reasoning will have at least one step that PRMlarge flags as negative, and will thus be correctly labeled as incorrect.

The three supervision series. In the experiment shown in Figure 4, the paper trains three series of reward models on identical sets of solutions, varying only the supervision signal:

PRM (PRMlarge supervised): trained with process supervision from PRMlarge — step-level labels up to the first error.
ORM (PRMlarge supervised): trained with outcome supervision from PRMlarge — a single correctness label per solution based on whether all steps are correct.
ORM (final-answer supervised): trained with outcome supervision from automatic final-answer checking — the standard ORM approach.

The key comparison is series 1 vs. series 2: both use PRMlarge as the supervision source, so any performance difference can be attributed to the form of supervision (process vs. outcome) rather than the quality of the supervision source. Series 3 vs. series 2 isolates the effect of false positives from automatic grading — how much ORM performance is damaged by spurious solutions that reach the correct answer through incorrect reasoning?

Why this framework matters. Without synthetic supervision, comparing process and outcome supervision requires either (a) comparing models trained on different datasets (as in the large-scale experiments), which confounds supervision type with dataset size and composition, or (b) collecting both process and outcome human labels for identical datasets, which is prohibitively expensive. The synthetic framework solves this by using a high-quality model as a cheap, consistent labeling oracle, enabling the first clean comparison of supervision types in the literature.

Best-of-N Evaluation Protocol

The paper evaluates reward models by their ability to perform best-of-N selection over uniformly sampled solutions from the generator (Section 2.1, Section 3). This is the primary metric throughout — not the reward model's classification accuracy, not its calibration error, but its downstream utility for selecting good solutions from a candidate pool.

Procedure. For each test problem:

Sample N solutions from the generator at temperature 1.0 (the same generator used for training data generation). For the large-scale experiments, N = 1860 total solutions are generated per problem; for evaluation at smaller N, results are averaged over many random subsamples (visualized as variance bands in Figure 3).
Score each solution with the reward model. For the ORM, the score is the final-token prediction. For the PRM, the score is the product of per-step positive probabilities (neutral treated as positive).
Select the top-ranked solution — the one with the highest score.
Automatically grade the selected solution by comparing its final answer to the ground-truth answer from the MATH dataset, using the grading function from Hendrycks et al. (2021).
Report accuracy: the fraction of test problems for which the top-ranked solution is correct.

Why best-of-N as the evaluation metric? This metric directly measures the reward model's practical utility. A reward model that achieves high classification accuracy but fails to identify the single best solution in a candidate pool is not useful for downstream applications like RLHF reward shaping or rejection sampling. Best-of-N selection is also the most direct test of the reward model's discriminative ability: can it distinguish correct reasoning from plausible-sounding incorrect reasoning?

Comparison with majority voting. The paper includes majority voting as a strong baseline (Wang et al., 2022; Lewkowycz et al., 2022). Majority voting selects the most common final answer among the N solutions — no learned reward model is involved. It is a surprisingly competitive baseline because incorrect solutions tend to produce diverse wrong answers (many ways to be wrong) while correct solutions converge on the same answer (one way to be right). Any learned reward model must outperform majority voting to justify its existence. The paper also experimented with "RM-weighted voting" (Li et al., 2022; Uesato et al., 2022) — combining the PRM's scores with answer frequency — but found this "did not noticeably improve performance" (Section 3).

Test set and evaluation scope. Due to including 4,500 MATH test problems in the PRM800K training set (to avoid overfitting on the 7,500 training problems), the paper evaluates only on the remaining 500 MATH test problems, selected uniformly at random (Appendix C). Figure 5 confirms that this subset is representative of the full MATH test set in both difficulty distribution and subject distribution. The paper also evaluates on a separate out-of-distribution set of 224 STEM questions from recent AP and AMC exams (Section 5, Table 1), using best-of-100 selection with 100 samples per problem.

Difficulty quintile analysis (Appendix G, Figure 6). To understand where the PRM's advantage comes from, the paper bins test problems into five quintiles based on the generator's pass rate (the fraction of sampled solutions that are correct). Quintile 1 is easiest (high pass rate); quintile 5 is hardest (low pass rate). The analysis reveals that the PRM outperforms the ORM across all difficulty levels, but the gap is particularly consequential on hard problems where the ORM struggles to distinguish genuinely correct solutions from plausible-looking incorrect ones. Notably, on the easiest problems (quintile 1), the ORM's performance actually decreases slightly as N increases — the model is being fooled by adversarial examples that score highly under the ORM but are incorrect, a phenomenon the PRM is more robust to.

Out-of-Distribution Generalization Evaluation

To assess whether the PRM's advantage holds beyond the MATH test set, the paper evaluates on a held-out set of 224 STEM questions drawn from the most recent AP Physics, AP Calculus, AP Chemistry, AMC10, and AMC12 exams (Section 5). These exams were released after GPT-4's pretraining dataset was compiled, so the model is guaranteed not to have seen these specific problems during training.

The evaluation uses best-of-100 selection (100 samples per problem) and compares the large-scale ORM, large-scale PRM, and majority voting. Table 1 reports results: the PRM achieves 72.9% aggregate accuracy versus 63.8% for the ORM and 61.3% for majority voting. The PRM outperforms both baselines on every individual exam, with particularly large margins on AP Calculus (86.7% vs. 68.9% ORM) and AMC10/12 (53.2% vs. 49.1% ORM vs. 32.8% majority voting).

This out-of-distribution evaluation serves two purposes. First, it addresses concerns about test set contamination — even if some MATH problems appeared in GPT-4's pretraining data, these recent exam problems did not, so the PRM's advantage cannot be attributed to memorization. Second, it demonstrates that the PRM's benefits generalize beyond the MATH dataset's specific problem style, suggesting that process supervision produces a reward model that has learned something fundamental about evaluating mathematical reasoning, not just something about the MATH dataset's idiosyncrasies.

Summary of Design Choices and Their Justifications

Generator fixed (no RL) throughout: isolates reward model quality as the sole variable; avoids confounding improvements from generator fine-tuning with improvements from better reward modeling.
Formatting fine-tuning on correct-only solutions: teaches the generator to produce parseable step-by-step output without altering its mathematical capability — ensures steps are clearly delineated for both human labelers and the PRM.
Human labeling with positive/negative/neutral categories: the neutral category acknowledges real ambiguity in mathematical reasoning and defers the handling decision to test time rather than forcing noisy binary labels.
Supervision stops at first error: provides a fair comparison with outcome supervision (which only indicates error existence) and keeps labeling costs manageable; process supervision's advantage comes from localizing the error, not from supervising subsequent steps.
Active learning via convincing wrong-answer solutions: targets annotations at the PRM's blind spots; 2.6× data efficiency means the same performance with less than half the human labeling budget.
PRM as a standard language model fine-tuning task: no custom architecture needed; the model predicts label tokens at step boundaries, leveraging the same infrastructure used for pretraining.
Product reduction for solution-level scoring: natural probabilistic interpretation (probability all steps are correct); slightly outperforms minimum reduction; accepts a small bias against longer solutions.
Synthetic supervision with PRMlarge oracle: enables controlled experiments comparing supervision types on identical datasets; isolates the form of supervision from the quality of the supervision source.
Two-regime experimental design (large-scale for SOTA, small-scale for controlled comparisons): large-scale shows what's possible with maximum resources; small-scale establishes causality with clean ablations.
Best-of-N evaluation metric: directly measures practical utility for downstream applications; majority voting provides a strong non-learned baseline.
Out-of-distribution evaluation on recent exam problems: addresses contamination concerns; demonstrates generalization beyond MATH-specific patterns.

4. Key Insights and Innovations

Innovation 1: Reframing the ORM-vs-PRM Debate as a Data Scaling Phenomenon, Not a Binary Choice

The most intellectually significant move in this paper is not simply showing that PRMs outperform ORMs — it's reframing the entire comparison as a question of data scaling regimes rather than a binary "which method is better" contest. Prior work, most notably Uesato et al. (2022), had reached the tentative conclusion that process and outcome supervision produced similar final performance, which might lead a reader to conclude that the extra cost of human step-level labels isn't justified. The dominant assumption in the field was that the choice between ORMs and PRMs was a fixed tradeoff: richer signal (process) versus cheaper data (outcome), with roughly equivalent asymptotes.

This paper's reframing is fundamentally different. Figure 4a tells the story: at small data volumes (1-10 labeled solutions per problem), process supervision and outcome supervision produce nearly identical reward model performance. This reproduces the Uesato et al. finding — in the low-data regime, the two methods are indeed comparable. But as the data scale increases, the curves diverge. Process supervision shows a steeper improvement trajectory, and by 200 labeled solutions per problem, the PRM substantially outperforms both forms of outcome supervision. The key intellectual move is recognizing that Uesato et al. (2022) didn't produce a conflicting result — they produced a result at a particular point on a scaling curve, and the true difference between methods only becomes apparent at larger scale.

This is a conceptual advance, not merely an empirical one, because it changes how researchers should think about comparing supervision methods. Rather than asking "is process supervision better than outcome supervision?" — which implicitly assumes a single answer — the better question is "at what data scale does process supervision's advantage manifest, and how much data is needed to realize it?" This mirrors the intellectual shift that occurred in pretraining with the Chinchilla scaling laws (Hoffmann et al., 2022): the insight wasn't that larger models are better, but that the optimal model size depends on the data budget in a predictable way. Similarly, this paper suggests that the optimal supervision method depends on the labeling budget, with process supervision becoming increasingly preferable as resources allow.

The paper makes this reframing explicit in Section 7.1: "The data scaling trend in Figure 4a suggests that a small amount of process supervision and a large amount of outcome supervision do in fact lead to similar performance, consistent with the results from Uesato et al. (2022). The trend also shows that process supervision beats outcome supervision when scaled up." This is a reconciliation, not a contradiction — and that reconciliation is more valuable to the field than simply reporting a higher number.

Why this matters beyond this paper: it implies that the entire literature comparing supervision methods needs to report data scaling curves, not single-point comparisons. A paper that trains an ORM on 1000 samples/problem and a PRM on 10 samples/problem and finds the ORM wins hasn't demonstrated that outcome supervision is superior — it's demonstrated that more data beats less data, which tells us nothing about the asymptotic properties of either method. The field needs to internalize that supervision method comparisons are fundamentally about sample efficiency curves, and this paper provides the template for how to conduct such comparisons.

Innovation 2: Process Supervision as a Negative Alignment Tax — Safety That Improves Performance

A second distinctive contribution is the paper's reframing of process supervision as an alignment intervention that incurs a negative alignment tax. The standard narrative in AI alignment is that safety measures typically reduce capability — making models more cautious, more filtered, or more constrained comes at a performance cost, known as an alignment tax (Ouyang et al., 2022; Askell et al., 2021). The existence of an alignment tax creates an unfortunate dynamic: organizations under competitive pressure to deploy the most capable model face an incentive to skimp on safety measures, since safer models perform worse.

This paper identifies a rare case where the alignment tax is negative: process supervision simultaneously improves both safety-relevant properties (interpretable reasoning, human-endorsed chain-of-thought, resistance to reward hacking) and raw capability (higher best-of-N accuracy). Section 6.2 makes this argument explicitly:

"Our results show that process supervision in fact incurs a negative alignment tax. This could lead to increased adoption of process supervision, which we believe would have positive alignment side-effects."

The alignment argument is not merely a rhetorical flourish — it's grounded in specific mechanisms that the paper identifies. Process supervision directly rewards models for following a human-endorsed reasoning process, meaning the PRM's scores reflect how an answer was reached, not just what the answer is. This addresses a well-documented failure mode of outcome supervision: "models trained with outcome supervision regularly use incorrect reasoning to reach the correct final answer" (Section 1, citing Zelikman et al., 2022; Creswell et al., 2022). An ORM, trained to predict final-answer correctness, may learn to reward solutions that happen to reach the right number through flawed logic — and a generator subsequently optimized against this ORM would learn to produce such solutions. The PRM, by contrast, penalizes incorrect intermediate steps regardless of the final answer, closing off this avenue of reward hacking.

What makes this insight fundamental rather than incremental is that it changes the adoption calculus. If process supervision were merely "safer but equally good," adoption would depend on how much organizations value safety relative to capability — a political and economic question with no guaranteed answer. The finding that process supervision is better on both axes means there is no tradeoff to debate. The safest method is also the most capable method, which removes the incentive structure that typically undermines safety adoption. This is a genuinely novel contribution to the alignment literature: a case study in how better supervision methodology can align safety and capability incentives, rather than pitting them against each other.

The paper is appropriately cautious about generalizability, noting that "it is unknown how broadly these results will generalize beyond the domain of math" (Section 6.2). But even if the negative alignment tax proves domain-specific, the conceptual contribution — identifying and naming this phenomenon — provides a template for what to look for in other domains. The field now has a term for "safety interventions that also improve capability" and a concrete example to study.

While active learning as a general concept predates this paper, the specific application to reward model training for reasoning tasks represents a methodological innovation with substantial practical impact. The key insight is that the most valuable human labels are those that reveal where the current reward model is confidently wrong, not those that confirm what the model already knows.

Prior to this work, the default approach to collecting human feedback for reward models was essentially uniform sampling — surface whatever the generator produces and label it. This is wasteful because most solutions fall into two low-information categories: obviously correct solutions (the PRM already scores them highly and they are indeed correct — no new information) and obviously incorrect solutions (the PRM already scores them low — no new information). The paper's "convincing wrong-answer" strategy targets the narrow slice of solution space where annotation provides maximum information: solutions that the current PRM rates highly but that reach an incorrect final answer. These are precisely the cases where the PRM's judgment is unreliable, and labeling them directly addresses the model's blind spots.

What elevates this from a simple trick to an intellectual contribution is the 2.6× quantification (Section 4.2, Figure 4a) and the experimental design that isolates the effect. The paper doesn't just claim active learning helps — it measures the slope of the data scaling curve with and without active selection, showing that each actively-selected label contributes roughly 2.6 times as much to final performance as a uniformly-sampled label. This is a concrete, actionable number that tells practitioners exactly what they gain by implementing active learning, and it's derived from a controlled experiment (the synthetic supervision framework) rather than a confounded large-scale comparison.

The 80/20 split (80% convincing wrong-answer, 20% most convincing remaining) is another conceptual contribution, even if presented as a practical detail. Pure wrong-answer selection would bias the training set entirely toward incorrect solutions, potentially damaging the PRM's calibration on correct solutions. The 20% buffer acknowledges that a good reward model must be accurate on both correct and incorrect solutions, and that training exclusively on errors would produce a model that is overly pessimistic. This is a balance principle for active learning in binary evaluation tasks that generalizes beyond this specific setup.

The negative result on iterative PRMselector retraining (Section 4.2) — "we observed instability in this process which we were unable to diagnose" — is itself a meaningful contribution. The intuitive approach of iteratively improving the selection model using accumulated labels didn't work, and the paper reports this honestly rather than burying it. This is valuable because it warns future researchers away from a natural but apparently unstable approach, and it identifies "instability in iterative retraining for active learning" as an open problem. Negative results that are clearly reported and properly contextualized are intellectually valuable, and this one qualifies.

Innovation 4: Large PRMs as Synthetic Supervision Oracles — Breaking the Human-Labeling Bottleneck for Controlled Experiments

The synthetic supervision framework in Section 4 represents a methodological innovation in experimental design for reward model research. The core problem the paper faced was this: to conduct a clean comparison of process versus outcome supervision, you need both types of labels on identical datasets, and you need enough data to draw a scaling curve. With human labelers, this is prohibitively expensive — you'd need to pay annotators to produce both per-step and per-solution labels for thousands of solutions, at a cost that would make systematic ablation studies infeasible.

The paper's solution is elegant: use a large, carefully-trained PRM (PRMlarge) as a labeling oracle for smaller models. PRMlarge can generate arbitrary amounts of synthetic process supervision (step-level labels with a simple threshold) or synthetic outcome supervision (solution-level labels based on whether all steps pass the threshold) at essentially zero marginal cost. This enables experiments that are otherwise impossible: training multiple reward models on identical datasets with different supervision signals, sweeping the data quantity from 1 to 200 samples per problem, and measuring the impact of active learning — all without spending a dollar on additional human annotation.

What makes this distinctive rather than obvious is the care with which the paper validates the approach and interprets its limitations. The synthetic supervision experiments don't replace the large-scale human-label experiments — they complement them. The large-scale results (Section 3) establish the real-world performance ceiling with human labels; the small-scale synthetic results (Section 4) establish why the large-scale results look the way they do, by isolating the form of supervision from confounding variables like dataset size and composition. This two-regime design — large-scale for state-of-the-art, small-scale for controlled science — is a template for empirical research in domains where the gold-standard intervention (human annotation) is too expensive for systematic ablation.

The specific finding that PRMlarge-based outcome supervision substantially outperforms final-answer-based outcome supervision (Figure 4b) is significant in its own right. It quantifies the damage caused by false positives from automatic grading — solutions that reach the right answer through wrong reasoning. The fact that replacing automatic grading with PRMlarge's judgment improves ORM performance demonstrates that part of the PRM's advantage comes from avoiding this specific failure mode, but not all of it (since PRMlarge-supervised ORMs still underperform PRMlarge-supervised PRMs). This decomposition — how much of the PRM advantage comes from better supervision of individual solutions versus better supervision format — would be impossible to measure without the synthetic framework.

This innovation also has implications beyond this paper. The ability to use a large, high-quality model as a labeling oracle for smaller models is a general pattern that appears in other contexts (e.g., knowledge distillation, weak-to-strong generalization). This paper provides a concrete case study and a set of methodological practices (threshold selection based on calibration analysis, the stop-at-first-error protocol for simulating human data collection constraints) that can inform similar efforts in other domains. The release of PRM800K further amplifies this contribution by providing the human-labeled dataset that other researchers can use to train their own "oracle" PRMs for downstream experiments.

5. Experimental Analysis

Evaluation Methodology

Dataset. All primary experiments use the MATH dataset (Hendrycks et al., 2021), consisting of high-school competition-level mathematics problems. The paper uses the specific split from Lightman et al. (2022): 12,000 training problems and 500 held-out test problems. The authors selected these 500 test problems uniformly at random from the original MATH test split, after expanding the PRM800K training set to include 4,500 MATH test split problems to avoid overfitting on only 7,500 training problems (Section 3, Appendix C). For out-of-distribution evaluation, the paper uses a separate set of 224 STEM questions drawn from recent AP Physics, AP Calculus, AP Chemistry, AMC10, and AMC12 exams — all released after GPT-4's pretraining data compilation, guaranteeing no contamination (Section 5, Table 1).
Base model(s). The large-scale experiments use GPT-4 (OpenAI, 2023) as the base model, pretrained solely for next-token prediction without RLHF (Section 2.2). The small-scale experiments use base models "similar in design to GPT-4, but they were pretrained with roughly 200 times less compute" (Section 2.2). All models undergo an additional pretraining stage on MathMix, a dataset of approximately 1.5 billion math-relevant tokens described in Appendix A, which the authors find improves mathematical reasoning capabilities similarly to the Minerva approach (Lewkowycz et al., 2022). The generator is then fine-tuned for a single epoch on few-shot-generated, filtered-to-correct MATH training solutions to enforce a newline-delimited step-by-step output format (Section 2.3). The choice of GPT-4 is motivated by its position as a state-of-the-art model with non-trivial but imperfect MATH performance — strong enough to produce subtle errors that benefit from precise supervision, but not so strong that the task is trivial.
Metrics. The primary metric throughout is best-of-N accuracy — the fraction of test problems for which the top-ranked solution (according to the reward model) is correct. For each test problem, the generator produces N candidate solutions, the reward model scores them, the highest-scoring solution is selected, and its final answer is automatically compared to the ground-truth answer using the grading function from Hendrycks et al. (2021). This metric directly measures the reward model's downstream utility for solution selection. For the large-scale experiments, the paper evaluates at N = 1 through N = 1860 (the total generated per problem), with variance bands computed across many random subsamples for N ≤ 1000 (Figure 3). For small-scale synthetic experiments, evaluation is at best-of-500 (Figure 4a) and best-of-N across varying N (Figure 4b). For out-of-distribution evaluation, best-of-100 is used (Table 1). The paper also reports generator pass rate — the fraction of sampled solutions that reach the correct final answer — which is used to bin problems into difficulty quintiles for analysis (Appendix G).
Baselines. The paper compares against three baselines:
- Outcome-supervised Reward Model (ORM) trained with final-answer checking (Cobbe et al., 2021): the standard approach where a reward model is trained to predict whether a complete solution reaches the correct answer, using automatically generated labels from the MATH dataset's ground-truth answers (Section 2.5, Section 3).
- Majority voting (Wang et al., 2022; Lewkowycz et al., 2022): selects the most common final answer among the N generated solutions without any learned reward model, exploiting the tendency of incorrect solutions to produce diverse wrong answers while correct solutions converge on the same answer (Section 3, Figure 3).
- ORM supervised by PRMlarge (Section 4): in the small-scale synthetic experiments, an additional baseline where outcome supervision is provided by the large-scale PRM rather than by final-answer checking, isolating the effect of false positives from automatic grading.
Generation budget / compute accounting. The paper measures computation in terms of number of solutions sampled per problem (N) from the generator. For the large-scale experiments, 1860 solutions are generated per test problem; evaluation at smaller N uses random subsampling from this pool (Figure 3). The ORM training set uses 100 uniformly sampled solutions per problem (Section 3). The PRM training set (PRM800K) contains approximately 75,000 solutions to 12,000 problems — roughly 6.25 solutions per problem on average, though the active learning strategy makes this distribution non-uniform (Appendix B). In the small-scale synthetic experiments, the data quantity is systematically varied from 1 to 200 solutions per problem (Figure 4a). There is no FLOPs accounting or latency analysis — compute is measured purely in generation count, and the cost of human labeling is not quantified in computational terms.
Cross-validation / statistical protocol. The paper does not use traditional cross-validation for the large-scale experiments — the 500-question test set is held out once and results are reported on this fixed set. However, for the best-of-N evaluation with N < 1860, the paper subsamples from the full pool of 1860 solutions many times and visualizes the variance as shaded regions in Figure 3. For the small-scale synthetic experiments (Section 4), results are reported as mean and standard deviation across three random seeds (Figure 4 caption: "Mean and standard deviation is shown across three seeds"). The test set composition is verified to be representative of the full MATH test set in terms of difficulty level and subject distribution (Appendix C, Figure 5). For the out-of-distribution evaluation (Section 5), 100 solutions are generated per problem, and best-of-100 accuracy is reported without variance estimates (Table 1). The difficulty quintile analysis (Appendix G) partitions the 500 test problems into five bins of 100 each based on the generator's pass rate, but no cross-validation or confidence intervals are reported within each bin, meaning the per-quintile results are based on approximately 100 problems each — a relatively small sample for fine-grained comparison.

The paper also includes a specific decontamination protocol: string-matching heuristics were applied to remove MATH problems from the MathMix pretraining dataset, though the authors acknowledge that "humans can post hard-to-detect rephrasings of a problem online, it is difficult to make any strong guarantees" (Section 6.3). The out-of-distribution evaluation on recent AP and AMC exams provides a contamination-free verification of the main findings.

Main Quantitative Results

Large-Scale ORM vs. PRM Comparison (Section 3)

The headline result appears in Figure 3: the process-supervised reward model (PRM) substantially outperforms both the outcome-supervised reward model (ORM) and majority voting across all values of N, with the performance gap widening as N increases. At the maximum budget of N = 1860 solutions per problem:

PRM best-of-1860: 78.2% of test problems solved.
ORM best-of-1860: 72.4% — a gap of 5.8 percentage points.
Majority voting best-of-1860: 69.6% — the PRM outperforms this non-learned baseline by 8.6 percentage points.

The widening gap is the crucial pattern. At small N (N ≈ 1-10), all three methods are relatively close — the PRM's advantage is modest and the ORM performs similarly to majority voting. But as N grows:

The ORM curve rises more slowly than the PRM curve, producing an increasingly large separation. The shaded variance bands in Figure 3 show that this separation is well outside the sampling noise for N > 100.
Majority voting plateaus earlier and lower than both learned reward models, confirming that even an imperfect learned reward model can extract more value from additional samples than simple answer-frequency counting.

The magnitude of the PRM's advantage is particularly striking given that the ORM was trained on an order of magnitude more data — 100 uniform samples per problem versus PRM800K's approximately 6.25 actively-selected samples per problem (Section 3). The PRM achieves higher performance despite being trained on substantially fewer solutions, demonstrating that the quality and type of supervision can outweigh raw data quantity.

The paper explicitly notes that the ORM and PRM training sets are "not directly comparable" (Section 3) — the ORM benefits from uniform sampling (preserving the natural distribution of correct and incorrect solutions) while the PRM benefits from richer per-sample supervision and active selection. This is why the large-scale comparison is complemented by the controlled small-scale experiments in Section 4.

Difficulty quintile breakdown (Appendix G, Figure 6). When problems are binned by the generator's pass rate into five quintiles (quintile 1 = easiest, quintile 5 = hardest), the PRM outperforms the ORM across all difficulty levels, not just the hardest problems:

On the easiest problems (quintile 1), the ORM's performance actually decreases slightly as N increases — a signature of the reward model being fooled by adversarial examples that score highly under the ORM but are incorrect. The PRM remains robust, with its performance continuing to improve or stay flat as N grows.
On the hardest problems (quintile 5), increasing N produces the largest relative gains, and the PRM's advantage over the ORM is most consequential. This makes sense: hard problems have low generator pass rates, so finding a correct solution requires searching over many candidates, and a reliable reward model is essential to identify the rare correct solution among many plausible-looking incorrect ones.
The gap is visible even on medium-difficulty problems (quintiles 2-4), confirming that the PRM's advantage is not driven solely by a few outlier hard problems where the ORM completely fails.

The paper does not report exact per-quintile numbers in the text; the evidence is visual from Figure 6. The quintile analysis uses 100 problems per bin, meaning the curves are based on relatively small sample sizes, and the paper does not provide confidence intervals within each bin.

Small-Scale Synthetic Supervision: Controlled Comparison (Section 4)

The controlled experiments in Section 4 use the large-scale PRM (PRMlarge) as a labeling oracle to train smaller reward models on identical datasets, varying only the supervision type. This isolates the effect of supervision format from confounding variables like dataset size and composition.

Figure 4a shows best-of-500 accuracy as a function of the number of labeled solutions per problem (from 1 to 200). The key comparisons:

Process supervision (PRM trained with PRMlarge step labels): the top-performing series across all data scales. At 200 samples/problem, it achieves approximately 56% best-of-500 accuracy (reading from the figure).
Outcome supervision from PRMlarge (ORM trained with PRMlarge solution-level labels): the middle series. At 200 samples/problem, it achieves approximately 51% — a gap of roughly 5 percentage points behind process supervision at the same data scale.
Outcome supervision from final-answer checking (standard ORM): the bottom series. At 200 samples/problem, it achieves approximately 46% — roughly 10 percentage points behind process supervision and 5 points behind PRMlarge-supervised outcome supervision.

The slopes of the best-fit lines are the most informative quantity. Process supervision shows the steepest improvement with additional data. Outcome supervision from PRMlarge shows a shallower slope but still consistently improves. Outcome supervision from final-answer checking shows the shallowest slope, suggesting that false positives from automatic grading fundamentally limit how much ORMs can improve with more data — adding more training examples doesn't help if many of them are mislabeled.

At the smallest data scale (1 sample per problem), all three methods perform similarly (around 28-30%), consistent with the Uesato et al. (2022) finding that process and outcome supervision are comparable in low-data regimes. The divergence only becomes apparent as data increases, confirming the paper's reframing of the ORM-vs-PRM comparison as a scaling phenomenon.

Figure 4b shows best-of-N performance for the best model from each series (trained on 200 samples/problem) across varying test-time compute budgets (N from 1 to 1000). The key findings:

The PRM (PRMlarge supervised) consistently outperforms both ORM variants across all values of N.
The ORM (PRMlarge supervised) consistently outperforms the ORM (final-answer supervised) — the gap between these two ORM variants quantifies the damage caused by false positives from automatic grading. At high N, this gap is roughly 5-8 percentage points.
The PRM's advantage over the best ORM grows with N, from roughly 2 percentage points at N = 1 to roughly 8-10 percentage points at N = 1000 (estimated from Figure 4b).

This comparison is methodologically important because it directly tests the paper's central causal claim: that the form of supervision (process vs. outcome), not just the quality of the supervision source, drives the PRM's advantage. By holding the supervision source constant (PRMlarge for both the PRM and the first ORM variant) and varying only whether the feedback is step-level or solution-level, the experiment isolates the effect of interest. The fact that process supervision still substantially outperforms outcome supervision even when both come from the same oracle is strong evidence for the paper's thesis.

The comparison between the two ORM variants (PRMlarge-supervised vs. final-answer-supervised) isolates a secondary effect: the impact of false positives from automatic grading. The PRMlarge-supervised ORM, which correctly labels spurious solutions as incorrect (since PRMlarge can detect incorrect intermediate steps even when the final answer matches), substantially outperforms the standard ORM. This demonstrates that part — but not all — of the PRM's advantage comes from avoiding the false positive problem. Process supervision provides additional benefits beyond cleaner outcome labels.

Active Learning Efficiency (Section 4.2)

Figure 4a includes a fourth series: PRM + Active Learning, where the training data is selected using the PRMselector model with the 80/20 strategy (80% most convincing wrong-answer solutions, 20% most convincing remaining solutions). The key quantitative finding:

"By comparing the slopes of the line of best fit with and without active learning, we estimate that this form of active learning is approximately 2.6× more data efficient than uniform data labelling." (Section 4.2)

This means that to achieve a given level of PRM performance, active learning requires roughly 2.6 times fewer human-labeled solutions than uniform sampling. At 200 samples per problem, the active learning series and the uniform process supervision series converge to similar performance, but the active learning series reaches any given performance level with fewer labeled examples.

The paper notes one qualification: "the model trained on the largest active learning dataset (200 samples per problem) appears to slightly underperform the expected trend line" (Section 4.2). The proposed explanation is that 200 samples represents a significant fraction of the overall selection pool (1000 samples per problem), and "this relative lack of diversity limits the possible upside from active learning." In other words, when the active selection draws too heavily from a finite pool, it becomes harder to find genuinely diverse "convincing wrong-answer" solutions, and the marginal value of each additional selected sample decreases.

The active learning experiment uses the synthetic supervision framework — PRMlarge provides the labels, and PRMselector is a small model trained on only a single sample per problem. This means the 2.6× figure is derived from synthetic labels, not human labels, and may not perfectly transfer to the human labeling setting. However, the large-scale data collection (Phase 2 of PRM800K) used a similar active learning philosophy (convincing wrong-answer selection with iterative retraining of the PRM), suggesting the principle transfers.

Out-of-Distribution Generalization (Section 5)

Table 1 reports best-of-100 accuracy on 224 STEM questions from recent AP and AMC exams — problems guaranteed to be uncontaminated since the exams were released after GPT-4's pretraining data was compiled:

Exam	ORM	PRM	Majority Vote	# Problems
AP Calculus	68.9%	86.7%	80.0%	45
AP Chemistry	68.9%	80.0%	71.7%	60
AP Physics	77.8%	86.7%	82.2%	45
AMC10/12	49.1%	53.2%	32.8%	84
Aggregate	63.8%	72.9%	61.3%	234

The PRM achieves the highest accuracy on every individual exam and on the aggregate. The aggregate gap over the ORM is 9.1 percentage points (72.9% vs. 63.8%), and over majority voting is 11.6 percentage points (72.9% vs. 61.3%). The pattern is qualitatively consistent with the MATH results: PRM > ORM > majority voting, with the PRM showing the largest advantage on the more difficult exam sets (AP Calculus, AMC10/12).

A notable pattern: on AP Physics, majority voting (82.2%) actually outperforms the ORM (77.8%), while the PRM (86.7%) beats both. This suggests that on certain problem distributions, the ORM can be worse than simple answer-frequency counting — it is actively misleading the selection process by ranking incorrect solutions above correct ones. The PRM avoids this failure mode.

The out-of-distribution results serve a dual purpose: they address contamination concerns (these problems are definitively unseen) and they demonstrate that the PRM's advantage generalizes beyond MATH-specific problem styles and difficulty distributions. However, the evaluation uses only 100 samples per problem and only 224 total problems, so the per-exam sample sizes (45-84 problems) are quite small — the exact percentages should be treated as approximate rather than precise.

Qualitative Evidence: Step-Level Score Visualizations (Appendix I)

The paper provides extensive qualitative evidence in Appendix I, showing cherry-picked examples of the PRM's step-level scoring on individual solutions. These visualizations (Figures 2 and the Appendix I examples) demonstrate:

True positives: the PRM correctly assigns high scores to all steps in genuinely correct solutions, even on problems with generator pass rates as low as 0.1% (Problem 1) or 1.7% (Problem 3) — showing it can recognize valid reasoning on problems where the generator almost never produces correct answers.
True negatives: the PRM correctly identifies specific incorrect steps, such as misapplication of algebraic identities (Problem 5: claiming an expression is a difference of squares when it isn't), arithmetic errors (Problem 7: a simple calculation mistake in step 11), and logical gaps (Problem 6: an incorrect simplification attempt).
False positives: the PRM occasionally misses subtle errors, such as a counting mistake that appears superficially reasonable (Problem 9: undercounting by a factor of 2), an algebraic manipulation error where terms are incorrectly moved (Problem 10), and a sequence periodicity error (Problem 12: claiming a sequence repeats every 12 terms when it's actually every 10).

These qualitative examples are not systematic — they are explicitly cherry-picked to illustrate the PRM's behavior — but they provide interpretable evidence that the PRM's step-level predictions correspond to human-understandable judgments about reasoning quality. The false positive examples are particularly informative because they reveal the PRM's remaining failure modes: errors that are subtle enough to appear superficially reasonable, often involving counting, bookkeeping, or algebraic manipulation that requires tracking multiple constraints simultaneously.

Ablation Studies and Robustness Checks

PRM scoring strategy (Appendix F.2, Table 4). The paper evaluates four combinations of design choices: treating neutral labels as positive or negative, and using product or minimum reduction over step scores. With best-of-1860 evaluation:

Neutral treatment	Reduction	Accuracy
neutral = positive	product	78.2%
neutral = positive	minimum	77.6%
neutral = negative	minimum	77.8%
neutral = negative	product	77.4%

All four strategies perform within 0.8 percentage points of each other — the specific choice of scoring strategy is not a major driver of PRM performance. The product reduction with neutral-as-positive edges ahead slightly and is used throughout the paper. The authors note that the product reduction "does create a slight bias against solutions with a larger number of steps" (Appendix F.2), but this bias does not appear to meaningfully harm performance in practice.

Label count and training epochs (Appendix F.1). The paper reports that training PRMs for 2 epochs improves performance over 1 epoch on smaller datasets but that "additional epochs, up to some point, don't noticeably help or hurt performance" and "on larger datasets, the benefits of 2 epoch training diminishes, but we continue doing it for consistency." No quantitative ablation is provided for the epoch count — the finding is reported as an observational note rather than a controlled experiment.

ORM training on mixed data (Section 3). The paper explored training the large-scale ORM on a superset of PRM800K solutions, mixing in uniformly sampled solutions to balance the active learning bias toward wrong-answer solutions, but "found that this did not improve ORM performance." No quantitative results are reported for this negative result.

RM-weighted voting (Section 3). The paper experimented with combining PRM scores and answer frequency (following Li et al., 2022; Uesato et al., 2022) but "this did not noticeably improve performance." Again, no quantitative results are reported.

Iterative retraining of PRMselector (Section 4.2). The paper attempted to iteratively retrain the PRMselector model during active learning — after each generation of data collection, retrain PRMselector on all accumulated labels and use the improved model for the next generation's selection. This is reported as a negative result: "we observed instability in this process which we were unable to diagnose. The resulting reward models performed no better than the models described above." No quantitative details on the nature of the instability are provided.

MathMix pretraining (Section 2.2, Appendix A). The paper states that MathMix fine-tuning "improves the model's mathematical reasoning capabilities" and is similar to the approach in Lewkowycz et al. (2022), but no ablation comparing models with and without MathMix pretraining is reported. The contribution of MathMix to the final results is therefore unquantified.

ORM training hyperparameters (Appendix E). The paper notes that ORM performance "is not sensitive to most other hyperparameters, within a reasonable range," suggesting that the specific choices (single epoch, no dropout, no joint language modeling objective) are not critical drivers of the ORM-vs-PRM comparison.

Critical Assessment

The experiments provide strong support for the paper's central claim that process supervision produces more reliable reward models than outcome supervision for mathematical reasoning, but they also reveal boundaries and limitations that temper the strength of specific sub-claims.

Claim: Process supervision significantly outperforms outcome supervision for training reward models.

This claim is well-supported by multiple independent lines of evidence: the large-scale comparison (Figure 3: 78.2% vs. 72.4% at N=1860), the controlled small-scale comparison on identical datasets (Figure 4a: process supervision consistently above both outcome supervision variants at all data scales), and the out-of-distribution evaluation (Table 1: 72.9% vs. 63.8% aggregate). The effect is not merely a large-scale artifact — it appears in the controlled synthetic supervision experiments where training data is held identical and only supervision type varies. The claim holds across multiple data scales, test-time compute budgets, and problem distributions.

However, the effect size is not enormous in absolute terms. The 5.8 percentage point gap at N=1860, while clearly significant, represents the PRM solving roughly 29 additional problems out of 500 — meaningful but not transformative. On a per-problem basis, the PRM and ORM agree on the vast majority of solutions; the PRM's advantage comes from a minority of cases where the ORM is confidently wrong. This is consistent with the paper's own framing: the PRM helps most on hard problems and on convincing wrong-answer solutions — precisely the cases where the ORM's credit assignment problem is most severe. The practical implication is that process supervision's value is concentrated in the hardest cases, which may or may not matter depending on the application's error tolerance.

Claim: The performance gap widens as N increases (Figure 3).

This is visually clear in Figure 3 and is one of the paper's most robust findings. The widening gap is the signature of a better reward model — one that continues to extract value from additional samples when a weaker model has saturated. However, the paper's N=1860 evaluation point is somewhat arbitrary — there's no evidence that the gap would continue widening beyond N=1860, or that either model would asymptote to some fundamental limit. The experiments stop at the largest N that was computationally feasible for generating and scoring solutions, not at a theoretically motivated saturation point. It's possible that the ORM would eventually catch up if N were pushed much higher, or conversely that the PRM's advantage would grow even larger. The paper cannot distinguish these possibilities.

Claim: Active learning yields a 2.6× improvement in the data efficiency of process supervision (Section 4.2).

This claim is supported but with significant caveats. The 2.6× figure comes from a specific experimental setup: small-scale models, synthetic supervision from PRMlarge, a particular PRMselector trained on a single sample per problem, a specific 80/20 selection ratio, and a 1000-sample selection pool. It is unclear how sensitive the 2.6× factor is to these choices — would a different PRMselector architecture, a different selection ratio, or a different pool size produce a different efficiency multiplier? More importantly, the 2.6× figure is derived from synthetic labels, and it's not obvious that the same efficiency gain would be realized with human labelers. Human labelers have different error patterns and calibration than PRMlarge, and the "convincing wrong-answer" selection criterion may surface different kinds of solutions when the scoring model is a human-calibrated PRM versus a synthetic oracle.

Additionally, the paper notes that the active learning model at 200 samples/problem "appears to slightly underperform the expected trend line" and that the 2.6× estimate comes from comparing slopes of best-fit lines — a linear approximation to what may be a nonlinear relationship. If the true data scaling curve is sigmoidal (diminishing returns at high data volumes), the linear slope comparison overestimates the efficiency gain at realistic data scales. The paper does not report statistical uncertainty on the 2.6× estimate, making it difficult to assess whether this is a precise measurement or a rough order-of-magnitude indication.

Claim: Process supervision incurs a negative alignment tax — it improves both safety and capability (Section 6.2).

This claim is conceptually important but empirically underdeveloped in the experimental sections. The capability improvement is well-demonstrated (Figures 3, 4, Table 1), but the "safety" and "alignment" benefits are asserted rather than measured. The paper argues that process supervision "directly rewards models for following a human-endorsed chain-of-thought" and is "more likely to produce interpretable reasoning," but no experiment quantifies the interpretability or alignment of PRM-selected solutions versus ORM-selected solutions. Do PRM-selected solutions actually contain fewer logical errors in their intermediate steps (not just correct final answers)? The paper provides qualitative examples in Appendix I showing the PRM correctly identifying errors, but no systematic measurement of step-level correctness in the selected solutions.

To properly establish the negative alignment tax claim, the paper would need to measure something like: the fraction of PRM-selected solutions that are fully correct in their reasoning (not just correct final answer) versus the same fraction for ORM-selected solutions, or the human-judged interpretability of the selected solutions. Without such measurements, the alignment argument remains plausible and well-motivated but empirically unvalidated — it's a reasonable interpretation of why process supervision works rather than a demonstrated fact about its effects.

Weakness: The ORM and PRM training sets are not comparable at large scale, and the small-scale synthetic experiments partially but incompletely address this.

The paper is admirably transparent about the training set non-comparability at large scale (Section 3): the ORM is trained on 100 uniform samples per problem, while the PRM is trained on approximately 6.25 actively-selected samples per problem from PRM800K. This confounds supervision type with data quantity, data selection strategy, and data distribution. The small-scale synthetic experiments (Section 4) address this by holding the training data identical and varying only the supervision type, but these experiments use synthetic labels from PRMlarge — not human labels — and use small-scale models, not GPT-4. The synthetic supervision framework therefore demonstrates that process supervision can outperform outcome supervision on identical data with a strong oracle, but it does not directly demonstrate that the large-scale PRM's advantage in Figure 3 is attributable to supervision type rather than to the PRM800K data selection strategy or other confounds.

A more definitive experiment would have collected both human process labels and human outcome labels on an identical set of solutions at large scale, enabling a direct comparison without the synthetic oracle intermediary. The paper acknowledges that this is prohibitively expensive, but the gap remains: the strongest causal evidence (Figure 4) comes from a different regime (small models, synthetic labels) than the headline result (Figure 3, large models, human labels).

Weakness: Single base model family and single dataset.

All experiments use GPT-4 or GPT-4-like models fine-tuned on MathMix, and all primary evaluation is on the MATH dataset. While the out-of-distribution evaluation on AP/AMC exams (Section 5) partially addresses the dataset limitation, it uses the same underlying model family. The paper cannot distinguish whether the PRM's advantage is specific to GPT-4's reasoning patterns or generalizes to other model architectures (e.g., Claude, Gemini, LLaMA). If GPT-4 produces errors that are particularly amenable to step-level detection — for instance, errors that are localized to specific steps rather than distributed across multiple steps — the PRM advantage may not transfer to generators with different error characteristics. Similarly, if the MATH dataset's problems have a particular structure (symbolic manipulation, algebraic reasoning) that benefits from step-level supervision, the advantage may not extend to other reasoning domains (code generation, legal reasoning, scientific explanation).

Weakness: The test set is relatively small (500 problems for MATH, 224 for OOD), and per-quintile analyses use even smaller bins (100 problems each).

With 500 total test problems, a 5.8 percentage point gap represents roughly 29 problems — a substantively meaningful but statistically modest difference. The difficulty quintile analysis (Appendix G, Figure 6) splits these 500 problems into bins of 100, meaning each curve is estimated from 100 data points. The paper does not report confidence intervals for the per-quintile results, making it impossible to assess whether the observed patterns (e.g., the ORM decreasing on quintile 1) are statistically reliable or could arise from sampling noise. The OOD evaluation (Table 1) has even smaller per-exam sample sizes (45-84 problems). This is a genuine limitation — the paper's results are almost certainly directionally correct given the consistency across multiple evaluation settings, but the exact magnitudes should be treated as approximate.

Weakness: Majority voting is the only non-learned baseline; other simple baselines are missing.

The paper compares the PRM against the ORM and majority voting, but several other simple baselines could have been informative. For instance, a length-based baseline (prefer shorter solutions, which are less likely to contain errors) or a confidence-based baseline (select solutions where the generator's own token probabilities indicate high confidence) would have provided additional reference points. The absence of these baselines is not a major flaw — majority voting is the strongest and most commonly used non-learned baseline in the literature — but their inclusion would have strengthened the argument that the PRM's advantage comes from genuine reasoning evaluation rather than exploiting simple heuristics that happen to correlate with correctness.

Missing experiment: Direct measurement of step-level correctness in selected solutions.

The paper's evaluation metric (best-of-N final-answer accuracy) measures whether the selected solution reaches the correct answer, not whether it does so through correct reasoning. A solution could earn a high PRM score for having plausibly correct steps while still reaching the right answer through a subtle error — the PRM's false positives in Appendix I demonstrate this possibility. The paper would be strengthened by a systematic evaluation of the reasoning quality of PRM-selected versus ORM-selected solutions: for a random sample of test problems, have human evaluators (or PRMlarge) judge whether the selected solutions' intermediate steps are actually correct, not just whether the final answer matches. This would directly test the alignment claim in Section 6.2 and would reveal whether the PRM's higher final-answer accuracy translates to genuinely more reliable reasoning.

Missing experiment: Sensitivity to the "stop at first error" protocol.

The paper's deliberate choice to supervise only up to the first error (Section 2.6) is well-motivated for fair comparison with outcome supervision, but no experiment tests how much additional performance would be gained by providing full process supervision (labeling all steps, including those after the first error). On hard problems where solutions contain multiple errors, full process supervision would provide substantially more information per solution. The absence of this ablation is understandable — it would require a separate data collection effort with a different labeling protocol — but it means the paper's PRM represents a lower bound on what process supervision could achieve if the "stop at first error" constraint were lifted.

Missing experiment: Correlation between PRM scores and human judgments of reasoning quality.

The paper provides qualitative examples of PRM scores aligning with human judgments (Figure 2, Appendix I), but no systematic measurement of inter-annotator agreement between the PRM and human labelers on held-out step-level judgments. Such a measurement would quantify how well the PRM has actually learned to replicate human judgments of step correctness, as opposed to learning correlated features that happen to predict final-answer correctness. The synthetic supervision framework could have been used for this purpose: evaluate PRMlarge's step-level accuracy against the human labels in a held-out portion of PRM800K, and report precision/recall for detecting incorrect steps.

Summary assessment. The experiments strongly support the paper's primary empirical claim — that a PRM trained with process supervision can achieve higher best-of-N accuracy than an ORM or majority voting — through converging evidence from large-scale human-labeled data, small-scale controlled synthetic experiments, and out-of-distribution evaluation. The claim that active learning improves data efficiency by approximately 2.6× is supported but should be treated as an estimate with unclear error bounds and uncertain transferability to human labeling settings. The alignment-related claims (negative alignment tax, improved interpretability) are conceptually well-motivated but not directly tested experimentally — they remain plausible interpretations rather than demonstrated facts. The main weaknesses are the modest absolute effect size, the inability to fully separate supervision type from data selection strategy at large scale, the single model family evaluation, and the relatively small test set for fine-grained difficulty analysis. These weaknesses do not undermine the paper's conclusions — the consistency of the PRM advantage across multiple settings makes the central finding robust — but they do bound the precision and generality with which those conclusions can be stated.

6. Limitations and Trade-offs

The Generator Is Fixed — All Gains Come From Better Selection, Not Better Generation

The assumption or constraint. The paper explicitly scopes its investigation to reward model training only, deliberately excluding any attempt to improve the generator through reinforcement learning or other techniques. Section 2.1 states:

"We do not attempt to improve the generator with reinforcement learning (RL). When we discuss outcome and process supervision, we are specifically referring to the supervision given to the reward model. We do not discuss any supervision the generator would receive from the reward model if trained with RL."

The generator is fine-tuned solely for formatting (producing newline-delimited step-by-step solutions from filtered correct-only examples) and is then frozen — all subsequent experiments use a single fixed generator to produce candidate solutions at temperature 1.0 (Section 2.3, Appendix E).

The consequence. The 78.2% best-of-1860 accuracy is fundamentally bounded by two factors: (1) the generator must produce at least one correct solution among the N candidates for best-of-N to succeed, and (2) the reward model must correctly identify it. If the generator's pass rate on a problem is 0% (i.e., it never produces a correct solution in 1860 attempts), no reward model — however perfect — can solve that problem through best-of-N selection. This means the paper's framework offers no path for improving the generator's fundamental capability to produce correct reasoning in the first place. The hardest MATH problems, where the generator's pass rate is near zero, remain essentially unsolvable regardless of reward model quality.

This limitation is particularly consequential because the paper's own difficulty analysis (Appendix G, Figure 6) shows that on the hardest quintile of problems, both the PRM and ORM achieve very low accuracy, and the performance gap between them, while present, is small in absolute terms. The generator's capability ceiling — not the reward model's discriminative power — is the binding constraint on hard problems. A practitioner hoping to deploy this system on challenging reasoning tasks would need to answer a separate question: how to improve the generator's ability to produce correct solutions, a question the paper explicitly does not address.

What evidence exists in the paper. The difficulty quintile analysis in Appendix G (Figure 6) directly visualizes this limitation. On quintile 5 (hardest problems), both PRM and ORM curves sit at low accuracy regardless of N — the generator simply does not produce correct solutions for most of these problems. The paper does not report the generator's pass rate at the problem level, but the quintile binning is defined by this pass rate, meaning quintile 5 problems have the lowest probability of containing any correct solution in the candidate pool. The out-of-distribution results in Table 1 reinforce this: on AMC10/12 problems (the hardest exam set), even the PRM achieves only 53.2% accuracy — nearly half of the problems remain unsolved.

Mitigation status. The paper acknowledges this scoping choice explicitly and frames it as a deliberate experimental design decision, not an oversight: "Although finetuning the generator with RL is a natural next step, it is intentionally not the focus of this work" (Section 2.1). The paper suggests RL fine-tuning of the generator using the trained PRM as a reward signal as the obvious next step but performs no experiments in this direction. The limitation is therefore acknowledged but not addressed — the paper provides a better reward model but does not demonstrate that this better reward model can be used to improve the generator through RL, which is the primary use case for reward models in the RLHF pipeline the paper references (Ziegler et al., 2019; Ouyang et al., 2022). A practitioner deploying this method would need to separately validate that the PRM's step-level scores provide a suitable training signal for generator improvement — the paper offers no evidence either way.

The 78.2% Headline Number Depends on an Impractically Large Test-Time Compute Budget

The assumption or constraint. The paper's headline result — 78.2% accuracy on the MATH test set — is achieved at N = 1860 (best-of-1860 selection), meaning the system generates and scores 1860 complete solutions per test problem before selecting one. At this scale, the test-time compute cost per problem is approximately 1860 forward passes of GPT-4 — an enormous computational expenditure that would be prohibitively expensive and slow for most practical deployments. The paper does not report the wall-clock time, FLOP count, or dollar cost of this evaluation protocol.

The consequence. The 78.2% figure is best understood as an upper bound on what best-of-N selection with a perfect generator and an ideal reward model could achieve, not as a practical deployment number. In any real application with latency or cost constraints, the usable N would be far smaller — perhaps N = 4, 8, or 16 — and the PRM's advantage at those budgets, while still present, is substantially smaller in absolute terms. Figure 3 shows that at N = 10, the PRM achieves roughly 64-66% accuracy (reading from the figure), compared to roughly 62-64% for the ORM — a gap of perhaps 2-4 percentage points rather than the 5.8-point gap at N = 1860. The widening gap with N is intellectually interesting (it demonstrates the PRM's superior scaling properties) but the headline number is misleading if interpreted as a practical system's expected performance.

Furthermore, best-of-1860 evaluation means generating 1860 solutions, scoring all 1860 with the reward model, and selecting the top one — a factor of 1860 increase in inference cost over greedy decoding. For GPT-4, which is already expensive per query, this is a very high cost multiplier. The paper provides no cost-effectiveness analysis comparing best-of-N selection at various N to other ways of spending the same compute budget (e.g., using a larger generator with lower N, or fine-tuning the generator on additional data).

What evidence exists in the paper. Figure 3 directly shows the N-dependence of performance. At N = 1 (equivalent to a single sample), the PRM and ORM perform similarly and both achieve modest accuracy. The gap grows with N, but the majority of the absolute accuracy gain comes from increasing N from 1 to roughly 100-200, with diminishing returns thereafter. The paper does not provide a cost-normalized comparison — for instance, what accuracy would the ORM achieve at N = 1860 with a generator that costs the same as the PRM's N = 100? — which would be necessary to determine whether the PRM is cost-effective at realistic budgets.

Mitigation status. The paper does not address this limitation directly. There is no discussion of latency, cost, or practical deployment budgets. The choice of N = 1860 appears to be driven by computational feasibility (generate a large pool once and subsample from it) rather than by a principled argument about what N is practically relevant. The small-scale synthetic experiments (Section 4) evaluate at best-of-500 (Figure 4a) and best-of-N for N up to 1000 (Figure 4b), which are still large budgets. The out-of-distribution evaluation (Table 1) uses N = 100, which is more realistic but still substantial. The paper does not report results at small N values (e.g., N = 4, 8, 16) that would be most relevant for latency-sensitive applications. A practitioner would need to extrapolate from Figure 3 to estimate performance at their budget constraint, with no guidance from the paper on what N provides the best cost-effectiveness tradeoff.

Human Data Collection Cost Is Not Quantified, and the PRM800K Release Does Not Eliminate the Cost Barrier for New Domains

The assumption or constraint. The paper's central practical contribution — the PRM800K dataset — represents a massive investment in human annotation that is not quantified in cost, time, or annotator-hours. The paper reports that PRM800K contains "800K step-level labels across 75K solutions to 12K problems" (Section 2.4), collected through two phases with iterative retraining and continuous quality control (Appendix B), but provides no estimate of the total cost to produce this dataset. For a practitioner considering whether to replicate this approach in a new domain (e.g., code generation, scientific reasoning, legal analysis), the absence of cost information makes it impossible to budget for a comparable effort.

The release of PRM800K is framed as removing a "significant barrier to entry" (Section 8), but this only removes the barrier for research that can use the existing MATH-focused dataset. Any application to a new domain would require collecting a new dataset of comparable scale and quality, with unknown cost. The paper does not provide the tools (labeling interface, annotator training materials, quality control protocols) that would reduce this cost, nor does it estimate how much cheaper the 2.6× active learning improvement makes the data collection in practice.

The consequence. Without cost quantification, the paper's central economic argument — that process supervision is worth its higher per-label cost because it produces better reward models — remains qualitative rather than quantitative. A practitioner deciding between process and outcome supervision needs to know: if I have a budget of X dollars for human annotation, should I spend it on process labels or outcome labels? The paper's scaling curves (Figure 4a) show that process supervision outperforms outcome supervision at a given number of labeled solutions, but process labels are substantially more expensive per solution than outcome labels (which can often be automated). The 2.6× active learning improvement partially addresses this — it means process supervision data can be collected more efficiently — but without knowing the baseline cost, the efficiency multiplier is not actionable.

Furthermore, the PRM800K dataset was collected using a specific quality control infrastructure: labeler screening tests (30 quality control questions with 75% agreement required), continuous quality monitoring (10-20 QC problems per generation), iterative retraining of the PRM to select the most informative solutions, and a two-phase collection process where the methodology was refined based on early experience (Appendix B). Replicating this infrastructure in a new domain would require significant upfront investment in developing QC materials, training labelers, and iterating on the collection protocol — costs that the paper does not acknowledge or estimate.

What evidence exists in the paper. The paper provides no cost data. The size of the dataset is reported (800K labels, 75K solutions, 12K problems), and the active learning efficiency gain is quantified (2.6×), but the absolute cost of human annotation is never mentioned. The paper does not report the number of annotators employed, the time required for data collection, the hourly rate or per-label cost, or the cost of the quality control infrastructure. The phased data collection description (Appendix B) mentions that Phase 1 was "cumbersome" and that labelers "spent a lot of time supervising long uninteresting solutions," suggesting that early inefficiencies were significant, but no estimate of wasted effort is provided.

Mitigation status. The paper does not address this limitation. The release of PRM800K is presented as the primary mitigation — other researchers can use the existing dataset rather than collecting their own — but this only helps for MATH-domain research. The paper does not discuss the cost of extending the approach to new domains, does not provide cost estimates or budgeting guidance, and does not release the labeling infrastructure (interface, training materials, QC rubrics) that would reduce replication costs. A practitioner seeking to apply process supervision to their own domain would need to independently solve the data collection cost estimation problem with no guidance from the paper.

Single Domain, Single Model Family — The PRM Advantage Has Not Been Demonstrated to Generalize

The assumption or constraint. All experiments in the paper use a single base model family (GPT-4 and its smaller variants) and a single task domain (competition-level mathematics, primarily the MATH dataset with a small out-of-distribution check on AP/AMC exams). The paper does not evaluate whether the PRM's advantage over the ORM transfers to other model architectures (e.g., models with substantially different pretraining data, scale, or reasoning patterns), other reasoning domains (e.g., code generation, multi-step planning, scientific argumentation), or other languages. The out-of-distribution evaluation in Section 5 uses the same base model (GPT-4) on different math problems — it tests generalization across problem sources, not across domains or model families.

The consequence. The paper's central claim — "process supervision can be used to train much more reliable reward models than outcome supervision in the domain of mathematical reasoning" (Section 8) — is accurately scoped to mathematical reasoning, but the broader implications for AI alignment that the paper discusses in Section 6.2 depend on the assumption that the finding generalizes. If the PRM's advantage is specific to mathematical reasoning — where correctness is objectively defined, errors are typically localized to specific steps, and human labelers can be reliably trained to identify mistakes — then the alignment benefits the paper claims (negative alignment tax, interpretable chain-of-thought, resistance to reward hacking) may not materialize in domains where correctness is more subjective, errors are more diffuse, or human labelers disagree more frequently.

Several features of mathematical reasoning make it particularly amenable to process supervision: (1) there is a ground-truth answer that enables automatic verification of final correctness, which supports the active learning strategy (convincing wrong-answer solutions require knowing which answers are wrong); (2) reasoning steps are discrete and well-defined, making the "step" unit natural for both generators and labelers; (3) labeler agreement on step correctness is likely high for well-defined mathematical operations. In domains like creative writing, open-ended dialogue, or complex strategic planning, none of these properties hold — the concept of a "step" is ill-defined, ground-truth correctness is absent, and human judgments of quality are subjective and variable. The paper provides no evidence that process supervision would outperform outcome supervision in such domains, and the mechanisms that drive the advantage in math (easier credit assignment, resistance to false positives) may be less relevant or absent.

What evidence exists in the paper. The paper's only generalization evidence is the out-of-distribution evaluation in Section 5 (Table 1), which tests the same GPT-4-based PRM and ORM on 224 STEM exam problems. This is a valuable but narrow generalization test — it shows that the PRM's advantage persists when the problem source changes (from MATH to recent AP/AMC exams), but not when the domain changes (still mathematics) or the model family changes (still GPT-4). The paper does not report results with any other base model, does not test on any non-math reasoning benchmark, and does not evaluate in domains without objective correctness criteria. The paper acknowledges the domain limitation explicitly in Section 6.2: "It is unknown how broadly these results will generalize beyond the domain of math, and we consider it important for future work to explore the impact of process supervision in other domains."

Mitigation status. The paper is transparent about the domain limitation (Section 6.2) and frames it as a direction for future work, but performs no experiments to address it. The release of PRM800K partially mitigates the model-family limitation — other researchers can train PRMs on this dataset using different base models and evaluate whether the advantage transfers — but it does nothing to address the domain limitation, since PRM800K is inherently math-specific. A practitioner considering process supervision for a non-math domain would need to extrapolate from the paper's math results with no empirical evidence that the extrapolation is valid. The paper's alignment claims (Section 6.2) depend heavily on this extrapolation being correct, making the domain limitation particularly consequential for the paper's broader argument.

The "Stop at First Error" Protocol Both Enables and Constrains the Comparison

The assumption or constraint. The paper makes a deliberate methodological choice: process supervision is provided only up to the first incorrect step. Section 2.6 states:

"When we provide process supervision, we deliberately choose to supervise only up to the first incorrect step. This makes the comparison between outcome and process supervision more straightforward. For correct solutions, both methods provide the same information, namely that every step is correct. For incorrect solutions, both methods reveal the existence of at least one mistake, and process supervision additionally reveals the precise location of that mistake. If we were to provide additional process supervision beyond the first mistake, then process supervision would have an even greater information advantage."

The motivation is clear: by limiting process supervision to the first error, the paper creates a conservative comparison where process supervision's information advantage is minimized — it reveals which step went wrong but not what happens after.

The consequence. This constraint means the paper's PRM represents a lower bound on what process supervision could achieve if used without the fairness constraint. In practice, if you are collecting human process supervision data and paying annotators to read solutions, there is no reason to stop at the first error — you might as well have them label all steps, including those after an error, to maximize the information extracted per labeled solution. The paper's results therefore understate the potential advantage of process supervision in practical deployments where the "fair comparison" constraint doesn't apply. The gap between the PRM and ORM in Figure 3 would likely be larger if the PRM had been trained with full process supervision on PRM800K rather than first-error-only supervision.

However, this constraint also creates an interpretive problem: how much of the PRM's observed advantage comes from error localization versus from the richer per-step training signal? The paper argues in Section 6.1 that error localization is the key mechanism: "Process supervision makes credit assignment easier, and we believe that this explains its strong performance." But the "stop at first error" protocol means the PRM also receives more positive step labels per incorrect solution than the ORM receives — an incorrect solution with 10 steps and an error at step 7 provides the PRM with 6 positive step labels and 1 negative label, while the ORM receives only a single "incorrect" label. The paper cannot distinguish whether the PRM's advantage comes from knowing which step is wrong (the localization benefit) or simply from seeing more training signal per solution (the data amplification benefit). Both mechanisms are present, and the experimental design does not isolate them.

What evidence exists in the paper. No experiment in the paper varies the depth of process supervision. The paper does not compare a "first error only" PRM against a "full supervision" PRM, nor does it ablate the number of steps supervised per solution. The synthetic supervision experiments (Section 4) use the same first-error-only protocol as the human-labeled data, so they don't address this question either. The paper's claim about credit assignment being the key mechanism (Section 6.1) is therefore an interpretation, not an experimentally validated finding.

Mitigation status. The paper does not acknowledge this as a limitation or suggest experiments to isolate the mechanism. The choice is presented as a methodological strength (enabling fair comparison) rather than as a constraint that limits mechanistic understanding. A practitioner who wants to maximize PRM performance in a deployed system would likely ignore the first-error-only constraint and collect full process supervision, but the paper provides no evidence about how much additional performance this would yield. The question is left entirely to future work.

Neutral Labels Defer an Important Design Decision to Test Time Without Systematic Analysis of the Tradeoff

The assumption or constraint. The human labeling protocol allows annotators to assign a neutral label to steps that are ambiguous — "subtly misleading" or "a poor suggestion that is technically still valid" (Section 2.4). The paper treats this as a deferral mechanism: "We permit neutral labels since this allows us to defer the decision about how to handle ambiguity: at test time, we can treat neutral labels as either positive or negative" (Section 2.4). In practice, the paper finds that treating neutral as positive yields slightly better best-of-1860 accuracy (78.2% vs. 77.4-77.8% for the other three combinations, Table 4 in Appendix F.2) and uses this setting throughout.

The consequence. The neutral category introduces a latent ambiguity into process supervision that has no counterpart in outcome supervision. An outcome label — "correct" or "incorrect" — is unambiguous in its implication (even if sometimes wrong due to false positives). A process label of "neutral" means the human annotator could not decide whether the step was correct or incorrect, and the paper's solution — treat it as positive — means the PRM is trained to treat ambiguous steps as correct. This has unknown effects on the PRM's behavior: does it learn to be overconfident about steps that are actually ambiguous? Does it fail to flag genuinely misleading reasoning because that reasoning is "technically valid"? The paper's scoring strategy ablation (Table 4) shows that the choice affects performance by less than one percentage point, but this is measured only by final-answer accuracy — it does not measure whether neutral-as-positive causes the PRM to endorse solutions with misleading intermediate steps more often than neutral-as-negative would.

The deeper issue is that the neutral category is a compressed representation of annotator uncertainty that discards information about why a step was ambiguous. Two steps labeled neutral might be ambiguous for entirely different reasons — one might be mathematically valid but poorly explained, another might be on the right track but missing crucial justification. By collapsing both into "neutral" and then treating neutral as positive, the PRM loses the ability to distinguish these cases. In a deployed system where the PRM is used to score solutions for RL training, this could lead to a generator being rewarded for producing reasoning that is "technically not wrong" but substantively misleading — a failure mode the paper's alignment argument (Section 6.2) specifically claims process supervision should prevent.

What evidence exists in the paper. The only quantitative evidence on neutral label handling is Table 4 (Appendix F.2), which shows a 0.4-0.8 percentage point difference between the four scoring strategies. This is a final-answer accuracy measurement — it does not measure the effect of neutral label treatment on step-level prediction quality, on the PRM's calibration, or on the types of errors the PRM makes. The paper does not report the frequency of neutral labels in PRM800K, does not analyze what kinds of steps receive neutral labels, and does not measure inter-annotator agreement on neutral vs. positive/negative distinctions. The qualitative examples in Appendix I do not show examples of neutral-labeled steps, so the reader cannot assess what "neutral" means in practice.

Mitigation status. The paper does not address this as a limitation. The neutral category is presented as a practical concession to ambiguous cases, and the test-time handling choice is justified by the small accuracy difference in Table 4. But the deeper question — whether the neutral category represents meaningful information that is lost by dichotomizing at test time — is not explored. A practitioner replicating this approach would need to decide whether to include a neutral category, how to train annotators to use it consistently, and how to handle it during training and evaluation — with essentially no guidance from the paper beyond the Table 4 ablation, which suggests the choice doesn't matter much for final-answer accuracy but provides no evidence about other potential effects.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reshapes the conversation about supervision methodology for reasoning models by providing the first large-scale, rigorous evidence that process supervision does not merely match outcome supervision — it substantially outperforms it when scaled, and the performance gap widens with test-time compute. Prior to this work, the field's dominant assumption, shaped by Uesato et al. (2022), was that process and outcome supervision produced roughly equivalent reward models, with process supervision offering alignment benefits but no clear capability advantage. That assumption guided research investment: why pay for expensive human step-level labels when cheap automatic outcome labels work just as well?

This paper overturns that assumption by reframing the comparison as a data scaling phenomenon rather than a fixed tradeoff. Figure 4a is the pivotal evidence: at small data volumes (1-10 labeled solutions per problem), process and outcome supervision perform similarly, exactly as Uesato et al. observed. But as the data budget grows, the curves diverge — process supervision's richer per-sample signal yields a steeper improvement trajectory that outcome supervision cannot match, even when the outcome labels come from an oracle PRM that eliminates false positives (the PRMlarge-supervised ORM in Figure 4a remains below the PRM curve). This reframing means the field can no longer ask "is process supervision better?" as a binary question — it must ask "at what data scale does process supervision's advantage become worth its cost?" The answer depends on the slope of the scaling curve, the active learning efficiency multiplier (2.6×), and the relative cost of human process labels versus automated outcome labels — all quantities this paper provides or enables others to measure.

The paper is not a paradigm shift in methods — both process and outcome supervision were established techniques, and the PRM architecture (language model fine-tuned to predict label tokens at step boundaries) is straightforward rather than novel. The shift is epistemological: the paper provides a template for how to rigorously compare supervision methods that differ in both cost and informativeness. The two-regime experimental design (large-scale for state-of-the-art, small-scale synthetic for controlled ablations) is a methodological contribution that other researchers can adopt when human annotation costs make systematic comparison prohibitive. The synthetic supervision framework — using a large, carefully-trained model as a labeling oracle for smaller models — demonstrates that the crucial experiments (varying only supervision type on identical data, measuring scaling curves, testing active learning) need not be bottlenecked by human annotation budgets. This is a practical breakthrough for empirical reward model research.

The paper also reconciles conflicting prior results in a way that advances the field beyond he-said-she-said comparisons. The finding that process supervision outperforms outcome supervision at scale does not contradict Uesato et al. (2022) — it contextualizes their result as a measurement at a particular point on a scaling curve where both methods happen to intersect. This is intellectually satisfying because it explains why reasonable researchers reached different conclusions: they were operating at different implicit data scales. The field now has a framework for understanding why supervision method comparisons have produced inconsistent results and how to design experiments that will produce consistent ones (report scaling curves, not single-point comparisons).

The identification of active learning as a force multiplier for human data (2.6× efficiency gain, Section 4.2) changes the economic calculus around process supervision. The primary objection to process supervision has always been cost — human step-level labels are expensive, and if they don't improve reward model quality, they're not worth it. The active learning result shows that the labeling budget can be spent much more intelligently than uniform sampling, substantially narrowing the cost gap between process and outcome supervision. Combined with the finding that process supervision's advantage grows with data scale, the implication is that process supervision becomes progressively more cost-effective as the labeling budget increases — the exact opposite of the intuitive expectation that expensive labels would show diminishing returns. The "convincing wrong-answer" selection criterion (target solutions that score highly under the current PRM but reach incorrect answers) is a specific, implementable strategy that any research group can adopt immediately.

The paper makes outcome supervision a less attractive default in several ways. First, it demonstrates that outcome supervision suffers from a fundamental weakness — false positives from automatic grading — that process supervision avoids. The PRMlarge-supervised ORM in Figure 4a consistently outperforms the final-answer-supervised ORM, quantifying the damage: roughly 5 percentage points of best-of-500 accuracy are lost to mislabeled spurious solutions. This means that even if you only care about final-answer accuracy (not alignment or interpretability), outcome supervision with automatic labels is leaving performance on the table. Second, the paper shows that majority voting — which requires no reward model at all — is surprisingly competitive with outcome-supervised reward models (Figure 3, Table 1), and on some problem distributions (AP Physics in Table 1), majority voting actually outperforms the ORM. If a simple non-learned baseline can match or beat your learned reward model, the case for that reward model's existence is weak. The PRM, by contrast, consistently beats majority voting by substantial margins (8.6 points at N=1860 in Figure 3, 11.6 points on the OOD aggregate in Table 1), justifying its additional complexity.

The alignment framing — process supervision as a negative alignment tax — shifts the conversation around safety-capability tradeoffs. The standard narrative in AI alignment is that safety measures impose capability costs (the alignment tax), creating an incentive to skimp on safety. The paper's finding that process supervision simultaneously improves both safety-relevant properties (interpretable chain-of-thought, resistance to reward hacking) and raw capability (higher best-of-N accuracy) provides a rare existence proof that the tradeoff is not inevitable. This matters for the broader alignment discourse: it demonstrates that research into better supervision methods can serve both safety and capability goals simultaneously, weakening the argument that safety research necessarily slows progress. The paper is appropriately cautious about generalizability (Section 6.2: "it is unknown how broadly these results will generalize beyond the domain of math"), but the conceptual contribution — naming and characterizing the negative alignment tax phenomenon — provides a target for other researchers to aim for in other domains.

However, the paper also reveals hard boundaries that should temper expectations. The generator remains fixed throughout — all gains come from better selection, not better generation — meaning that problems the generator cannot solve at all (the hardest MATH problems in quintile 5 of Appendix G, Figure 6) remain unsolvable regardless of reward model quality. The paper provides a better reward model but does not demonstrate that this reward model can be used to improve the generator through RL (the obvious next step and the primary use case for reward models in RLHF pipelines). The 78.2% headline number requires generating and scoring 1860 solutions per problem — an enormous test-time compute expenditure that is impractical for most deployments. These boundaries do not diminish the paper's core contribution but define its limits: process supervision improves reward model reliability, but it does not create new generator capabilities, and extracting its full benefit requires test-time compute budgets that may be infeasible in latency-sensitive applications.

Follow-Up Research This Work Enables

RL fine-tuning of the generator using the PRM as a reward signal. The paper explicitly scopes out generator improvement (Section 2.1: "finetuning the generator with RL is a natural next step, it is intentionally not the focus of this work"), but the PRM's step-level scores are precisely the kind of dense reward signal that RL algorithms need to improve chain-of-thought reasoning. The critical experiment is straightforward: take the fixed generator from this paper, use the trained PRM to score intermediate steps during rollouts, and apply PPO or a similar RL algorithm to fine-tune the generator toward higher PRM-scored reasoning. The key measurement would be whether RL training with the PRM produces a generator that (a) achieves higher pass@1 than the original generator, and (b) produces solutions that are actually more correct in their intermediate reasoning (not just better at optimizing the PRM score). The paper's own evidence on PRM over-optimization — the observation that the ORM's performance decreases on easy problems at high N (Appendix G, Figure 6, quintile 1) — raises the possibility that RL training against the PRM could lead to reward hacking, where the generator learns to produce steps that score highly under the PRM but are not actually correct. A strong follow-up would measure both final-answer accuracy and human-judged step-level correctness after RL training, and would test whether the PRM's resistance to false positives (relative to the ORM) translates to more robust RL training. The out-of-distribution evaluation in Section 5 (Table 1) provides a natural generalization test: does RL training with the PRM improve performance on the AP/AMC exams where the PRM already shows strong discriminative ability?

Full process supervision beyond the first error. The paper's deliberate choice to supervise only up to the first incorrect step (Section 2.6) was motivated by fair comparison with outcome supervision, but it artificially constrains the information content of process supervision. The natural follow-up is to collect a dataset — even a modest one, perhaps 10-20% the size of PRM800K — where human labelers annotate every step in a solution, including steps after the first error, and train a PRM on this richer signal. The comparison point is the paper's first-error-only PRM, evaluated on the same test set. The hypothesis is that full process supervision would provide additional benefit on hard problems where solutions contain multiple errors — knowing that a solution contains errors at steps 3, 7, and 11 is more informative than knowing only that step 3 is wrong. The paper's difficulty quintile analysis (Appendix G, Figure 6) provides a natural stratification: the benefit of full supervision should be largest on harder problems (quintiles 4-5) where the generator's solutions are longer and more likely to contain multiple errors. A secondary measurement would be whether full supervision improves the PRM's ability to provide useful feedback for correction — if the PRM can identify all errors in a solution, it can potentially guide a revision model toward the correct answer by specifying exactly what needs to be fixed.

Cross-model generalization of the PRM advantage. The paper uses a single base model family (GPT-4 and its smaller variants) for all experiments. A critical open question is whether the PRM advantage transfers across model architectures and scales. The concrete experiment: train ORMs and PRMs using identical supervision protocols (synthetic labels from a large oracle PRM, following Section 4's methodology) for a diverse set of base models — e.g., LLaMA-2, Claude, Gemini, Mistral — at comparable scales, and measure whether the process-vs-outcome gap is consistent or varies by model family. The paper's synthetic supervision framework makes this feasible without additional human annotation: PRMlarge (the GPT-4-based PRM) can serve as the oracle for training reward models on other base models' outputs. The key measurement is the interaction between base model architecture and supervision type: do some model families produce errors that are more amenable to step-level detection? Do models with different pretraining data (e.g., code-heavy vs. text-heavy) show different PRM advantage magnitudes? If the PRM advantage is universal across architectures, the case for process supervision strengthens considerably; if it is specific to GPT-4-like models, the finding is narrower than the paper's alignment arguments imply.

Process supervision for non-math reasoning domains. The paper explicitly acknowledges the domain limitation (Section 6.2) and calls for exploration of process supervision in other domains. The most natural extensions are to domains that share structural features with mathematical reasoning but differ in important ways. Code generation is the closest analog: solutions have discrete steps (lines or blocks of code), correctness can be partially verified automatically (unit tests, execution), and errors are typically localized. The key difference is that code reasoning involves different types of errors (off-by-one, API misuse, logic errors) that may be harder or easier for humans to label at the step level. A direct replication of the paper's methodology on a code generation benchmark (e.g., HumanEval, MBPP, or competition programming problems) would test whether the PRM advantage transfers to a domain where the "step" unit is syntactic (code lines) rather than semantic (reasoning steps). Multi-step planning and tool use is a harder test: solutions involve sequences of actions with environmental feedback, and step correctness depends on context (an action that is correct in one state may be incorrect in another). The paper's labeling protocol (positive/negative/neutral at each step) would need modification — action correctness is path-dependent in ways that mathematical step correctness is not — but the core question remains: does step-level supervision outperform outcome supervision when the definition of a "correct step" is more context-sensitive? A strong follow-up would identify a domain where human labelers show substantially lower inter-annotator agreement on step correctness than they do for MATH (the paper does not report agreement statistics, but math labeling likely has high agreement), and test whether process supervision's advantage persists or diminishes as label noise increases.

Scaling the active learning strategy to larger pools and iterative retraining. The paper's active learning result (2.6× efficiency, Section 4.2) is demonstrated with a specific setup: a PRMselector trained on a single sample per problem, selecting from a pool of 1000 samples per problem, with an 80/20 wrong-answer/remaining split, and without iterative retraining of the selector (which the paper tried and found unstable). Several follow-up experiments would refine this result. First, vary the pool size: if the selection pool is 10,000 samples instead of 1,000, does the active learning efficiency multiplier increase (because there are more "convincing wrong-answer" solutions to find) or decrease (because the PRMselector saturates and cannot distinguish among the top candidates)? The paper notes that at 200 samples per problem (the largest active learning data point), performance "appears to slightly underperform the expected trend line" due to "relative lack of diversity" — this suggests pool size matters and should be systematically studied. Second, diagnose and fix the iterative retraining instability: the paper reports that iteratively retraining PRMselector caused instability but provides no details. A careful ablation of the retraining protocol — varying the retraining frequency, the amount of new data between retrainings, the learning rate schedule, and whether to reset or continue training — could identify the source of instability and potentially unlock additional efficiency gains. Third, test whether PRMselector can be replaced by a simpler heuristic: if the goal is to find convincing wrong-answer solutions, perhaps a simple confidence-thresholded outcome check (select solutions where the generator's own token probabilities indicate high confidence but the final answer is wrong) could approximate the PRMselector's selection at near-zero cost. If such a heuristic works, active learning becomes essentially free, dramatically improving the economics of process supervision data collection.

Measuring the alignment benefits directly. The paper claims that process supervision produces more aligned, interpretable reasoning (Section 6.2), but provides no direct measurement of these properties. A strong follow-up would operationalize and measure the alignment benefits concretely. One experiment: take the best-of-N selected solutions from the PRM and the ORM on the MATH test set (at matched N, say N=100), and have human evaluators (or an independent large model) judge the step-level correctness and clarity of each selected solution, blinded to which reward model selected it. The hypothesis is that PRM-selected solutions would have fewer undetected intermediate errors and clearer reasoning than ORM-selected solutions, even when both reach the correct final answer. The paper's qualitative false positive examples in Appendix I (Problems 9-12) show cases where the PRM misses subtle errors — quantifying how often this happens relative to the ORM would directly test the alignment claim. A second experiment: train a generator with RL using the PRM as a reward signal, train another generator with RL using the ORM as a reward signal, and measure whether the PRM-trained generator produces solutions that are more faithful to correct reasoning (fewer spurious solutions that reach the right answer through wrong reasoning) than the ORM-trained generator. The paper's own citation of Zelikman et al. (2022) and Creswell et al. (2022) establishes that ORM-trained models "regularly use incorrect reasoning to reach the correct final answer" — the open question is whether PRM training reduces this behavior, and by how much.

Practical Applications and Downstream Use Cases

Best-of-N selection for high-stakes mathematical reasoning at modest compute budgets. While the paper's headline 78.2% requires N=1860, Figure 3 shows that the PRM's advantage over the ORM and majority voting is already substantial at N=100, where the PRM achieves roughly 72-74% accuracy (reading from the figure) — a gap of approximately 4-6 percentage points over the ORM. For applications where mathematical correctness is critical and some latency can be tolerated — automated grading of student solutions, verification of engineering calculations, financial modeling with objective correctness criteria — generating 100 candidate solutions and selecting the top-ranked one with a PRM provides meaningful accuracy gains over the standard ORM or majority voting approaches, at a test-time compute cost that is high but not prohibitive for batch processing. The PRM800K dataset enables training such a PRM without the data collection burden, making this application immediately implementable for any group with access to a GPT-4-class generator.

Active learning for cost-efficient reward model training in new domains. The 2.6× data efficiency improvement from active learning (Section 4.2, Figure 4a) is directly actionable for any team that needs to train a reward model with expensive human feedback. The "convincing wrong-answer" selection strategy requires only a current best reward model and a way to identify incorrect final answers (automatic checking or oracle judgment) — both of which are available in any domain with ground-truth outputs. The practical workflow is: (1) train an initial reward model on a small uniformly-sampled dataset; (2) generate a large pool of candidate solutions; (3) score them with the current reward model; (4) surface the highest-scoring incorrect-answer solutions for human labeling; (5) retrain; (6) optionally repeat. The paper's 80/20 split (80% wrong-answer, 20% most convincing remaining) provides a specific recipe that balances informativeness with calibration. This workflow can reduce human annotation costs by more than half compared to uniform sampling, making high-quality reward model training feasible for teams with limited annotation budgets. The release of PRM800K provides a reference dataset against which new active learning strategies can be benchmarked.

Verification layers for chain-of-thought reasoning systems. The PRM's architecture — a language model fine-tuned to predict step-level correctness — is simple enough that it can be deployed as a lightweight verification layer on top of existing chain-of-thought systems. For any application where a language model produces step-by-step reasoning (code generation, legal analysis, scientific explanation, medical diagnosis), a domain-specific PRM can be trained (using the paper's Monte Carlo rollout or human labeling methodology) to score each intermediate step and flag potential errors before they propagate. The PRM's step-level scores are inherently interpretable (visualized as color-coded steps in Figure 2 and Appendix I), meaning a human-in-the-loop system can surface the specific step where the PRM's confidence drops, allowing targeted human review rather than requiring the human to read the entire solution. The paper's finding that the PRM's advantage over the ORM widens with test-time compute (Figure 3) suggests that this verification layer is particularly valuable when the underlying generator is sampled multiple times — the PRM can effectively choose among many candidate reasoning paths, each with step-level quality annotations.

Negative alignment tax as a design principle for safety interventions. The paper's finding that process supervision simultaneously improves capability and safety (Section 6.2) provides a concrete design target for safety researchers: seek supervision methods where the alignment tax is negative. Rather than accepting a capability-safety tradeoff as inevitable and focusing on minimizing the tax, researchers can actively search for methods where safety and capability are aligned — where providing more detailed, human-endorsed feedback to the model improves its performance on the target metric. The paper's analysis of why process supervision achieves this (easier credit assignment, resistance to false positives from spurious solutions) suggests specific properties to look for in other domains: tasks where errors are localized, where outcome-based feedback suffers from false positives due to spurious correctness, and where human evaluators can reliably identify errors at a finer granularity than the final output. This reframes the safety research agenda from "how much capability must we sacrifice for safety?" to "which supervision methods improve both axes simultaneously, and what properties do they share?"