The Lessons of Developing Process Reward Models in Mathematical Reasoning

ArXiv: 2501.07301

🎯 Pitch

Monte Carlo estimation, the dominant method for generating process reward model training data, is actually inferior to LLM-as-a-judge and human annotation because it routinely mislabels step correctness. To fix this, the authors introduce a consensus filtering mechanism that uses only data where MC and LLM-judge methods agree, cutting data by 60% while producing a new state-of-the-art PRM that scores 78.3 mean F1 on ProcessBench—exceeding o1-mini’s 87.9%—and finally aligns process scoring with process verification rather than just rewarding final answers.

---

1. Executive Summary

This paper analyzes the data annotation and evaluation challenges in developing Process Reward Models (PRMs) for mathematical reasoning, conducting extensive experiments using Qwen2.5-Math-Instruct models on benchmarks including GSM8K, MATH, and PROCESSBENCH. The work identifies that the commonly used Monte Carlo (MC) estimation method (estimating step correctness by computing the empirical probability of reaching correct final answers through repeated completions) yields inferior performance and generalization compared to LLM-as-a-judge (using a language model to directly verify each reasoning step) and human annotation, and further reveals that conventional Best-of-N (BoN) evaluation (selecting the highest-scored response from N candidates) suffers from a misalignment where unreliable policy models generate responses with correct answers but flawed processes, leading to inflated scores and a process-to-outcome shift in PRM behavior. To address these limitations, the authors propose a consensus filtering mechanism that retains only instances where MC estimation and LLM-as-a-judge agree on error step locations, achieving both improved data efficiency—using only ~40% of the original data—and state-of-the-art performance on PROCESSBENCH, with their released Qwen2.5-Math-PRM-72B reaching 78.3 mean F1 on process error identification while their 7B variant surpasses maj@8 across all seven BoN evaluation tasks, establishing that effective PRM training requires combining complementary annotation sources and supplementing response-level evaluation with step-level metrics.

2. Context and Motivation

The Core Problem: PRMs Should Supervise Reasoning, But Their Training Data Is Fundamentally Flawed

The central puzzle this paper confronts is simple but deeply consequential: if Process Reward Models (PRMs) are designed to identify and correct intermediate errors in mathematical reasoning, why do PRMs trained using the most popular automated data generation method often fail to actually detect those errors?

A PRM's purpose is to assign a correctness score to each step in a chain of mathematical reasoning. This fine-grained supervision over the reasoning process — as opposed to Outcome Reward Models (ORMs) that only score the final answer — is the entire rationale for PRMs over simpler alternatives. If a PRM cannot reliably distinguish a logically sound derivation from one that stumbles into the right answer through faulty reasoning, it fails at its defining task. The paper's opening framing in Section 1 underscores this directly:

"even when achieving correct final answers, these powerful models can still regularly make up plausible reasoning steps, where the final answers build upon flawed calculations or derivations, which undermine the reliability and trustworthiness of LLMs' reasoning processes."

This problem is not merely academic. Mathematical reasoning serves as a microcosm for broader LLM reliability concerns — medicine, law, finance, and engineering all require confidence that a model's conclusions follow from valid intermediate logic, not just that the final output happens to be correct in some fraction of cases. A PRM that genuinely supervises the reasoning process would enable safer deployment of LLMs in high-stakes domains, more effective reinforcement learning from process-level feedback, and automated self-correction pipelines where models iteratively improve their own reasoning. But such a PRM must actually work — and the paper's preliminary experiments in Section 2.3 suggest that, under current data construction practices, they largely don't.

The Gap: MC Estimation Is the Dominant Data Generation Method Despite Unknown Limitations

To train a PRM, you need labeled data indicating which steps in a solution are correct and which are incorrect. Human annotation — the approach used in the foundational PRM800K dataset (Lightman et al., 2023) — produces high-quality labels but is prohibitively expensive. The natural response from the research community has been to automate the labeling process, and the dominant automation method is Monte Carlo (MC) estimation, popularized by Math-Shepherd (Wang et al., 2024b).

The MC estimation procedure works as follows: for each reasoning step in a solution, you generate multiple completions from that step onward using a completion model, check how many of those completions reach the correct final answer, and use the fraction of successful completions as a proxy for step correctness. If most completions from a given step lead to the correct answer, that step is labeled correct; if few do, it is labeled incorrect.

This approach has been widely adopted. As the paper notes in Section 1:

"one commonly used approach is to assess process correctness by estimating the empirical probability of leading to the correct final answers through Monte Carlo (MC) methods, which has attracted great research interests and has also been commonly employed in practice (Xiong et al., 2024; Wang et al., 2024b; Luo et al., 2024)."

The appeal is clear: it requires no human labor, scales to massive datasets, and has face validity — surely a step is correct if it reliably leads to correct answers? But the paper argues that this face validity is deceptive. The gap the paper addresses is that no prior work had systematically compared MC estimation against alternative data construction methods (LLM-as-a-judge, human annotation) on the same models and benchmarks, nor identified the mechanisms by which MC estimation might fail.

Where Prior Approaches Fall Short: The Mechanisms Behind MC Estimation Failure

The paper identifies specific, mechanistic reasons why MC estimation underperforms, which prior work had not characterized:

1. MC estimation conflates process correctness with future outcome probability. The paper draws a critical distinction in Section 3.1.1 between PRMs and value models:

"The key difference between PRM and value model lies in that PRMs function as deterministic evaluators of current step correctness, while value models operate as predictive estimators of future solution potential."

MC estimation, by design, estimates the probability that a correct final answer will be reached from the current step. But this is a value model's job — estimating future potential — not a PRM's job of deterministically judging whether the current step itself is correct. A step can be mathematically sound yet the completion model may fail to capitalize on it (producing an incorrect final answer due to its own errors), leading MC estimation to falsely label the step as incorrect. Conversely, a step can contain an error yet the completion model might "correct" it in subsequent reasoning or arrive at the correct answer through an alternative path, leading MC estimation to falsely label the step as correct. The paper articulates this in Section 3.1.3:

"We attribute the inferior performance of MC estimation compared to LLM-as-a-judge and human annotation to its high noise in reasoning step correctness estimation and inaccurate error position identification due to its heavy dependence on the policy model."

2. MC estimation introduces high variance with crude estimation. The standard practice uses a small number of completions (typically 8) to estimate the probability of reaching the correct answer. With only 8 binary outcomes, the variance is high — a step that is truly correct might get 5/8 successful completions, yielding a soft label of 0.625 rather than 1.0. The paper shows (Section 3.1.4) that this noise actively degrades training, and that hard labels with a threshold of 0 (label the step correct if any completion succeeds) outperform soft labels. But even with optimal thresholding, the fundamental noise remains.

3. The completion model can "correct" errors, masking process failures. The paper highlights a specific failure mode: "the policy model may generate correct final answers but incorrect reasoning steps" (Section 3.1.3). When the completion model takes over from a flawed intermediate step and somehow reaches the right answer — perhaps by ignoring the error, serendipitously compensating for it, or following a different reasoning path — MC estimation labels that flawed step as correct. This means MC estimation systematically under-annotates process errors, training the PRM to overlook the very mistakes it is supposed to catch.

4. Prior work lacked comprehensive evaluation, so these failures went undetected. The paper argues (Section 3.2) that the research community has relied almost exclusively on Best-of-N (BoN) evaluation to measure PRM quality — selecting the highest-scored response from N candidates and checking if the answer is correct. But BoN only measures whether the PRM can identify responses with correct answers, not whether it can identify responses with correct reasoning. If a PRM assigns high scores to answers that happen to be correct regardless of reasoning quality — which is exactly what MC-trained PRMs learn to do — it can score well on BoN while being useless for actual process supervision. The paper demonstrates this empirically in Section 3.2.2 and Table 5, showing that most open-source PRMs fail to identify process errors in responses that have correct answers, with detection accuracy rates below 50%.

5. The BoN optimization objective creates perverse incentives. The paper's analysis in Section 3.2.3 reveals that BoN-optimized PRMs exhibit a "process-to-outcome shift" — their minimum step scores concentrate disproportionately on the final answer step (Figure 8), meaning the PRM is essentially acting as an ORM, judging solutions primarily by their final answer rather than their intermediate reasoning. This is not a bug in implementation; it is an emergent consequence of the BoN evaluation paradigm itself. The paper states:

"optimizing solely for the BoN evaluation has made current PRMs perform more like ORMs in practice."

Conflicting Signals in the Literature, Resolved

The paper's findings also help resolve a tension in the existing literature. Prior work generally reported positive results from MC-estimation-based PRMs when evaluated on BoN (Wang et al., 2024b; Luo et al., 2024). These positive BoN numbers created a perception that MC estimation was working effectively. But the paper shows that this BoN performance is inflated — it reflects the PRM's ability to select for correct answers rather than correct processes (Section 3.2.2). When the same models are evaluated on PROCESSBENCH (Zheng et al., 2024), which explicitly tests for erroneous step identification, the MC-trained PRMs collapse: Qwen2.5-Math-7B-PRM-MC-hard achieves only 40.2 mean F1 compared to 56.5 for the human-annotation-trained PRM800K baseline, despite being trained on far more data (860k vs. 264k samples).

The paper thus exposes a systematic overestimation of MC-estimation-based PRM quality in the prior literature, driven by evaluation methodology that failed to measure the thing PRMs are supposed to be good at.

How This Paper Positions Itself

The paper positions itself as a diagnostic and corrective effort, not as a wholly new methodological contribution. Its stance is: the community has converged on a suboptimal data generation practice (MC estimation) and a misleading evaluation protocol (BoN-only), and until both are revised, PRM development will stagnate. The paper's contributions — the consensus filtering mechanism and the advocacy for dual evaluation — are presented as practical remedies that emerge naturally once the underlying problems are understood.

The positioning is deliberately empirical rather than theoretical. The authors conducted their own internal PRM training following "conventional principles" (Section 2) and discovered the limitations through direct experimentation:

"during the training of our own PRM following conventional principles to construct data using MC estimation and evaluate on BoN, we gained several crucial lessons."

This framing — "we tried the standard approach, it didn't work well, and here's why" — is effective because it positions the authors as practitioners who share the reader's assumptions and only revise them in response to evidence. The paper does not claim to introduce a new model architecture or a fundamentally new training objective; instead, it claims to identify and correct flawed assumptions in the existing pipeline of data construction and evaluation.

The paper's relationship to existing work can be summarized along two axes:

• Regarding data construction: Prior work either used pure MC estimation (Wang et al., 2024b; Luo et al., 2024; Xiong et al., 2024), pure LLM-as-a-judge (Zhang et al., 2024; Gao et al., 2024), or pure human annotation (Lightman et al., 2023). The paper's consensus filtering mechanism (Section 3.1.3) is the first to systematically combine MC estimation and LLM-as-a-judge through agreement filtering, rather than treating them as alternative approaches. The key insight is that the two methods have complementary error modes — MC estimation misses process errors when the completion model "corrects" them, while LLM-as-a-judge can directly analyze step-by-step logic — and requiring consensus eliminates noise that either method alone would propagate.

• Regarding evaluation: Prior work treated BoN as the primary or sole evaluation metric for PRMs (Lightman et al., 2023; Wang et al., 2024b; Luo et al., 2024). The paper argues that BoN is necessary but insufficient, and that PROCESSBENCH-style step-level evaluation must be added to detect when PRMs have degenerated into ORMs. This dual-evaluation framework is positioned not as replacing BoN but as complementing it — BoN measures downstream utility, PROCESSBENCH measures fidelity to the stated objective.

Why This Matters Now

The paper's intervention is timely because the field is at an inflection point. PRMs are being integrated into reinforcement learning pipelines (RLHF) and search-based decoding strategies, where their process supervision signal directly influences model behavior. If that signal is unreliable — if the PRM is essentially an ORM in disguise — then these downstream applications inherit all the limitations of outcome-only supervision while incurring the additional cost of step-level scoring. The paper's finding that even strong open-source PRMs like Skywork-PRM-7B assign their minimum step scores to the final answer step 42.2% of the time (Figure 8) is alarming in this context: it means that in nearly half of all cases, the PRM's judgment about which solution is best is driven entirely by its assessment of the final answer, not the reasoning that led there.

The release of PROCESSBENCH (Zheng et al., 2024) also makes this work possible — prior to its availability, there was no standardized benchmark for measuring step-level error identification in PRMs. The paper leverages PROCESSBENCH extensively and contributes to its adoption as a complementary evaluation standard alongside BoN.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a diagnostic analysis paper that uses extensive experiments to identify why the standard pipeline for training Process Reward Models (PRMs) — Monte Carlo (MC) estimation for data annotation paired with Best-of-N (BoN) evaluation — systematically fails to produce models that can actually identify reasoning errors, and then develops two corrective interventions: a consensus filtering mechanism for cleaner training data and a dual-evaluation framework for honest assessment. The system being built is a better training-and-evaluation pipeline for PRMs, not a fundamentally new model architecture, and the core "shape" of the solution is to triangulate step correctness using two complementary and partially independent signals (LLM-as-a-judge and MC estimation), keeping only data where they agree, and to measure PRM quality along two orthogonal axes (response-level BoN for downstream utility, step-level PROCESSBENCH for process fidelity) to prevent evaluation from drifting away from the model's stated purpose.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's technical contribution has two distinct subsystems that interact:

Subsystem 1 — Training Data Construction Pipeline:

1. Query-Response Generator: Takes ~500K math queries with golden answers; samples 6–8 responses per query from Qwen2-Math-Instruct and Qwen2.5-Math-Instruct (7B and 72B variants) at elevated temperature.
2. Step Splitter: Splits each response into individual reasoning steps using the delimiter "\n\n".
3. MC Estimator: For each step, runs 8 independent completions from Qwen2.5-Math-Instruct (same model size as the response generator), computes the fraction reaching the correct answer. Assigns a hard label — step is positive if any completion succeeds, negative only if none succeed.
4. LLM-as-a-Judge Critic: Independently, Qwen2.5-72B-Instruct analyzes each step's logical correctness given the problem context and previous steps, producing a per-step correctness label.
5. Consensus Filter: Retains only instances where the MC estimator and LLM-as-a-judge agree on where errors occur in the solution. Disagreements are discarded. This reduces the dataset to ~40% of its original size.
6. PRM Training: A Qwen2.5-Math-Instruct model (7B or 72B) with its language modeling head replaced by a scalar-value head (two linear layers) is fine-tuned with cross-entropy loss on the last token of each step, predicting the binary correctness label from the consensus-filtered data.

Subsystem 2 — Evaluation Framework:

1. BoN Evaluation (response-level): For a policy model's N sampled responses, the PRM scores each step, combines scores via product (or minimum/last depending on context), selects the highest-scored response, and checks answer correctness against ground truth.
2. PROCESSBENCH Evaluation (step-level): The PRM scores each step in a solution; the model identifies the first step whose score falls below a detection threshold as the erroneous step, and is scored against ground-truth error locations using F1.

The flow through the system during deployment: a query → policy model generates N candidate responses → each response is split into steps → the PRM scores every step → scores are aggregated within each response → the best response is selected (BoN) or the first low-scoring step is flagged as an error (PROCESSBENCH).

3.3 Roadmap for the Deep Dive

• First, the MC estimation data construction pipeline — the dominant automated approach the paper critiques. Understanding its mechanics is prerequisite to understanding why it fails.
• Second, the LLM-as-a-judge data construction pipeline — the comparator method that reveals MC estimation's weaknesses and provides the second signal for consensus filtering.
• Third, the consensus filtering mechanism — the paper's core methodological contribution. This is where the two annotation sources are combined, and understanding why this works requires understanding the failure modes of each individually.
• Fourth, the PRM architecture and training procedure — the model itself (scalar head replacement, loss function, initialization) and the design choices around hard vs. soft labels, threshold selection, and scoring strategy during inference.
• Fifth, the dual evaluation framework — the complementary BoN and PROCESSBENCH metrics, how they measure different things, and why using both is essential rather than optional.
• Sixth, the PRM-guided greedy search variant — a secondary application of the trained PRM explored in Appendix A, which provides additional evidence about the distinction between PRMs and value models.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a diagnostic and corrective paper whose core technical idea is that PRM training data suffers from a fundamental aliasing problem: MC estimation measures future outcome probability, not current step correctness, and BoN evaluation rewards outcome-based scoring, creating a vicious cycle where PRMs trained on MC data and evaluated on BoN converge to behaving like ORMs. The corrective is twofold: filter training data using agreement between MC estimation (outcome-based) and LLM-as-a-judge (process-based) to remove instances where the two signals conflict, and supplement BoN with step-level evaluation to detect when a PRM has drifted from its stated purpose.

---

MC Estimation Data Construction Pipeline

What MC estimation does. For a given math problem and a candidate solution broken into steps, MC estimation attempts to evaluate the correctness of step $k$ by asking: "if a completion model continues from this step, how often does it reach the correct final answer?" The procedure for each step $k$ is:

1. Take the partial solution consisting of the problem statement plus steps 1 through $k$.
2. Generate $C$ independent completions from this partial solution using a separate completion model (Qwen2.5-Math-Instruct, same parameter scale as the response generator). Each completion continues the reasoning from step $k$ onward until a final answer is produced.
3. For each completion, check whether the final answer matches the known golden answer for the problem.
4. Compute the empirical success rate.

Label construction. The paper uses $C = 8$ completions throughout (Section 2.1: "we conduct 8 independent completions starting from this step"). From these 8 completions, two labeling schemes are explored:

• Soft labels: The label $y \in [0, 1]$ is the proportion of the 8 completions that reach the correct final answer. So $y = 5/8 = 0.625$ if 5 completions succeed.
• Hard labels: A step is labeled positive (correct) if any of the 8 completions succeeds — equivalent to a threshold of 0, where the step is negative only if $y = 0$ (all completions fail). Formally, hard_label = 1 if $\exists$ at least one successful completion, else 0.

The paper also experiments with higher thresholds (Section 3.1.4, Figure 5), testing thresholds from 1/8 through 7/8, but reports that threshold 0 performs best on both BoN and PROCESSBENCH:

"as the threshold increases, the performance deteriorates on both Best-of-8 and PROCESSBENCH, indicating that using an MC estimated value of 0 as the negative label and all others as positive labels yields the best results."

Step truncation rule. The paper applies a critical filtering step during data construction: once a step is labeled as incorrect (hard label 0), all subsequent steps in that solution are discarded from the training data (Section 2.1):

"we eliminated all steps subsequent to those labeled as incorrect (label 0), as their validity becomes irrelevant after an error occurs. This removal was implemented to prevent potential model confusion during training."

This means the PRM is never trained on steps that follow an error — it only sees correct sequences where every step is labeled 1, plus the first erroneous step in a sequence which gets label 0. The rationale is that after an error, the rest of the reasoning is built on a faulty foundation and training the PRM to evaluate those downstream steps could teach it to accept flawed derivations that happen to be logically consistent with a prior mistake.

Scale of data. The paper constructs datasets at three scales for MC estimation experiments (Section 3.1.2, 3.1.4):

• Math-Shepherd opensource dataset: ~440K samples
• The authors' own MC-estimated dataset: ~860K samples
• A larger 3M-sample dataset used for the hard-vs-soft label and threshold experiments

The query set consists of approximately 500,000 unique math problems with golden answers. For each query, 6–8 diverse responses are generated by mixing outputs from Qwen2-Math-Instruct and Qwen2.5-Math-Instruct at both 7B and 72B scales (Section 2.1).

Why this approach is widely used. MC estimation requires no human labor, scales arbitrarily with compute, and provides a natural scalar signal — the estimated probability of success — that can be used as either a soft regression target or thresholded into a hard binary label. Its conceptual simplicity (good steps lead to correct answers, bad steps don't) has face validity, and prior work (Wang et al., 2024b) reported strong BoN results using MC-estimation-trained PRMs.

Why it underperforms — the paper's mechanistic diagnosis. The paper identifies three specific failure modes of MC estimation:

Failure mode 1 — Correct steps with unsuccessful completions. A step can be logically and mathematically sound, but the completion model — which is itself imperfect — may fail to complete from that step correctly due to its own limitations. In this case, $y < 1$ or even $y = 0$ despite the step being correct. This introduces false negatives into the training data.

Failure mode 2 — Incorrect steps with successful completions. This is the more insidious failure. A step can contain a computational error or logical flaw, but the completion model might "repair" the error in subsequent reasoning (e.g., by re-deriving the correct value independently, or by reaching the correct answer through a different path that doesn't depend on the error). In this case, $y > 0$ despite the step being incorrect, introducing false positives. The paper highlights this in Section 3.1.3:

"the policy model may generate correct final answers but incorrect reasoning steps"

This is the fundamental aliasing problem: MC estimation measures whether the answer can be reached from here, not whether the reasoning step here is correct.

Failure mode 3 — Only 8 completions = high variance. With only 8 binary outcomes, the soft label is a crude estimate of the true underlying probability. A step that is truly correct (if we imagined infinite completions, the success rate would be, say, 0.9) might get 5/8 successes due to random variation, yielding a soft label of 0.625 — far from 1.0. This introduces label noise that the paper shows degrades training (Section 3.1.4). The authors note:

"only 8 completions for step correctness estimation exhibit high variance and are relatively crude. Although we can achieve better estimation accuracy by increasing the number of completions, the associated costs may outweigh the incremental benefits."

Design choice — Why hard labels outperform soft labels. The paper finds (Figures 3 and 4, Section 3.1.4) that after consensus filtering, hard labels substantially outperform soft labels on both BoN and PROCESSBENCH. The authors' explanation (Section 3.1.4) is:

"the correctness of steps (i.e., rewards) should be deterministic. Training PRMs with soft labels that represent future possibilities introduces additional noise. For instance, when numerous completely correct steps are assigned with soft labels lower than 1, it actually reduces the model's ability to discriminate between positive and negative labels."

The conceptual point is that a PRM's job is to make a binary judgment — is this step correct or not? — and training it with soft targets that represent an entirely different quantity (probability of future success) forces it to learn a value-like function rather than a verification function. The hard label threshold of 0 is essentially saying: the only reliable signal from MC estimation is the absence of any path to the correct answer; if even one path succeeds, we cannot confidently label the step as incorrect, so we conservatively label it correct. This asymmetry (high recall for correct labels at the cost of precision) is a deliberate design choice that acknowledges MC estimation's fundamental limitation.

The conceptual category error. The paper's most important theoretical point (Section 3.1.1) is that MC estimation trains a value model (which estimates the expected future return from a state) rather than a reward model (which evaluates the quality of the current state itself). The authors state:

"The key difference between PRM and value model lies in that PRMs function as deterministic evaluators of current step correctness, while value models operate as predictive estimators of future solution potential."

In reinforcement learning terms: reward $R(s)$ depends only on the current state $s$; value $V(s)$ depends on the expected sum of future rewards starting from $s$. MC estimation estimates $V(s)$ (will the final answer be correct?), but PRMs are supposed to output $R(s)$ (is this step logically sound?). When you train a model to predict $V(s)$ and then use it as if it outputs $R(s)$, you get a systematic mismatch — the model will assign high scores to steps from which correct answers are reachable, even if those steps themselves are flawed, and low scores to correct steps from which the (imperfect) completion model happens to fail.

---

LLM-as-a-Judge Data Construction Pipeline

What LLM-as-a-judge does. Instead of estimating future outcomes, LLM-as-a-judge directly assesses each reasoning step for logical and mathematical validity using a powerful language model as a critic. The paper employs Qwen2.5-72B-Instruct as the judge (Section 3.1.2), using a carefully constructed prompt template (Appendix C) that instructs the model to:

1. Review each paragraph (step) of the solution sequentially.
2. Analyze, verify, and critique the reasoning in detail — including recalculation where appropriate.
3. Determine whether the step is logically and mathematically valid.
4. If an error is found in any paragraph, stop further analysis of subsequent paragraphs and conclude "Incorrect."
5. If all paragraphs are error-free, conclude "Correct."

The prompt template (reproduced in Appendix C) is structurally significant. It requires the judge to output its analysis for each step in XML-style tags (<analysis_1>, <analysis_2>, etc.) and then provide a binary conclusion (<conclusion>Correct/Incorrect</conclusion>). Crucially, the prompt instructs an early-stopping rule:

"If an error is detected in any paragraph, you should describe the nature and cause of the error in detail, and suggest how to correct the error or the correct approach. Once a paragraph is found to contain any error, stop further analysis of subsequent paragraphs (as they may depend on the identified error) and directly provide the conclusion of 'Incorrect.'"

This early-stopping rule mirrors the MC estimation pipeline's truncation rule (discard steps after the first error) but is applied at analysis time rather than filtering time. It also means that the judge provides detailed error analyses only for the first erroneous step, with later steps left unanalyzed — matching the training data's structure where only the first error location matters.

Data scale. The LLM-as-a-judge annotation is applied to the same 860K query-response pairs used for MC estimation, enabling direct comparison between the two methods on identical underlying data (Section 3.1.2).

How LLM-as-a-judge avoids MC estimation's failure modes. The judge evaluates each step on its own terms — checking whether the derivation follows from the previous steps and the problem constraints — rather than looking forward to the final answer. This means:

• No false positives from answer-correction: If a step contains a computational error, the judge can identify it directly (e.g., "this step incorrectly computes $3 \times 7 = 24$ instead of $21$"), regardless of whether the final answer happens to be correct.
• No false negatives from completion model failures: If a step is correct, the judge can verify it without needing the completion model to successfully finish the solution.

However, LLM-as-a-judge introduces its own potential failure modes: the judge model may itself make errors in verification (especially for complex mathematical reasoning), may be over-confident or under-confident, and may exhibit systematic biases (e.g., being more lenient on certain types of errors). The paper does not deeply analyze LLM-as-a-judge failure modes in isolation, instead using the consensus filtering mechanism to mitigate them by requiring agreement with MC estimation.

Comparison of data construction methods. Table 3 and Table 4 (Section 3.1.2) present the crucial comparison. Three models are trained on exactly the same base model (Qwen2.5-Math-7B-Instruct) using the same architecture and training procedure, differing only in data source:

The pattern is striking and counterintuitive: MC estimation achieves the best BoN scores but the worst PROCESSBENCH scores, while human annotation achieves the worst BoN scores but the best PROCESSBENCH scores — with less than one-third the data. LLM-as-a-judge sits in the middle on both metrics. This inverse relationship between BoN and PROCESSBENCH performance is the paper's clearest empirical evidence that BoN alone is an unreliable metric for PRM quality.

---

Consensus Filtering Mechanism

The core insight. MC estimation and LLM-as-a-judge have complementary error modes. MC estimation is outcome-oriented: it measures reachability of the correct answer, producing false positives when the completion model "corrects" errors and false negatives when the completion model fails from correct steps. LLM-as-a-judge is process-oriented: it directly analyzes logical validity, producing errors when the judge model makes mistakes in mathematical reasoning. The two methods use fundamentally different information — future outcomes vs. step-by-step logic — and are therefore conditionally independent given the true step correctness, or at least have weakly correlated errors.

The consensus filtering mechanism exploits this: retain only those instances where both methods agree on error locations. When they disagree, at least one method is wrong about something, and that disagreement is more informative than the individual labels. Discarding disagreements removes noise at the cost of reducing dataset size.

What "consensus" means operationally: for a given solution with steps $[s_1, s_2, ..., s_n]$, MC estimation produces a hard label sequence $[l^{MC}_1, l^{MC}_2, ..., l^{MC}_n]$ and LLM-as-a-judge produces a label sequence $[l^{LLM}_1, l^{LLM}_2, ..., l^{LLM}_n]$ (with the judge's early-stopping rule, all steps after the first error are left unlabeled or labeled identically to MC). The consensus filter keeps the instance if and only if the two label sequences agree on the location of the first error (or agree that all steps are correct). Specifically:

"the instances are only retained when both LLM-as-a-judge and MC estimation show consensus on the error reasoning step locations in the solution" (Section 3.1.3)

This is not a simple per-step label agreement check — it must account for the sequential structure. If MC says the error is at step 3 and the judge says the error is at step 2, the instance is discarded even if both agree that steps 4+ are irrelevant. If MC says all steps are correct but the judge identifies an error at step 5, the instance is discarded.

Data retention rate. Figure 2's accompanying text states:

"only approximately 40% of the data are preserved after consensus filtering"

From the 860K original MC-estimated samples, approximately 350K survive the consensus filter. From the 3M larger dataset (Section 3.1.4), approximately 1.5M survive — a consistent ~50% retention rate, suggesting that MC estimation and LLM-as-a-judge disagree on roughly half of all step-level labels.

What the filter actually removes. The disagreement cases fall into two categories, though the paper does not explicitly break them down:

1. MC says positive, judge says negative. These are instances where MC estimation labeled a step as correct (because the completion model could reach the answer) but the judge identified a logical error. These are precisely the cases that cause the BoN-PROCESSBENCH divergence — retaining them would inflate BoN while degrading process verification.

2. MC says negative, judge says positive. These are instances where MC estimation labeled a step as incorrect (because completions failed) but the judge verified the step as logically sound. These are false negatives from MC estimation that would unnecessarily penalize correct reasoning.

By removing both types of disagreement, the filter eliminates the systematic errors from each method while keeping the instances where both independent assessments converge — which are presumably the high-confidence labels.

Performance impact. Figure 2 shows the effect on the 860K dataset:

• MC estimation alone (860K): BoN 65.9%, PB F1 40.1%
• LLM-as-a-judge alone (860K): BoN 65.3%, PB F1 46.5%
• Consensus filtered (350K): BoN 65.7%, PB F1 46.3%

The filtered dataset achieves comparable PROCESSBENCH performance to LLM-as-a-judge while using only ~40% of the data and comparable BoN performance to MC estimation. It essentially inherits the best of both methods: the BoN optimization from MC estimation's outcome-oriented signal (which helps select correct-answer responses) and the process verification from LLM-as-a-judge's step-by-step analysis (which ensures the PRM actually learns to identify errors).

On the larger 3M dataset (Section 3.1.4, Figures 3 and 4), the filtered 1.5M samples with hard labels achieve BoN 67.2% (vs. 65.4% for the unfiltered 3M) and PB F1 66.5% (vs. 40.2% for unfiltered) — a dramatic improvement in process verification with a modest gain in BoN.

Why this works — the theoretical justification. The paper does not provide a formal theory, but the empirical logic is: MC estimation provides a noisy but unbiased (or at least differently biased) estimate of a different quantity (value) than what we want (reward). LLM-as-a-judge provides a noisy estimate of the quantity we actually want (reward/correctness). Requiring consensus between two noisy estimators that use different information sources acts as a form of triangulation: the probability that both estimators are simultaneously wrong in the same direction is lower than the probability that either alone is wrong, assuming their errors are not perfectly correlated. The cost is reduced data quantity, but the paper's results show that quality dominates quantity in this regime — 350K clean samples outperform 860K noisy ones.

---

PRM Architecture and Training Procedure

Model initialization. The paper initializes PRMs from the supervised fine-tuned Qwen2.5-Math-Instruct models (Section 2.1), at both 7B and 72B parameter scales. These base models have already been fine-tuned on mathematical instruction-following data, giving them strong mathematical reasoning capabilities that can be redirected toward verification.

Architecture modification. The standard language model head (which outputs logits over the vocabulary for next-token prediction) is replaced with a scalar-value head:

"we replace the original language modeling head (used for next token prediction) with a scalar-value head, consisting of two linear layers." (Section 2.1)

This means the model processes the entire input sequence through its transformer layers as usual, but instead of predicting the next token at each position, it extracts the hidden state at a specific token position (the last token of each reasoning step) and passes it through two linear layers to produce a single scalar value. This scalar represents the model's predicted probability that the step is correct.

The two-layer structure (linear → activation → linear) provides a minimal non-linear transformation from the high-dimensional hidden state to a scalar, which is standard practice for reward model heads. The paper does not specify the activation function between the two linear layers, though ReLU or GELU would be typical choices.

Training objective — Binary classification with cross-entropy. Since the paper uses hard labels after consensus filtering (Section 3.1.4 recommends hard labels), the training task is a binary classification: predict whether each step is correct (1) or incorrect (0). The loss is computed only on the last token of each step, not on every token in the sequence:

"We calculated the cross-entropy (CE) loss and mean squared error (MSE) loss on the last tokens of each step for the binary classification task using hard labels and for the regression task using soft labels, respectively." (Section 2.1)

For the hard-label case (the primary and recommended approach), the loss for a single step is:

$\mathcal{L}_{\text{step}} = -\left[y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})\right]$

where $y \in \{0, 1\}$ is the hard label (1 for correct, 0 for incorrect) and $\hat{y} \in [0, 1]$ is the model's predicted probability of correctness after applying a sigmoid activation to the scalar head's output.

What it computes: For each reasoning step, the cross-entropy loss measures the divergence between the model's predicted correctness probability and the binary ground-truth label. When $y = 1$ (step is correct), the loss reduces to $-\log(\hat{y})$, which heavily penalizes the model for predicting low probability. When $y = 0$ (step is incorrect), the loss reduces to $-\log(1 - \hat{y})$, which heavily penalizes the model for predicting high probability. The total loss for a solution is the sum (or average) of these per-step losses.

Why this form: The sigmoid + binary cross-entropy combination is the standard maximum-likelihood objective for binary classification. It is well-calibrated for probability estimation (the optimal prediction under this loss is the true conditional probability of correctness given the input) and produces gradients that are well-behaved — large when the prediction is confidently wrong, small when it's confidently correct. The paper explicitly compares this against MSE with soft labels and finds CE with hard labels superior after consensus filtering (Figures 3 and 4).

Why not MSE with soft labels? The paper's negative results with soft labels even after filtering (Figure 4: soft labels achieve PB F1 ~49% vs. hard labels' ~67%) demonstrate that the soft label signal — which represents future outcome probability, not current-step correctness — remains fundamentally misaligned with the PRM's objective even when filtered for consensus. The hard label removes the continuous-valued noise and forces a clean binary decision, which matches the PRM's purpose as a deterministic verifier. The paper articulates this in Section 3.1.4:

"the correctness of steps (i.e., rewards) should be deterministic. Training PRMs with soft labels that represent future possibilities introduces additional noise."

Training hyperparameters. The paper does not provide explicit optimizer, learning rate, batch size, or dropout specifications for PRM training in the main text. The training details in Section 2.1 state the architecture change but defer hyperparameter specifics. Section 4.1 for the final models states:

"For the training task, we employ cross-entropy loss on the tokens at the end of each step to train the binary classification task."

No learning rate schedule, optimizer choice, or training duration is specified. This is a notable gap in the paper's reproducibility — the training procedure for the released models (Qwen2.5-Math-PRM-7B/72B) is described at a high level without the specific numerical configurations that would be needed to replicate the results.

MC estimation threshold selection. The paper performs a systematic sweep of hard-label thresholds in Section 3.1.4, testing values from 1/8 to 7/8 (i.e., requiring 1, 2, ..., 7 out of 8 successful completions to label a step as positive). The results in Figure 5 show a monotonic degradation as the threshold increases:

Operational interpretation: Threshold 0 means "label the step correct if there's any evidence the answer can be reached; label it incorrect only if there's no evidence." This is maximally permissive — it minimizes false negatives at the cost of false positives. The fact that this permissive strategy works best supports the paper's diagnosis: MC estimation's primary failure mode is false negatives (correct steps where the completion model fails), and being permissive reduces this noise. Higher thresholds require more completions to succeed, which amplifies the false negative problem (correct steps that happen to get few successes get incorrectly labeled as wrong).

The paper's final recommendation (Section 3.1.5):

"if we have to rely on MC estimation for step-wise correctness verification, we suggest setting the threshold to 0, meaning that a step is considered correct if any completion start from this step reaches the correct final answer."

Scoring strategy during inference. When using the trained PRM to score a complete solution during BoN evaluation, the per-step scores must be aggregated into a single solution-level score. The paper explores three aggregation methods (Sections 3.2.4 and Appendix B):

• Product: Multiply all step scores together: $\text{score}_{\text{solution}} = \prod_{i=1}^{n} \hat{y}_i$
• Minimum: Take the lowest step score: $\text{score}_{\text{solution}} = \min_i \hat{y}_i$
• Last: Use only the final step's score: $\text{score}_{\text{solution}} = \hat{y}_n$

The paper finds (Figure 9, Section 3.2.4) that the optimal scoring strategy depends on the training data:

• For MC-estimation-trained PRMs: The last score strategy significantly outperforms product and minimum. This makes sense: if the model was trained to predict "can the correct answer be reached from here" (value-like), then the final step's prediction already integrates information from the entire solution, and multiplying or taking the minimum across steps introduces unnecessary noise. The authors state:

"In this case, each step's score actually estimates the probability of reaching the correct final answer in the future from the current position. Given this forward-looking nature of MC estimation, we should neither multiply the estimated probabilities across steps (as these estimates are dependent on each other), nor simply take the minimum estimated value from a particular step as the overall score."

• For LLM-as-a-judge and human-annotation-trained PRMs: Product and minimum are more appropriate than last score. This is because these PRMs were trained to predict actual step correctness (reward-like), so each step score is an independent assessment. The product captures the logical "all steps must be correct" requirement; the minimum captures the weakest-link intuition. The last score alone would ignore errors in earlier steps.

The paper's released models (Qwen2.5-Math-PRM-7B/72B) use the product scoring strategy as default (Table 6), consistent with their training on consensus-filtered data that inherits LLM-as-a-judge's process-oriented labeling.

---

Dual Evaluation Framework

The framework's structure. The paper advocates for — and implements — a two-dimensional evaluation framework for PRMs (Section 4.2):

1. Response-level BoN evaluation: Measures whether the PRM can select responses whose final answers are correct. This is a downstream utility metric — given $N$ candidate solutions, can the PRM pick one that gets the right answer?

2. Step-level PROCESSBENCH evaluation: Measures whether the PRM can identify specific erroneous reasoning steps within a solution. This is a fidelity metric — does the PRM actually do the thing it's designed to do?

The paper's thesis is that both dimensions are necessary and that evaluating only BoN leads to systematically misleading conclusions about PRM quality.

Best-of-N (BoN) evaluation — detailed protocol. The BoN evaluation follows the standard protocol from prior work (Lightman et al., 2023; Wang et al., 2024b) with specific choices documented in Section 2.2 and Section 4.2:

1. Policy model sampling. For a given math query, the policy model Qwen2.5-Math-7B-Instruct (or 72B-Instruct for supplementary experiments) generates $N = 8$ candidate solutions. The paper does not specify sampling temperature, but standard practice would be $T > 0$ to ensure diversity.

2. Step splitting. Each candidate solution is split into steps using the delimiter "\n\n", matching the training data's step segmentation.

3. Step scoring. The PRM processes each step and outputs a scalar $\hat{y}_i$ for step $i$, representing the predicted probability that step $i$ is correct.

4. Solution scoring. Step scores are aggregated into a solution-level score. The default aggregation for the final models is the product of step scores (Section 4.2):

$\text{score}_{\text{solution}} = \prod_{i=1}^{n} \hat{y}_i$

This means if any single step has a low score, the overall solution score is pulled down — a correct chain requires all steps to be correct.

5. Response selection. The PRM selects the solution with the highest aggregate score among the $N$ candidates.

6. Answer verification. The final answer from the selected solution is extracted and compared against the ground-truth answer using standard answer parsing and matching procedures (which vary by benchmark — e.g., MATH uses the Hendrycks et al. grading function, GSM8K uses numeric matching).

7. Metric computation. prm@N is the proportion of test queries for which the PRM-selected solution's final answer is correct. This is compared against:

   • pass@N: the proportion of queries where any of the N candidates has a correct answer (upper bound — the best one could do with a perfect selector).
   • maj@N: the proportion of queries where the majority vote among the N candidates' final answers is correct (a strong baseline using only answer-level consensus, no process verification).

Benchmarks for BoN. The paper evaluates BoN across seven mathematical reasoning benchmarks (Section 4.2): GSM8K, MATH, Minerva Math, GaoKao 2023 En, OlympiadBench, College Math, and MMLU STEM. These span a range of difficulties from grade-school arithmetic (GSM8K) to competition-level problems (OlympiadBench).

PROCESSBENCH evaluation — detailed protocol. PROCESSBENCH (Zheng et al., 2024) tests a PRM's ability to locate the first erroneous step in a reasoning solution or correctly determine that all steps are correct. The evaluation protocol (Section 2.2) is:

1. Test data. PROCESSBENCH provides math problems with annotated solutions where each step is labeled as correct or incorrect, and the first erroneous step (if any) is identified by human annotators. The paper evaluates on four sub-categories: GSM8K, MATH, OlympiadBench, and Omni-MATH.

2. PRM scoring. For each solution, the PRM scores every step, producing a sequence $\hat{y}_1, \hat{y}_2, ..., \hat{y}_n$.

3. Error localization. The PRM identifies the first step where the predicted score falls below a threshold (or uses the minimum score across steps as a signal). The paper states:

"Following the evaluation methods for PRMs in PROCESSBENCH, we locate the first erroneous step from predict scores yielded by PRMs."

The exact threshold or detection method is not specified in detail — the reference to PROCESSBENCH's evaluation methodology implies following their standard protocol, which uses the step with the lowest score as the predicted error location.

4. Metrics. PROCESSBENCH reports per-category metrics:
   • error (recall for error steps): How often the PRM correctly identifies the ground-truth erroneous step.
   • correct (recall for all-correct solutions): How often the PRM correctly determines that all steps are correct.
   • F1: Harmonic mean of error detection and correct classification performance.
   • Avg. F1: Macro-average of F1 scores across the four sub-categories.

The BoN-PROCESSBENCH tension — what it reveals. The paper's key empirical finding in Section 3.2 is that PRMs trained on different data sources exhibit opposite performance orderings on BoN vs. PROCESSBENCH (Figure 7):

This inverse ranking is not coincidental — it arises from the systematic differences in what each data source teaches the PRM:

• MC estimation teaches the PRM to assign high scores to steps from which correct answers are reachable, regardless of process quality. This makes it good at BoN (selecting responses with correct final answers) but bad at identifying process errors (it overlooks flawed reasoning that happens to lead to correct answers).
• Human annotation teaches the PRM to assign high scores only to genuinely correct steps. This makes it good at process error identification, but it may assign low scores to correct-answer solutions with minor process issues — which hurts BoN performance because those solutions are penalized even though they have the right answer.

The process-to-outcome shift — quantitative evidence. Section 3.2.3 presents Figure 8, which measures the percentage of BoN-evaluated responses where the minimum step score occurs at the final answer step. If a PRM is genuinely evaluating the reasoning process, the minimum score should be distributed across all steps roughly uniformly (or at least not concentrated at the end). If the minimum score is consistently at the final step, it suggests the PRM is primarily judging the final answer, not the intermediate reasoning.

The results across 11 PRMs (Figure 8) show a wide range:

The interpretation: models with low proportions (like the paper's Qwen2.5-Math-PRM-7B at 17.5%) are more genuinely evaluating intermediate steps. Models with high proportions (EurusPRM at 52.9–54.6%, Skywork-PRM-7B at 42.2%) have effectively degenerated into ORMs — their selection decisions are driven primarily by the final step, which typically contains the answer. The paper states:

"This analysis reveals that some PRMs' performance in BoN evaluation is predominantly determined by final answer scores rather than intermediate reasoning steps, indicating a model degradation from process-based to outcome-oriented assessment."

Why BoN evaluation alone is insufficient — the logical argument. The paper builds a chain of reasoning in Section 3.2 that explains the mechanism by which BoN-only evaluation creates perverse incentives:

1. Policy models are unreliable (Section 3.2.1, Figure 6): When you sample solutions from Qwen2.5-Math-7B-Instruct, a substantial fraction of correct-answer solutions contain process errors. The proportion increases with problem difficulty — from 5.1% on GSM8K to 43.4% on Omni-MATH. This means that among the N candidate solutions, there are often several with the right answer but wrong reasoning.

2. PRMs cannot distinguish these cases (Section 3.2.2, Table 5): Most existing PRMs have detection accuracy below 50% on solutions with correct answers but erroneous steps. The paper's extracted PROCESSBENCH subset (Table 5) shows that even strong models like Skywork-PRM-7B only detect errors in 27.8% of such cases.

3. BoN does not penalize this failure (Section 3.2.2): If a PRM assigns high scores to correct-answer-but-flawed-process solutions (because it's essentially judging the answer), BoN will show good performance — the PRM selects a solution with the right answer, and BoN only checks the answer. The PRM looks effective, but it's not doing its job.

4. Optimization toward BoN reinforces the problem (Section 3.2.3): When PRMs are developed and improved using BoN as the sole metric, the optimization pressure pushes them toward outcome-based assessment — because that's what BoN rewards. Over time, PRMs "drift" toward ORM-like behavior, as evidenced by the minimum-score concentration at the final step.

The paper's conclusion (Section 3.2.5):

"we argue that supplementary step-level evaluation plays a crucial role in PRM evaluation."

---

PRM-Guided Greedy Search (Appendices A and B)

Motivation and setup. The paper also explores using the trained PRM not just for post-hoc selection (BoN) but for active guidance during solution generation (Appendix A). The setup is:

1. Policy model: Qwen2.5-7B-Instruct (chosen for greater generation diversity), sampling at temperature 1.0, top-p 1.0.
2. At each step: The policy model generates $N = 8$ candidate next steps. The PRM scores each candidate. The highest-scoring candidate is selected, and generation continues from there.
3. Comparison baseline: ORM Best-of-8 (Qwen2.5-Math-RM-72B), which scores 8 complete independently-generated solutions and selects the highest-scored one.

Results (Table 8). The PRM-guided greedy search with Qwen2.5-Math-PRM-72B achieves an average accuracy of 66.6% across seven benchmarks, slightly outperforming ORM Best-of-8 (66.5%). However, Qwen2.5-Math-PRM-7B with greedy search (63.5%) underperforms ORM Best-of-8.

Why the performance differential is small. The paper attributes the modest gap between PRM search and ORM BoN to two factors:

1. Token count consistency: Greedy search generates a single solution path, while BoN samples 8 complete solutions. The total generation budget differs in ways not fully controlled.
2. Local optimality vs. global optimality: Greedy search selects the locally best step at each decision point, but the step that looks best locally may not lead to the globally best solution:

"the highest-scoring step may not be the correct one. Therefore, implementing either Depth-First Search (DFS) with backtracking capabilities or search approaches incorporating score constraints could prove more suitable for this cases."

The deeper point — PRMs vs. value models for search. The paper uses this experiment to reinforce its conceptual distinction between PRMs and value models (Section 3.1.1):

"value models can predict the future probability of reaching the correct answer, rather than reflecting the correctness of the current step like rewards do, making them particularly well-suited for integration with search strategies."

A PRM tells you "is this step correct?" — which is what you want for verification and error identification. A value model tells you "from this step, what's the probability of eventually succeeding?" — which is what you want for search, because it enables lookahead planning. Using a PRM for greedy search is possible but likely suboptimal because the PRM's scores don't account for future difficulty — a step that is perfectly correct might lead to an intractable subproblem, and a PRM would still score it highly.

The paper concludes Appendix A with a direction for future work:

"there is still significant potential for exploration in the future regarding more appropriate search strategies or combining rewards and values to simultaneously consider both the correctness of the current step and the possibility of reaching the correct future outcomes."

---

Summary of Design Choices and Their Justifications

• MC estimation with hard labels (threshold 0): Hard labels avoid the noise of soft probability estimates; threshold 0 minimizes false negatives from imperfect completions; together they acknowledge MC estimation's fundamental unreliability for per-step correctness while preserving its utility as a coarse signal.

• LLM-as-a-judge for complementary annotation: Direct step-by-step logical analysis avoids MC estimation's confound between process quality and outcome reachability; the judge's errors are different in kind from MC's errors, enabling effective consensus filtering.

• Consensus filtering (intersection of MC and LLM labels): Removes instances where the two independent signals disagree, exploiting their complementarity to produce a cleaner training set at ~40% of the original size; works because the methods have weakly correlated error modes.

• Scalar head (two linear layers) replacing language model head: Standard architecture for reward models; extracts a single probability from the rich transformer hidden state at each step boundary.

• Cross-entropy loss on last tokens of steps only: Matches the binary classification framing (correct/incorrect); restricting to step-final tokens avoids diluting the training signal across non-boundary tokens; CE is the proper objective for probability estimation.

• Product scoring for solution-level aggregation in final models: Appropriate for PRMs trained on process-oriented labels (consensus-filtered data inheriting LLM-as-a-judge's process verification); captures the logical requirement that all steps must be correct.

• Dual evaluation (BoN + PROCESSBENCH): BoN measures downstream answer-selection utility; PROCESSBENCH measures process-fidelity; together they prevent the "process-to-outcome shift" that occurs when PRMs are optimized solely for BoN.

• Initialization from Qwen2.5-Math-Instruct rather than base Qwen2.5: Leverages existing mathematical reasoning capabilities from instruction tuning; the model already understands mathematical language and reasoning patterns before being fine-tuned for verification.

• Step truncation after first error in training data: Prevents the PRM from learning to evaluate steps that follow errors; these downstream steps are built on faulty premises and training on them would confuse the model about what constitutes correct reasoning.

• No human annotation used in the final model: The consensus filtering mechanism achieves PROCESSBENCH performance approaching human-annotation levels (46.3 vs. 56.5 F1 on the 7B scale with filtered data) while being fully automated and scalable — a practical compromise between quality and cost.

4. Key Insights and Innovations

Innovation 1: Reframing MC Estimation as a Category Error — PRMs Trained This Way Learn to Be Value Models, Not Process Verifiers

The paper's most intellectually distinctive contribution is not discovering that MC estimation underperforms, but rather diagnosing precisely why it underperforms in terms that reconfigure the problem from "noisy data" to "wrong objective." Prior to this work, the implicit framing of MC estimation was that it provides a noisy but unbiased estimate of step correctness — the noise could be reduced by more completions, better completion models, or larger datasets. The paper demonstrates that this framing is categorically wrong.

The core reframing appears in Section 3.1.1:

"The key difference between PRM and value model lies in that PRMs function as deterministic evaluators of current step correctness, while value models operate as predictive estimators of future solution potential."

This distinction — drawn from reinforcement learning, where reward $R(s)$ depends on the current state and value $V(s)$ depends on expected future returns — had not previously been applied to PRM data construction. The field had been treating MC estimation as an approximation to human annotation (a cheaper, noisier way to get step-correctness labels), when in fact it estimates an entirely different quantity. This is a fundamental conceptual advance, not an incremental empirical finding.

What changes intellectually. Before this paper, one could reasonably believe that scaling MC estimation (more completions, more data) would eventually match human annotation quality. After this paper, that hypothesis is falsified: no amount of scaling can make $V(s)$ equal $R(s)$ when the completion model can reach correct answers from incorrect steps. The finding in Figure 7 — where MC-trained PRMs rank #1 on BoN but #3 on PROCESSBENCH, and human-annotation-trained PRMs show the exact opposite ranking — is the smoking gun. It demonstrates that the two data sources are teaching the model fundamentally different functions, not the same function with different noise levels. Human annotation teaches "is this step logically sound?" MC estimation teaches "can you reach the right answer from here?" These are correlated but not identical, and optimizing for the latter produces a model that is good at outcome prediction (hence strong BoN) but poor at process verification (hence weak PROCESSBENCH).

Relationship to prior work. The reinforcement learning distinction between rewards and values is textbook. But prior PRM literature — Math-Shepherd (Wang et al., 2024b), OmegaPRM (Luo et al., 2024), the RLHFlow models (Xiong et al., 2024) — used MC estimation without explicitly grappling with this distinction. Math-Shepherd called its models "process reward models" while training them on a value-like signal. The paper's contribution is to name this mismatch and demonstrate its empirical consequences in a controlled comparison (Tables 2, 3, 4; Figure 7), transforming an implicit confusion into an explicit diagnostic framework that future work can use.

Significance beyond performance. Even if a future data construction method surpasses consensus filtering, this reframing will persist as a conceptual tool. It provides a vocabulary and a test for any proposed data generation method: does this method estimate $R(s)$ or $V(s)$? If the latter, it will likely produce models that perform well on outcome-based metrics and poorly on process-based ones. This is a durable intellectual contribution that will shape how researchers think about PRM data generation going forward.

---

Innovation 2: The Consensus Filtering Mechanism — Triangulation via Complementary Error Modes as a General Data Cleaning Strategy

At first glance, consensus filtering looks like a simple ensembling trick: take two annotators, keep where they agree, discard where they disagree, and you get cleaner data. But the paper's implementation is more interesting than that summary suggests, because the two annotators are not just different instances of the same method — they estimate different underlying quantities with fundamentally different failure modes.

MC estimation's primary failure mode is false positives: it labels incorrect steps as correct when the completion model "repairs" the error downstream. LLM-as-a-judge's primary failure mode is different — the judge model may make errors in mathematical analysis, particularly for complex derivations, but these errors are not systematically correlated with whether the completion model can reach the answer. The two methods use non-overlapping information sources: MC estimation looks forward to outcomes, LLM-as-a-judge looks at step-by-step logic.

The conceptual move. Rather than treating this difference as a problem to be solved (by making one method more like the other), the paper treats it as a feature to be exploited. When two methods with uncorrelated error modes agree on an error location, the probability that both are simultaneously wrong is lower than the probability that either alone is wrong. This is a form of triangulation — using two independent lines of evidence that converge on the same conclusion — and it yields cleaner labels than either method provides individually.

What makes this distinctive. The paper's consensus filtering is not generic ensembling. It is specifically cross-method consensus between methods that estimate different quantities, which is a stronger condition than intra-method consensus (e.g., multiple LLM-as-a-judge runs with different prompts). The agreement rate of ~40–50% (Section 3.1.3) shows that the two methods disagree on roughly half of all step labels — this is a high disagreement rate that signals genuine complementary information, not just correlated noise.

Evidence and significance. The filtered 350K dataset achieves 46.3 PROCESSBENCH F1, matching LLM-as-a-judge's 46.5 on the full 860K dataset (Figure 2), while slightly improving BoN (65.7 vs. 65.9 for pure MC estimation). On the 3M dataset, the 1.5M surviving samples achieve 66.5 PB F1 vs. 40.2 for the unfiltered 3M (Figure 4) — a 65% improvement in process verification at half the data volume. This is quality dominating quantity in a regime where standard data scaling would predict the opposite.

This is a fundamental insight, not an incremental refinement, because it demonstrates that for PRM training, data quality improvements from cross-method filtering are not interchangeable with data quantity increases. Doubling MC-estimated data from 860K to 3M produced only marginal PROCESSBENCH gains (40.1 → 40.2 F1, Table 4 vs. Figure 4), while filtering to 1.5M samples produced a 65% jump. This is the opposite of what standard ML scaling intuition would predict, and it implies that the PRM data construction problem is bottlenecked by label noise in a way that cannot be brute-forced with more data.

---

Innovation 3: The Dual-Evaluation Critique — Exposing the BoN-PROCESSBENCH Inverse Relationship as Proof That BoN Alone Is Actively Misleading

The paper's most provocative empirical finding is the inverse relationship between BoN and PROCESSBENCH performance across data construction methods (Figure 7, Section 3.2.2): MC-trained PRMs achieve the best BoN but worst PROCESSBENCH; human-annotation-trained PRMs achieve the best PROCESSBENCH but worst BoN. This is not just a failure of correlation — it is a negative correlation that means optimizing for BoN pushes PRMs away from their stated purpose of process verification.

The intellectual contribution is the diagnosis of why this happens, not just the observation that it does. The paper constructs a three-step causal chain (Section 3.2) that was not previously articulated in the literature:

1. Policy models produce solutions with correct answers but flawed processes (Section 3.2.1, Figure 6). The proportion grows with problem difficulty: 5.1% on GSM8K, 43.4% on Omni-MATH. This means that among candidate solutions in a BoN setup, there are systematically responses that will appear correct to an outcome-based evaluator but contain process errors. The paper's manual annotation confirming this is an important empirical contribution in its own right — prior work had not quantified how prevalent this phenomenon is across difficulty levels.

2. Existing PRMs cannot detect these cases (Section 3.2.2, Table 5). The paper extracts the subset of PROCESSBENCH where answers are correct but processes are erroneous, and finds that most open-source PRMs have detection accuracy below 30%. Even the strongest pre-existing model (Skywork-PRM-7B) detects only 27.8%. This means PRMs are systematically blind to the very process errors they are supposed to catch, specifically in the cases where outcome-based signals would fail.

3. Optimizing for BoN reinforces the blindness (Section 3.2.3, Figure 8). The "process-to-outcome shift" — where PRMs assign their minimum step scores to the final answer step — is quantified across 11 models. Models with high BoN scores tend to show this shift (EurusPRM-Stage1 at 54.6%, Skywork-PRM-7B at 42.2%), while models with strong PROCESSBENCH scores show less of it (Qwen2.5-Math-PRM-72B at 18.0%). This provides a mechanistic explanation for the BoN-PROCESSBENCH inverse relationship: BoN optimization selects for models that behave more like ORMs, because that's what BoN measures.

Why this is fundamental, not incremental. Prior work treated BoN as a noisy but directionally correct signal of PRM quality. The paper demonstrates that BoN is not just noisy — it is actively misleading in the specific sense that it ranks models in the opposite order from their process verification capability. This is a stronger claim than "BoN has limitations." It means that any PRM development pipeline that uses BoN as the sole or primary evaluation metric will systematically favor models that have learned to ignore process quality, regardless of the researchers' intentions. This is reminiscent of Goodhart's Law ("when a measure becomes a target, it ceases to be a good measure") but applied specifically to PRM evaluation, with rigorous empirical evidence for the mechanism.

Practical implications. The paper's advocacy for dual evaluation (BoN + PROCESSBENCH) is not a minor methodological suggestion — it follows necessarily from the finding that BoN and process verification are in tension. A PRM developer who only looks at BoN will make the wrong decisions about data construction (favoring MC estimation), thresholding, and model selection. The paper establishes PROCESSBENCH as a necessary supplement, not an optional extra, and its release of strong PRMs that score well on both metrics (Table 6: 67.6 BoN, Table 7: 73.5 PB F1 for the 7B model) demonstrates that the tradeoff is not inevitable — good process verification and good answer selection can be achieved simultaneously with the right training data.

---

Innovation 4: The Scoring Strategy as a Diagnostic for Training Signal Type — "Last" Score Works Better for Value-Trained Models, "Product" for Reward-Trained Ones

At first glance, the question of how to aggregate per-step scores into a solution-level score appears to be a minor implementation detail. The paper elevates it to a diagnostic tool that reveals what kind of function the PRM has actually learned — a value function (predicting future outcomes) or a reward function (judging current-step correctness).

The finding in Section 3.2.4 and Figure 9 is that the optimal scoring strategy depends systematically on the training data source:

• For MC-estimation-trained PRMs, the last step score alone is the best aggregator. This makes sense if the model learned a value function: $V(s_n)$ at the final step already integrates the model's assessment of whether the solution reaches the correct answer, and earlier-step scores are redundant or noisy.
• For LLM-as-a-judge and human-annotation-trained PRMs, product and minimum outperform last-score. This makes sense if the model learned a reward function: each step score is an independent assessment of that step's correctness, and aggregating them via product (all must be correct) or minimum (weakest link matters most) captures the logical structure of multi-step reasoning.

Why this is conceptually significant. This pattern provides a post-hoc diagnostic for any trained PRM: test all three aggregation strategies on BoN. If last-score outperforms product/minimum, the model likely learned a value-like function regardless of what it was trained to do — a sign that the training data was outcome-contaminated. If product/minimum outperforms last, the model likely learned a reward-like function. This diagnostic could be used to evaluate whether a proposed data construction method actually produces process-oriented labels, without needing a step-level benchmark like PROCESSBENCH.

Relationship to the consensus filtering logic. The paper's final models (Qwen2.5-Math-PRM-7B/72B) use product scoring, consistent with their training on consensus-filtered data that inherits LLM-as-a-judge's process-oriented labeling. This is a deliberate choice that aligns the inference-time scoring with the model's training objective — a detail that prior PRM work often overlooked because the distinction between value and reward training wasn't clearly drawn.

Evidence. Figure 9 shows multiple PRMs evaluated under all three strategies. The pattern is consistent: MC-trained models favor last-score, human-annotation models favor product/minimum. The paper's own models (not shown in Figure 9 but in Appendix B, Tables 13–14) exhibit product/minimum superiority, confirming that the consensus filtering successfully shifted their training signal from value-like to reward-like. This is a conceptual contribution rather than a performance gain — it provides a vocabulary and a test for understanding what a PRM has actually learned, which will be useful for future PRM development even if the specific data construction methods change.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses multiple mathematical reasoning benchmarks for BoN evaluation: GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), Minerva Math (Lewkowycz et al., 2022), GaoKao 2023 En (Liao et al., 2024), OlympiadBench (He et al., 2024), College Math (Tang et al., 2024), and MMLU STEM (Hendrycks et al., 2021a). For step-level evaluation, PROCESSBENCH (Zheng et al., 2024) is used across its four sub-categories: GSM8K, MATH, OlympiadBench, and Omni-MATH error identification tasks. Chinese benchmarks (CMATH, CN Middle School 24, GaoKao Math Cloze and QA) appear in Appendix B. The PRM training data uses approximately 500,000 queries with golden answers, with responses generated from Qwen2-Math-Instruct and Qwen2.5-Math-Instruct series models.

• Base model(s). All PRMs are initialized from Qwen2.5-Math-Instruct models (Yang et al., 2024c) at 7B and 72B parameter scales. The policy model for primary Best-of-8 evaluation is Qwen2.5-Math-7B-Instruct, with supplementary results on Qwen2.5-Math-72B-Instruct in Appendix B. The choice is motivated by these models being "representative of the capabilities of many contemporary LLMs" (stated rationale in Section 4), with strong but imperfect mathematical reasoning that leaves room for PRM-based selection to demonstrate value.

• Metrics. Two orthogonal metrics are used: (1) prm@N (Best-of-N accuracy) — the proportion of test queries where the PRM-selected solution's final answer matches the ground truth, computed across N = 8 (default) or N = 64 (Appendix B.4) candidate solutions. This is compared against pass@N (upper bound — any candidate correct) and maj@N (majority voting baseline). (2) PROCESSBENCH F1 — the harmonic mean of error-step detection recall (correctly identifying the first erroneous step) and correct-classification recall (correctly identifying solutions with no errors), reported as a macro-average across the four PROCESSBENCH sub-categories.

• Baselines. The paper compares against an extensive set of open-source PRMs: Math-Shepherd-PRM-7B (Wang et al., 2024b), RLHFlow-PRM-Mistral-8B and RLHFlow-PRM-Deepseek-8B (Xiong et al., 2024), Skywork-PRM-1.5B and Skywork-PRM-7B (Skywork, 2024), EurusPRM-Stage1 and EurusPRM-Stage2 (Cui et al., 2025), plus two in-house baselines Qwen2.5-Math-7B-Math-Shepherd and Qwen2.5-Math-7B-PRM800K trained on the respective opensource datasets. For PROCESSBENCH, additional LLM-as-a-judge baselines include GPT-4o-0806 (Hurst et al., 2024), o1-mini (OpenAI, 2024), Llama-3.3-70B-Instruct (Dubey et al., 2024), Qwen2.5-Math-72B-Instruct, Qwen2.5-72B-Instruct (Yang et al., 2024b), and QwQ-32B-Preview (Qwen, 2024). The ORM baseline for BoN is Qwen2.5-Math-RM-72B (Yang et al., 2024c).

• Generation budget / compute accounting. For BoN evaluation, the budget is N = 8 candidate solutions generated from the policy model; each solution is scored by the PRM at the per-step level, with solution-level scores computed as the product of step scores (default for the final models). For MC estimation data construction, 8 completions are generated per reasoning step to estimate correctness probabilities. For PRM-guided greedy search (Appendix A), 8 candidate steps are generated and scored at each decision point, with the highest-scored candidate selected. No FLOPs-based accounting is provided — all comparisons use generation counts as the compute proxy. The paper does not account for the cost of difficulty estimation (relevant since the LLM-as-a-judge step uses a 72B model to annotate training data).

• Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the referenced example paper). The training data is constructed independently of the evaluation benchmarks. For PROCESSBENCH, the authors follow the standard evaluation methodology from Zheng et al. (2024), locating the first erroneous step from PRM-predicted scores and computing per-category metrics. The 7 evaluation benchmarks for BoN are treated as independent test sets with no reported statistical significance testing or confidence intervals.

Main Quantitative Results

Preliminary Trials: MC Estimation Underperforms Human Annotation on Process Verification

The paper's initial experiments (Section 2.3) establish the core problem. Table 1 shows that on Best-of-8 evaluation with Qwen2.5-Math-7B-Instruct as the policy model, none of the MC-estimation-trained PRMs surpass maj@8 (66.2 average): Qwen2.5-Math-7B-PRM-MC-hard achieves 65.5, Qwen2.5-Math-7B-PRM-MC-soft achieves 64.4, and the human-annotation-trained Qwen2.5-Math-7B-PRM800K achieves 64.9. All three PRMs cluster below the 66.2 majority voting baseline, meaning the PRMs are not adding value over simple answer-level consensus for response selection.

The divergence becomes stark on PROCESSBENCH (Table 2). Qwen2.5-Math-7B-PRM800K achieves 56.5 average F1 across the four sub-categories, while Qwen2.5-Math-7B-PRM-MC-hard and MC-soft achieve only 40.2 each. The per-category breakdown reveals where MC estimation fails most dramatically: on OlympiadBench, MC-hard scores 17.9 F1 vs. PRM800K's 50.7; on Omni-MATH, MC-hard scores 20.2 vs. PRM800K's 44.3. Notably, MC-trained models actually outperform PRM800K on GSM8K error detection (77.0 and 77.1 F1 vs. 68.2), suggesting that MC estimation's failure mode is specifically on harder problems where the completion model's ability to "correct" errors creates more false positives.

The data scale asymmetry is important: the MC-trained models used 860K samples while PRM800K used only ~264K (after deduplication), yet the latter dramatically outperforms on process verification.

Data Construction Method Comparison: The BoN-PROCESSBENCH Inverse Relationship

Tables 3 and 4 (Section 3.1.2) present the paper's central comparative experiment, training PRMs on four data sources with the same base model and architecture:

The key result is the inverse ranking: MC estimation (authors') achieves the highest BoN (65.9) but second-worst PROCESSBENCH (40.1); human annotation achieves the lowest BoN (64.9) but the highest PROCESSBENCH (56.5) with the least data. LLM-as-a-judge sits in the middle on both metrics (65.3 BoN, 46.5 PB).

Table 4 provides per-category PROCESSBENCH detail. Human annotation's generalization advantage is clearest on the hardest tasks: OlympiadBench F1 of 50.7 (vs. 19.4 for MC estimation, authors' data) and Omni-MATH F1 of 44.3 (vs. 19.8). LLM-as-a-judge also generalizes better than MC estimation on these harder tasks (OlympiadBench 39.4, Omni-MATH 36.1), despite MC estimation's slight edge on GSM8K (74.0 vs. 60.9). This pattern — MC estimation performing competitively on easy problems but collapsing on hard ones — is consistent with the diagnosis that MC estimation misses process errors that the completion model can "repair," and such repairable errors are more common in simpler problems.

Consensus Filtering: Quality Over Quantity

Figure 2 (Section 3.1.3) shows the consensus filtering result on the 860K dataset. Starting from MC estimation (BoN 65.9, PB 40.1) and LLM-as-a-judge (BoN 65.3, PB 46.5), the consensus-filtered 350K dataset achieves BoN 65.7 and PB 46.3 — matching LLM-as-a-judge's PROCESSBENCH performance while using only ~40% of the data, and preserving (actually slightly improving) BoN relative to pure MC estimation.

The effect is more dramatic at larger scale (Section 3.1.4, Figures 3 and 4). On the 3M-sample MC-estimated dataset:

• Before filtering: hard labels achieve BoN 65.4, PB 40.2; soft labels achieve BoN 65.4, PB 40.2
• After filtering (1.5M samples): hard labels achieve BoN 67.2, PB 66.5; soft labels achieve BoN 65.4, PB 49.3

Several patterns emerge: (1) Unfiltered data shows no performance difference between hard and soft labels — the noise dominates any signal difference. (2) Filtering reveals that hard labels dramatically outperform soft labels on PROCESSBENCH (66.5 vs. 49.3), confirming that deterministic correctness labels are better aligned with the PRM objective than probabilistic value estimates. (3) The PB improvement from filtering is massive (40.2 → 66.5 for hard labels), representing a 65% relative improvement with half the data.

Hard Label Threshold Selection

Figure 5 (Section 3.1.4) sweeps the threshold for converting MC estimation's 8-completion results into hard labels, from 1/8 (step positive if ≥1 completion succeeds) to 7/8 (step positive only if ≥7 completions succeed). The optimal threshold is unambiguously 0 (equivalent to 1/8 in practice: positive if any completion succeeds):

Performance degrades monotonically as the threshold increases, with the drop in PROCESSBENCH being particularly severe (40.2 → 29.5 from threshold 0 to 7/8). This supports the diagnosis that MC estimation's primary failure mode is false negatives (correct steps where completions fail), and that being maximally permissive about labeling steps as correct minimizes this noise.

Best-of-8 Evaluation of Final Models

Table 6 presents the main BoN results with Qwen2.5-Math-7B-Instruct as the policy model. The headline numbers:

• Qwen2.5-Math-PRM-7B achieves 67.6 average accuracy across 7 benchmarks, surpassing maj@8 (66.2) by 1.4 percentage points. It outperforms all other 7B-scale PRMs, with the next best being Skywork-PRM-7B at 64.8 and RLHFlow-PRM-Deepseek-8B at 64.9.
• Qwen2.5-Math-PRM-72B achieves 69.3 average, slightly outperforming the ORM baseline Qwen2.5-Math-RM-72B (68.9). The advantage over the ORM is most pronounced on Minerva Math (46.0 vs. 42.6) and MMLU STEM (81.1 vs. 78.7).
• The pass@8 upper bound is 74.7, leaving a ~7 point gap that represents the maximum possible improvement from better response selection.

Per-benchmark patterns reveal where PRMs add value over majority voting. The 7B PRM outperforms maj@8 on MATH (88.0 vs. 87.1), Minerva Math (42.6 vs. 41.2), GaoKao 2023 En (74.5 vs. 72.5), OlympiadBench (47.6 vs. 44.4), College Math (48.7 vs. 47.8), and MMLU STEM (74.5 vs. 73.8) — it only ties on GSM8K (97.1 vs. 96.7).

The 72B PRM shows larger margins: it outperforms maj@8 on MATH (88.7 vs. 87.1, +1.6), Minerva Math (46.0 vs. 41.2, +4.8), MMLU STEM (81.1 vs. 73.8, +7.3), and OlympiadBench (48.1 vs. 44.4, +3.7). The MMLU STEM improvement is particularly large, suggesting that for knowledge-intensive STEM problems, process-level verification provides substantial gains over answer-level consensus.

Table 9 (Appendix B.1) shows BoN results with the larger policy model Qwen2.5-Math-72B-Instruct. The 7B PRM achieves 69.9 average, which is below maj@8 (70.2) — the 7B verifier cannot reliably supervise a 72B generator's outputs. The 72B PRM achieves 70.7, roughly matching the ORM (70.8) and slightly above maj@8 (70.2). The performance ceiling is higher here (pass@8 = 77.5), and the gap to PRM selection is wider (~7 points), suggesting that supervising stronger policy models is inherently harder.

Appendix B.4, Table 10 shows Best-of-64 results on MATH500, AIME24, AMC23, Minerva Math, GaoKao 2023 En, and OlympiadBench. Qwen2.5-Math-PRM-7B achieves 56.1 average vs. the best competing 7B-scale PRM (Qwen2.5-Math-7B-PRM800K at 55.2). The 72B PRM achieves 59.2, which is below the ORM baseline Qwen2.5-Math-RM-72B (60.5). The standout result is on MATH500: the 72B PRM achieves 87.8, matching the 7B PRM's performance on this subset and substantially outperforming all other PRMs (next best: Skywork-PRM-7B at 84.6). However, on competition-level problems (AIME24), the 72B PRM scores only 23.3 vs. the ORM's 36.7, suggesting that outcome-based verification remains stronger for the hardest problems.

PROCESSBENCH Evaluation of Final Models

Table 7 presents the step-level error identification results. The headline numbers:

• Qwen2.5-Math-PRM-7B achieves 73.5 average F1 across the four PROCESSBENCH sub-categories.
• Qwen2.5-Math-PRM-72B achieves 78.3 average F1.

These results place the models in context:

• vs. open-source PRMs: The 7B model's 73.5 F1 dwarfs the next best 7B-scale model (Qwen2.5-Math-7B-PRM800K at 56.5, a 17-point gap). All other open-source PRMs score below 43 F1. The performance gap is largest on the hardest categories: on OlympiadBench, the paper's 7B model scores 67.5 vs. 50.7 for PRM800K; on Omni-MATH, 66.3 vs. 44.3.
• vs. LLM-as-a-judge baselines: The 7B PRM (73.5) substantially outperforms all open-source language models used as judges, including QwQ-32B-Preview (71.5), Qwen2.5-72B-Instruct (61.2), and Llama-3.3-70B-Instruct (58.0). This is notable because the PRM is a 7B model outperforming 70B+ general-purpose models at mathematical error detection.
• vs. proprietary models: The 7B PRM (73.5) outperforms GPT-4o-0806 (61.9) but trails o1-mini (87.9) by a substantial margin. The 72B PRM (78.3) narrows this gap but still trails o1-mini by 9.6 points.
• vs. ORM: The ORM Qwen2.5-Math-RM-72B achieves only 38.9 F1, confirming that outcome-level training produces poor process-level verification despite strong BoN performance. This is a key validation of the paper's thesis.

Per-category analysis reveals the 72B PRM's strengths: on GSM8K, it achieves 87.3 F1 (error detection recall 78.7, correct classification recall 97.9) — near-perfect at identifying when solutions are correct, good but imperfect at identifying errors. On MATH, 80.6 F1 (74.2 error recall, 88.2 correct recall). The error detection recall drops on harder tasks (OlympiadBench: 67.9, Omni-MATH: 64.8), indicating that even the strongest PRM still misses a significant fraction of process errors on competition-level problems.

The BoN-PROCESSBENCH Disconnect Across All Evaluated PRMs

Figure 7 (Section 3.2.2) visualizes the disconnect by extracting from PROCESSBENCH only the cases where answers are correct but processes are erroneous, and measuring detection accuracy. The results reveal that:

• The MC-trained model (authors' data) achieves the highest BoN (65.9) but detects errors in only 22.2% of these extracted cases.
• The human-annotation-trained PRM800K model achieves the lowest BoN (64.9) but detects errors in 38.2% of cases.
• The LLM-as-a-judge model sits between (65.3 BoN, 26.2% detection).

Table 5 expands this analysis across all evaluated PRMs on the extracted PROCESSBENCH subset. The results are sobering: only the paper's 7B model (53.9%) and 72B model (58.1%) achieve above-50% detection accuracy. All other PRMs — including strong BoN performers like Skywork-PRM-7B (27.8%) and EurusPRM-Stage2 (27.4%) — detect fewer than 30% of process errors when the final answer is correct. The sample sizes vary by benchmark (GSM8K has only 7 such cases, while Omni-MATH has 259), but the pattern is consistent across difficulty levels.

PRM-Guided Greedy Search

Table 8 (Appendix A) evaluates using the PRM for step-by-step greedy search rather than post-hoc BoN selection, with Qwen2.5-7B-Instruct as the policy model. Qwen2.5-Math-PRM-72B with greedy search@8 achieves 66.6 average accuracy, marginally outperforming ORM Best-of-8 (66.5). The 7B PRM with greedy search achieves only 63.5, trailing the ORM baseline. The paper notes that greedy search may be suboptimal for PRMs because "the highest-scoring step may not be the correct one," and suggests that search methods incorporating backtracking or lookahead would be more appropriate — a natural consequence of PRMs estimating current-step correctness rather than future value.

Supplementary Findings

BoN with LLM-as-a-judge (Appendix B.5, Tables 11–12): The paper compares PRM-based BoN against using LLMs directly as pairwise or pointwise judges for response selection. On the 7B policy model, pairwise LLM-as-a-judge with QwQ-32B-Preview achieves 67.6 — matching Qwen2.5-Math-PRM-7B (67.6). On the 72B policy model, pairwise QwQ-32B-Preview achieves 70.3 vs. the 7B PRM's 69.9 and the 72B PRM's 70.7. This suggests that for BoN-style response selection, a capable general-purpose LLM prompted as a pairwise judge can be competitive with a specialized PRM, though the computational cost of N-1 pairwise comparisons per query (vs. N forward passes through the PRM) is substantially higher.

Chinese benchmarks (Appendix B.3, Tables 15–16): On three Chinese math benchmarks with Qwen2.5-Math-7B-Instruct as policy, the 7B PRM achieves 81.3 average vs. maj@8's 79.7, and the 72B PRM achieves 82.6. This provides evidence of cross-lingual generalization, though the PRMs were trained primarily on English math data.

Scoring strategy comparison (Appendix B.2, Tables 13–14): The paper's final models use product scoring. Table 13 confirms that for Qwen2.5-Math-PRM-7B on the 7B policy model, product scoring (67.6) slightly outperforms last (66.1) and minimum (67.1). For the 72B PRM, product (69.3) outperforms last (68.5) and minimum (69.2). The differences are modest, suggesting the final models are relatively robust to scoring strategy — a positive sign that consensus filtering successfully shifted them toward reward-like (process-oriented) behavior where product/minimum are appropriate, away from value-like behavior where last would dominate.

Ablation Studies and Robustness Checks

• Hard vs. soft labels in MC estimation (Section 3.1.4, Figures 3 and 4): Before consensus filtering, hard and soft labels produce identical BoN (65.4) and PB performance (40.2). After filtering, hard labels achieve substantially higher PB (66.5 vs. 49.3) and slightly higher BoN (67.2 vs. 65.4). This is a nuanced finding: the label type only matters once the data is clean enough for the distinction to surface. With noisy data, both label types are equally bad; with clean data, hard labels are clearly superior because deterministic correctness signals align with the PRM's verification objective, while soft value estimates introduce misalignment.

• MC estimation threshold sweep (Section 3.1.4, Figure 5): Performance on both BoN and PROCESSBENCH degrades monotonically as the positive-label threshold increases from 0 to 7/8. The degradation is steeper for PROCESSBENCH (40.2 → 29.5 F1, a 27% relative drop) than for BoN (65.5 → 64.4, a 1.7% relative drop). This reinforces the diagnosis: higher thresholds amplify false negatives (correct steps mislabeled as incorrect), which primarily hurts process verification. The recommended threshold 0 minimizes this damage.

• Scoring strategy by training data source (Section 3.2.4, Figure 9): The optimal aggregation method depends on what the PRM was trained to predict. For MC-trained PRMs (authors' 860K hard-label, 3M hard-label, 3M soft-label, Math-Shepherd), the "last" score strategy consistently outperforms product and minimum because the model learned a value function where the final step's prediction integrates forward-looking information. For human-annotation (PRM800K) and LLM-as-a-judge models, product and minimum outperform last because these models learned per-step reward functions. This ablation serves dual purpose: it's both a robustness check (confirming that scoring strategy choice matters) and a diagnostic tool (revealing what kind of function a PRM has learned).

• Consensus filtering retention rate (Section 3.1.3, Figure 2 accompanying text): Only ~40% of the 860K dataset and ~50% of the 3M dataset survive consensus filtering. The fact that roughly half of all step-level labels show disagreement between MC estimation and LLM-as-a-judge (using different completion/judge models across the two dataset scales) confirms that the two methods have substantially different error modes — they are not just noisy versions of the same signal.

• Data scaling for MC estimation alone (Tables 3–4 vs. Figures 3–4): Pure MC estimation shows diminishing returns from data scaling: going from 440K (Math-Shepherd) to 860K to 3M samples produces only modest PROCESSBENCH gains (28.9 → 40.1 → 40.2 F1). This supports the claim that the bottleneck is label quality, not data quantity — adding more MC-estimated data reinforces the same systematic errors rather than correcting them.

• Cross-model generalization in BoN (Tables 6 vs. Table 9): The 7B PRM successfully supervises a 7B policy model (67.6, above maj@8) but cannot supervise a 72B policy model (69.9, below maj@8's 70.2). The 72B PRM can supervise both (70.7 on 72B policy, 69.3 on 7B policy). This is intuitive — a smaller verifier lacks the capacity to evaluate reasoning from a substantially more capable generator — but it quantifies the scaling requirement for practical deployment.

• Comparison against LLM-as-a-judge for BoN selection (Appendix B.5, Tables 11–12): PRMs are competitive with but do not dramatically outperform LLM-as-a-judge for BoN. On the 7B policy model, the 7B PRM (67.6) matches pairwise QwQ-32B (67.6) and slightly exceeds pointwise Qwen2.5-72B-Instruct (65.5). This is a robustness check showing that the paper's PRMs are at least as good as using a larger general-purpose model for response selection, while being specialized and more computationally efficient at inference time.

• PROCESSBENCH performance vs. data construction cost tradeoff (Tables 3–4, Figure 2): The consensus-filtered 350K dataset achieves 46.3 PB F1, approaching human annotation's 56.5 (on 264K MATH-only data) while being fully automated and scalable to arbitrary domains. This quantifies the quality-cost tradeoff: human annotation provides the highest ceiling but is domain-constrained and expensive; consensus filtering provides strong process verification with full automation and domain flexibility.

• Negative result — ReST-style revision training (not directly tested, but implicit): The paper does not test whether the consensus-filtered PRM can be further improved through iterative self-training or RL. This is noted as future work (Section 6) and represents a missing ablation: does additional training on the PRM's own high-confidence predictions further improve or degrade performance?

• Missing ablation — number of completions in MC estimation: The paper uses 8 completions throughout but does not ablate this number. The authors acknowledge (Section 3.1.4) that more completions would reduce variance but "the associated costs may outweigh the incremental benefits." An experiment varying the number of completions (e.g., 4, 8, 16, 32) and measuring downstream PRM quality would have strengthened the claim that 8 is a pragmatic but suboptimal choice.

Critical Assessment

Claim: MC estimation yields inferior performance and generalization compared to LLM-as-a-judge and human annotation.

What the experiments demonstrate: On PROCESSBENCH, this claim is unambiguous: MC estimation (authors' 860K) achieves 40.1 F1 vs. LLM-as-a-judge's 46.5 and human annotation's 56.5 (Table 4). The performance gap is large and consistent across all four PROCESSBENCH sub-categories, with the most dramatic differences on harder problems.

What is less clear: On BoN, the claim does not hold — MC estimation actually achieves the highest BoN scores (65.9 vs. 65.3 for LLM-as-a-judge and 64.9 for human annotation, Table 3). The paper's framing of this as "inferior performance" therefore depends entirely on valuing process verification over answer selection, or on using the dual-evaluation framework to argue that BoN alone is misleading. An organization that only cares about answer accuracy in a BoN deployment would, based on these results, rationally prefer MC estimation. The paper's argument that BoN is inflated and misleading (Section 3.2) is well-supported, but the claim of "inferior performance" requires accepting this argument — it is not a direct empirical fact.

Generalization claim: Human annotation's superior PROCESSBENCH performance on OlympiadBench and Omni-MATH despite being trained only on MATH data is strong evidence for generalization. However, the generalization claim for LLM-as-a-judge vs. MC estimation is weaker: LLM-as-a-judge does generalize better to hard problems (OlympiadBench 39.4 vs. 19.4, Omni-MATH 36.1 vs. 19.8), but both are trained on the same 860K query-response pairs. The difference is in annotation quality, not in domain coverage of the underlying queries.

Missing experiments: The paper does not test whether MC estimation's weaknesses can be mitigated by using a better completion model. All MC estimation uses Qwen2.5-Math-Instruct for completions. Would a stronger completion model (e.g., a 72B model for 7B response evaluation) reduce false positives and false negatives? The paper's theoretical argument (Section 3.1.1) suggests this wouldn't fundamentally solve the problem — a perfect completion model would still reach correct answers from incorrect steps if it "corrects" errors — but the empirical question remains open.

Claim: MC estimation's failure stems from a fundamental category error — it estimates value, not reward.

What the experiments demonstrate: The scoring strategy analysis (Figure 9) provides the strongest direct evidence: MC-trained PRMs perform best with "last" score aggregation (value-like behavior), while human-annotation-trained PRMs perform best with "product" (reward-like behavior). This behavioral difference is consistent with the value-vs-reward distinction and would be difficult to explain otherwise.

The BoN-PROCESSBENCH inverse relationship (Figure 7) also supports this: if MC estimation were simply a noisier version of human annotation, it would score lower on both metrics, not higher on one and lower on the other. The fact that MC-trained models are better at answer selection (BoN) but worse at process verification (PROCESSBENCH) is exactly what you'd expect from a value model: good at predicting outcomes, bad at evaluating intermediate states.

What is not directly tested: The paper does not train an explicit value model (e.g., with temporal-difference learning or Monte Carlo returns with discounting) and compare it to the MC-estimation-trained PRM. Such a comparison would directly test whether MC estimation produces value-like behavior, or whether the observed behavior has some other explanation (e.g., the last-step score being more informative due to position bias in the training data). The evidence is behavioral and inferential, not causal.

Claim: Conventional BoN evaluation is biased and leads to inflated PRM performance estimates.

What the experiments demonstrate: This claim is strongly supported by multiple lines of evidence:

1. Figure 6 shows that 5.1–43.4% of correct-answer solutions contain process errors, with rates increasing with problem difficulty. This establishes that the phenomenon (correct answers from flawed reasoning) is real and prevalent.

2. Figure 7 and Table 5 show that most PRMs detect fewer than 30% of process errors in correct-answer solutions. This establishes that BoN scores are inflated — the PRMs are selecting responses whose answers happen to be correct, not responses whose reasoning is sound.

3. Figure 8 shows that many PRMs have 40–55% of their minimum scores at the final answer step, indicating they are primarily judging answers, not processes. This provides a mechanistic explanation for the inflation.

4. The BoN-PROCESSBENCH inverse ranking across training methods (Tables 3–4, Figure 7) shows that optimizing for BoN pushes models away from process verification — a clear sign of evaluation misalignment.

Weaknesses in the evidence: The manual annotation for Figure 6 sampled 8 responses per query and randomly chose correct-answer responses for annotation. The sample size and selection procedure are not fully detailed — how many responses total were annotated? Was there any selection bias? The number of test cases in Table 5 varies widely by benchmark (7 for GSM8K, 259 for Omni-MATH), making the average detection rates somewhat sensitive to benchmark weighting.

Missing analysis: The paper does not analyze why policy models produce correct answers with flawed processes — is it due to the model "getting lucky" with arithmetic errors canceling out? Following a plausible but incorrect derivation that coincidentally reaches the right number? The mechanism matters for understanding whether this phenomenon is specific to mathematical reasoning or generalizes to other domains.

Claim: The consensus filtering mechanism significantly improves both model performance and data efficiency.

What the experiments demonstrate: On the 3M dataset, consensus filtering improves PROCESSBENCH from 40.2 to 66.5 F1 (65% relative improvement) while reducing data volume by 50% (Figure 4). This is a genuine efficiency gain — more performance from less data. On the 860K dataset, the filtered 350K subset matches LLM-as-a-judge's full-dataset performance on PROCESSBENCH (46.3 vs. 46.5, Figure 2).

Qualifications: The comparison is between consensus-filtered data and unfiltered MC-estimated data. A fairer comparison would be: consensus-filtered MC+LLM data vs. an equivalent budget spent on pure LLM-as-a-judge annotation (i.e., using the 72B judge to annotate more data rather than filtering MC data). The paper does not test whether simply scaling up LLM-as-a-judge annotation to match the computational cost of MC estimation + LLM judging + filtering would achieve better or worse results. This is a meaningful omission because the LLM-as-a-judge annotation step (running a 72B model on all steps of 860K solutions) is computationally expensive — potentially more expensive than generating the MC completions. The filtering "saves" 60% of data but the annotation cost was already paid.

The claim of improved data efficiency is specifically about training data efficiency, not annotation efficiency. The filtered dataset is smaller, so training is faster, but the total annotation pipeline (MC estimation + LLM judging + filtering) is more expensive than either method alone.

Claim: The released PRMs achieve state-of-the-art performance.

What the experiments demonstrate: On PROCESSBENCH, this is clearly true among open-source models: Qwen2.5-Math-PRM-7B (73.5 F1) and 72B (78.3) substantially outperform all other open-source PRMs and language models (Table 7). On BoN, the 7B model (67.6) leads all 7B-scale PRMs and surpasses maj@8 (66.2); the 72B model (69.3) slightly edges the ORM baseline (68.9) (Table 6).

Qualifications and limitations:

1. Single policy model family: All BoN evaluations use Qwen2.5-Math-Instruct as the policy model. The paper does not test whether the PRMs generalize to other policy model families (e.g., Llama, DeepSeek). Given that the PRMs were trained on responses from Qwen2-Math-Instruct and Qwen2.5-Math-Instruct, there may be a distributional match advantage that wouldn't transfer.

2. Single domain: All evaluation is on mathematical reasoning. The paper does not test whether the PRMs or the consensus filtering approach work for code generation, logical reasoning, or other multi-step reasoning domains.

3. The o1-mini gap: The 72B PRM trails o1-mini by 9.6 PROCESSBENCH F1 points (78.3 vs. 87.9). This is a substantial gap, and the paper does not analyze what capabilities account for it — is o1-mini better at error detection, better at identifying all-correct solutions, or both? Table 7 shows o1-mini achieves 83.5 error recall and 95.1 correct recall on MATH vs. the 72B PRM's 74.2 and 88.2 — gaps in both dimensions.

4. The BoN ceiling: The 7B PRM achieves 67.6 BoN vs. a pass@8 upper bound of 74.7. This 7.1-point gap represents the maximum possible improvement from better response selection. The paper does not analyze what fraction of this gap is irreducible (due to all 8 candidates being wrong) vs. addressable through better PRM scoring.

5. N=8 only for main results: The primary BoN evaluation uses N=8. Appendix B.4 shows N=64 results, but only on a subset of benchmarks. The relative performance of different PRMs might change at different N values, and the paper doesn't systematically explore this.

Overall Assessment

The paper's experiments collectively make a compelling case for its central thesis: that MC estimation produces PRMs which appear effective under BoN evaluation but fail at actual process verification, that this failure is systematic rather than due to insufficient data, and that combining MC estimation with LLM-as-a-judge via consensus filtering produces models that are genuinely better at both tasks. The evidence for these claims is multi-layered (performance comparisons, behavioral analyses, failure mode characterizations) and internally consistent.

The primary limitation of the experimental design is its narrowness: all results are on mathematical reasoning, all models are from the Qwen family, all policy models used for evaluation are from the same family as the training data generators. The paper does not demonstrate that the lessons learned — about the value-reward distinction, about the BoN-PROCESSBENCH tradeoff, about consensus filtering — generalize beyond this specific ecosystem. The paper's own framing as "lessons" and "guidelines" (Section 6) is appropriately modest, but the experimental scope does not fully support prescriptive claims about how PRMs should be developed in general.

Additionally, the paper's claim that consensus filtering "significantly improves data efficiency" (Section 1) is true in the narrow sense of training data volume, but the full annotation pipeline (MC estimation + LLM judging + filtering) is more computationally expensive than pure MC estimation. A more complete accounting would compare methods at equal total annotation cost, not just equal training data volume.

6. Limitations and Trade-offs

Single Policy Model Family and Domain

The assumption or constraint. All PRM training and evaluation uses models from the Qwen2.5-Math-Instruct family (7B and 72B parameter scales) and all experiments are restricted to mathematical reasoning. The training data is generated by sampling responses from Qwen2-Math-Instruct and Qwen2.5-Math-Instruct models, and the completion model for MC estimation is Qwen2.5-Math-Instruct at the corresponding scale. The BoN evaluation uses Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct as policy models. The paper does not test PRM generalization to other model families (Llama, DeepSeek, Mistral) or other reasoning domains (code generation, logical deduction, scientific reasoning). Section 2.1 describes the data synthesis pipeline:

"we collected a large-scale dataset of approximately 500,000 queries with golden answers. For each query, we generate 6-8 diverse responses by mixing outputs from the Qwen2-Math-Instruct and Qwen2.5-Math-Instruct series models"

The consequence. There is a systematic distributional match between the PRM's training data generators, the completion models used for MC estimation, and the policy models used for evaluation — all are Qwen2.5-Math variants. This means the PRM is being evaluated in the most favorable possible setting: it was trained on responses from the same model series it is asked to evaluate. A practitioner deploying this PRM to supervise a Llama-based policy model, or to verify reasoning in a code generation task, would be operating in an untested regime. The consensus filtering mechanism's effectiveness may depend on the specific characteristics of Qwen models — the LLM-as-a-judge step uses Qwen2.5-72B-Instruct as the critic, and its error patterns may correlate with those of the generator models in ways that inflate the apparent filtering benefit. More broadly, the paper's central lessons (MC estimation trains value models, not reward models; BoN evaluation inflates performance) are argued to be general, but the empirical evidence comes entirely from a single ecosystem.

What evidence exists in the paper. The paper itself provides indirect evidence of the distributional sensitivity: Table 9 shows that the 7B PRM, which successfully supervises Qwen2.5-Math-7B-Instruct (67.6 BoN, above maj@8's 66.2), fails to supervise Qwen2.5-Math-72B-Instruct (69.9 BoN, below maj@8's 70.2). The performance inversion across only model scales within the same family suggests that PRM quality is sensitive to the capability gap between verifier and generator. Cross-family generalization is never tested — there are no experiments with non-Qwen policy models, and the PROCESSBENCH evaluation (Table 7) only measures error detection on static, pre-written solutions, not on live generations from diverse policy models.

Mitigation status. The paper does not address this limitation explicitly. Section 6 (Conclusion) frames the contributions as "practical guidelines and best practices for future research and development for reasoning process supervision" and the Limitation section acknowledges that "the best practices for utilizing PRMs in reinforcement learning remain unexplored," but the single-ecosystem scope is not mentioned as a limitation. Future work would need to replicate the key findings — the MC estimation vs. LLM-as-a-judge comparison, the BoN-PROCESSBENCH inverse relationship, and the consensus filtering approach — on different model families and reasoning domains to establish generality.

---

The Consensus Filtering Pipeline Is More Computationally Expensive Than Pure MC Estimation, but This Cost Is Not Accounted For

The assumption or constraint. The paper frames consensus filtering as improving "data efficiency" because the filtered dataset is ~40–50% the size of the original. However, producing this filtered dataset requires running both the MC estimation pipeline (8 completions per step) and the LLM-as-a-judge pipeline (a full forward pass of Qwen2.5-72B-Instruct on every step of every solution). The total annotation cost for a single instance is therefore:

• MC estimation: 8 completions per step × number of steps × cost of Qwen2.5-Math-Instruct inference
• LLM-as-a-judge: 1 full evaluation per step × number of steps × cost of Qwen2.5-72B-Instruct inference

The consensus filtering step then discards 50–60% of this doubly-annotated data. The paper reports "data efficiency" improvements in terms of training data volume — the model trains on fewer examples — but the annotation cost to produce those training examples is substantially higher than either method alone.

The consequence. A practitioner deciding how to allocate their annotation budget faces a more complex tradeoff than the paper's framing suggests. Given a fixed compute budget for data construction, the options include: (a) pure MC estimation on a very large dataset, (b) pure LLM-as-a-judge on a moderate dataset, (c) consensus filtering on a smaller dataset (since the dual annotation costs more per retained instance), or (d) human annotation on a small dataset. The paper's comparisons in Tables 3–4 and Figure 2 show performance at fixed training data volume (same number of initial query-response pairs, approximately), not at fixed annotation cost. At equal annotation FLOPs, a pure LLM-as-a-judge approach might annotate more total instances than the consensus-filtering pipeline (since it skips the MC estimation step) and potentially match or exceed the consensus-filtered model's performance. This comparison is never made.

What evidence exists in the paper. The paper provides the retention rate (Figure 2: ~40% of 860K data preserved; Section 3.1.4: 1.5M of 3M preserved) and the performance of each component method (Tables 3–4), but never provides a FLOPs-based cost accounting for the data construction pipeline. The LLM-as-a-judge implementation uses Qwen2.5-72B-Instruct — a 72B parameter model — to annotate every step of every solution in the 860K dataset. That is a very large computational investment, likely exceeding the cost of the MC estimation step (which uses same-scale completion models but generates only 8 completions per step rather than a full analysis). The paper does not report the wall-clock time, GPU-hours, or estimated FLOPs for any data construction method. Section 3.2.5 states simply that "the limited process verification capability makes PRMs demonstrate tolerance for the responses with correct answers but flawed reasoning processes," without addressing the cost of the solution.

Mitigation status. The paper does not acknowledge this as a limitation. The term "data efficiency" (used in the abstract, Section 1, and Section 3.1.5) refers exclusively to training data volume, not to annotation cost. This is a significant omission because the headline claim of improved efficiency could be misinterpreted as end-to-end cost reduction. A full accounting would need to compare: (total annotation FLOPs) vs. (downstream PRM performance), not (training dataset size) vs. (downstream PRM performance).

---

The o1-mini Gap: Even the Best Released PRM Falls Substantially Short of Proprietary Frontier Performance on Process Error Identification

The assumption or constraint. The paper's strongest PRM, Qwen2.5-Math-PRM-72B, achieves 78.3 PROCESSBENCH F1. This is state-of-the-art among open-source models (the next best is Qwen2.5-Math-7B-PRM800K at 56.5). However, o1-mini achieves 87.9 F1 — a gap of 9.6 points, or roughly 12% relative improvement. The paper does not deeply analyze what accounts for this gap or whether the consensus filtering approach can close it.

The consequence. Process error identification is the defining task of a PRM — the entire rationale for process-level supervision over outcome-level supervision. The fact that even the paper's best model misses roughly 1 in 5 process errors (or 1 in 10, depending on the metric) on MATH-level problems, and substantially more on harder problems (OlympiadBench error recall: 67.9 for the 72B PRM vs. 80.2 for o1-mini), means that in high-stakes applications where reasoning fidelity is critical, the current state-of-the-art open PRMs are not yet reliable enough to trust. The gap to o1-mini also raises the question of whether the consensus filtering approach — which the paper presents as a principled solution to MC estimation's failures — has a fundamental performance ceiling, or whether scaling up model size, data quantity, or judge quality could close the gap. The paper does not establish whether the o1-mini gap is due to better training data, better base model capabilities, scale, or some combination.

What evidence exists in the paper. Table 7 provides the per-category breakdown. The gap between Qwen2.5-Math-PRM-72B and o1-mini exists across all four PROCESSBENCH sub-categories:

The gap widens on harder problems (OlympiadBench, Omni-MATH), suggesting that the PRM's error detection capability degrades more sharply with problem difficulty than o1-mini's does. The error detection recall in particular shows large gaps: on MATH, PRM-72B achieves 74.2 vs. o1-mini's 83.5; on OlympiadBench, 67.9 vs. 80.2. This means the PRM is missing roughly 25–32% of actual errors on these harder problem sets.

Mitigation status. The paper acknowledges the o1-mini gap implicitly by including o1-mini in the comparison (Table 7) but does not discuss it as a limitation or analyze its causes. The Limitation section (Section 6 Conclusion) states only that "there exists a considerable performance gap between our PRMs and the BoN upper bound (pass@8), suggesting substantial optimization potential" — this refers to the BoN gap, not the PROCESSBENCH gap to o1-mini. The paper does not hypothesize what o1-mini does differently (better base model? RL training? different data?) or whether the consensus filtering approach can realistically reach o1-mini-level performance with further scaling.

---

The Difficulty-Estimation Bottleneck: The Method Provides No Mechanism for Allocating Compute Differently Across Problems

The assumption or constraint. The paper evaluates PRMs uniformly: every problem gets the same N = 8 candidate solutions in BoN, and every solution gets the same per-step scoring procedure. There is no difficulty estimation, no adaptive allocation of test-time compute, and no mechanism for determining when the PRM's verification is reliable vs. when it is likely to fail. The paper demonstrates (implicitly, through the per-benchmark breakdowns) that PRM performance varies dramatically with problem difficulty: on GSM8K (easiest), the 72B PRM achieves 87.3 PROCESSBENCH F1; on Omni-MATH (hardest), it achieves 71.1. But this variation is observed post-hoc — the PRM itself is not equipped to estimate its own confidence or to signal when a problem exceeds its reliable operating range.

The consequence. In a deployment setting, a practitioner needs to know not just the average PRM performance but whether the PRM is reliable on this specific problem. The paper provides no per-instance confidence measure, no calibration analysis, and no difficulty-aware deployment strategy. A PRM that achieves 78.3 average PROCESSBENCH F1 but 64.8 error recall on Omni-MATH (Table 7) will miss more than one-third of process errors on the hardest problems — but the system cannot warn the user that its verification is less reliable on these inputs. This is particularly consequential for the paper's framing of PRMs as enabling "finer-grained supervision on the reasoning process" (Section 1): if the PRM is blind to certain classes of errors on hard problems, it provides a false sense of security exactly where human oversight is most needed.

What evidence exists in the paper. The per-category PROCESSBENCH results (Table 7) and the per-benchmark BoN results (Table 6) clearly show difficulty-dependent performance variation. The BoN gap between the 72B PRM and maj@8 varies from -0.3 points on GSM8K (96.4 vs. 96.0 with the 72B policy model, Table 9) to +4.8 points on Minerva Math (46.0 vs. 41.2, Table 6), demonstrating that the PRM's value-add over simple majority voting is highly problem-dependent. The extracted PROCESSBENCH analysis (Table 5) shows that PRM error detection accuracy on correct-answer-but-wrong-process cases ranges from 58.1% (72B PRM average) to well below 30% for most baselines. None of this variation is surfaced to the user or used to adapt the verification strategy.

Mitigation status. The paper does not address this limitation. The evaluation framework is purely aggregate (average accuracy, average F1) with per-benchmark breakdowns provided for analysis but not integrated into any adaptive mechanism. The Limitation section does not mention difficulty estimation or confidence calibration. This is a notable gap given that other work in the test-time compute literature (as in the reference example paper) has demonstrated that difficulty-adaptive allocation can yield ~4× efficiency improvements. Integrating difficulty awareness into PRM deployment — perhaps by using the PRM's own score distribution as a confidence signal, or by training a separate difficulty estimator — would be a natural extension.

---

The MC Estimation Threshold and Hard-Label Design Choices Are Empirically Driven but Not Theoretically Justified, Limiting Transferability

The assumption or constraint. The paper makes specific, strongly-recommended design choices about MC estimation: use threshold 0 (label a step correct if any of 8 completions succeeds), use hard labels rather than soft labels, and use 8 completions per step. These choices are empirically validated within the paper's experimental setup — Figure 5 shows monotonic degradation as the threshold increases, and Figures 3–4 show hard labels outperforming soft labels after consensus filtering. However, the paper provides no theoretical framework for predicting whether these choices would transfer to different settings. Would threshold 0 still be optimal if the completion model were substantially better (or worse)? Would 8 completions still be the right tradeoff if the cost of completions changed relative to the cost of LLM-as-a-judge? Would hard labels still dominate soft labels if the consensus filtering were done with a different judge model?

The consequence. A practitioner adapting the consensus filtering approach to a new domain, a new model family, or a different compute budget cannot confidently apply the paper's recommended settings. The choice of threshold 0 is the most consequential: it means that a step is labeled positive if any of the 8 completions succeeds, which is maximally permissive about labeling steps as correct, minimizing false negatives at the cost of maximizing false positives. In the paper's setting, this works because MC estimation's limiting failure mode is false negatives (completion model fails from correct steps) and the consensus filter subsequently removes many false positives (where MC says positive but LLM-as-a-judge says negative). But if the completion model were much stronger — producing fewer false negatives — a higher threshold might be optimal. Conversely, if the judge model were weaker — producing more false negatives in its own right — the consensus filter might remove too many true positives, and a different threshold might be needed. The paper provides no framework for reasoning about these tradeoffs; the recommendations are purely empirical and potentially brittle to changes in the underlying models.

What evidence exists in the paper. Figure 5 shows the threshold sweep on the 3M MC-estimated dataset (before filtering), and the monotonic degradation from threshold 0 to 7/8 is clear. The hard-vs-soft comparison (Figures 3–4) shows that the label type matters only after filtering. However, these experiments use a single completion model (Qwen2.5-Math-Instruct), a single number of completions (8), and a single judge model (Qwen2.5-72B-Instruct). The paper does not ablate the number of completions (4, 8, 16, 32), does not test different completion models, and does not test whether the threshold optimum shifts when the consensus filtering step uses a different judge. Section 3.1.4 acknowledges the variance issue:

"only 8 completions for step correctness estimation exhibit high variance and are relatively crude. Although we can achieve better estimation accuracy by increasing the number of completions, the associated costs may outweigh the incremental benefits."

But this acknowledgment is qualitative — no experiment varies the number of completions and measures the downstream effect on PRM quality.

Mitigation status. The paper presents its recommendations as practical guidelines derived from the experimental results (Section 3.1.5: "if we have to rely on MC estimation for step-wise correctness verification, we suggest setting the threshold to 0"), but does not caveat these recommendations with the conditions under which they were derived. The Limitation section does not address the transferability of these design choices. A more robust contribution would include a sensitivity analysis showing how the optimal threshold varies with completion model quality, or a theoretical argument for why threshold 0 is optimal under broad conditions (e.g., if the false negative rate of MC estimation exceeds the false positive rate by a sufficient margin).

---

The Revision Model and Reinforcement Learning Integration Are Not Explored, Leaving the PRM's Downstream Utility Incompletely Characterized

The assumption or constraint. The paper evaluates PRMs in two modes: post-hoc response selection (BoN) and error identification (PROCESSBENCH). It does not evaluate PRMs in their most impactful potential application: providing process-level feedback for reinforcement learning or iterative self-improvement. The PRM-guided greedy search experiment (Appendix A, Table 8) is a limited step in this direction — it uses the PRM to select among candidate next steps during generation — but this is not RL, not iterative refinement, and not self-correction. The paper explicitly acknowledges this gap in the Limitation section:

"the best practices for utilizing PRMs in reinforcement learning remain unexplored."

Additionally, the paper does not test whether the PRM can be used as a reward signal for training better policy models through RLHF-style fine-tuning, or whether the consensus filtering approach produces PRMs that are more effective as RL reward models than MC-estimation-trained PRMs.

The consequence. The paper's central argument — that MC-estimation-trained PRMs are value models in disguise and that consensus filtering produces genuine process verifiers — has its strongest test in the RL setting. If the paper's thesis is correct, then using a consensus-filtered PRM as a reward model for RL fine-tuning should produce policy models with better reasoning processes (fewer logical errors, more reliable intermediate steps) compared to using an MC-estimation-trained PRM, even if both achieve similar final-answer accuracy. Conversely, if an MC-estimation-trained PRM and a consensus-filtered PRM produce similar downstream policy improvements when used as RL reward models, the practical significance of the paper's diagnostic framework would be diminished — it would mean the value-reward distinction matters for error identification benchmarks but not for actually improving models. Without RL experiments, the paper's claims about the superiority of consensus-filtered PRMs are restricted to the evaluation benchmarks and do not extend to the use case that most motivates PRM development in the first place (enabling finer-grained supervision for model improvement).

A secondary consequence: the PRM-guided greedy search results (Table 8) show only marginal benefits over ORM-based BoN (66.6 vs. 66.5 for the 72B PRM), and the paper itself notes that greedy search may be suboptimal for PRMs because "the highest-scoring step may not be the correct one." This suggests that even the paper's strong PRMs may not be directly suitable as search guidance without additional mechanisms (lookahead, backtracking, value estimation). The paper's PRMs verify current-step correctness; they do not estimate whether a correct step leads to a solvable subproblem. This limitation — which follows directly from the paper's own value-reward distinction in Section 3.1.1 — means that additional research is needed to make these PRMs useful for guided generation, and the paper does not explore what that research would look like.

What evidence exists in the paper. Appendix A (Table 8) provides the only downstream application beyond BoN selection. The results show modest gains at best: the 72B PRM with greedy search@8 achieves 66.6 average accuracy vs. ORM Best-of-8 at 66.5, while the 7B PRM with greedy search achieves only 63.5. The pass@1 baseline (single greedy generation, no search) is 57.4, and maj@8 is 62.5 — so PRM-guided search adds 4–9 points over the no-search baseline, but most of this gain is achievable with simple ORM Best-of-8 (66.5). The paper's conclusion that "there is still significant potential for exploration in the future regarding more appropriate search strategies" (Appendix A) acknowledges that the current PRMs are not plug-and-play solutions for guided generation.

Mitigation status. The paper explicitly lists this as a limitation: "the best practices for utilizing PRMs in reinforcement learning remain unexplored" (Section 6, Limitation). It also suggests future work on "combining rewards and values to simultaneously consider both the correctness of the current step and the possibility of reaching the correct future outcomes" (Appendix A). However, the paper does not provide even a preliminary experiment — e.g., using the PRM as a reward model for a single RL training run and comparing against an ORM baseline — that would establish the practical relevance of the consensus filtering approach for the most important downstream application. This is a significant gap in the evidence base for the paper's central claims.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture or training objective. Instead, it provides something arguably more consequential for a maturing field: a diagnostic framework that explains why the dominant approach to PRM data construction has been systematically failing, paired with a corrective that is simple enough to adopt immediately. The magnitude is not paradigm-shifting in the sense of inventing Process Reward Models — those already existed. Rather, it is a reframing that resolves a latent contradiction in the literature and redirects research effort toward more productive bottlenecks.

The central reframing — MC estimation trains value models, not reward models — is the pivot on which this contribution turns. Prior to this paper, the community implicitly treated MC estimation as a noisy approximation to human annotation: more completions, better completion models, or more data would eventually close the gap. The paper demolishes this assumption by showing that the two methods teach fundamentally different functions. The evidence is the BoN-PROCESSBENCH inverse ranking (Figure 7, Section 3.2.2): MC-trained PRMs are best at BoN (65.9) but worst at process verification (40.1 F1); human-annotation-trained PRMs are best at process verification (56.5 F1) but worst at BoN (64.9). This is not a noise pattern — it is a signature of different underlying quantities being estimated. If MC estimation were merely a noisier version of human labels, both metrics would degrade together. The fact that they move in opposite directions is the closest thing to a mathematical proof that these two data sources produce categorically different models.

This reframing resolves the tension that has quietly plagued the PRM literature: why do MC-estimation-trained PRMs look good on BoN (Wang et al., 2024b; Luo et al., 2024) while human-annotation-trained PRMs like PRM800K (Lightman et al., 2023) are widely considered higher quality? The answer is that BoN rewards outcome-based scoring — exactly what value models are good at — and penalizes the process-scrupulous behavior of genuine reward models. Different papers reported positive results for different methods because they were optimizing for different (and partially opposing) objectives, without recognizing the distinction. After this paper, any PRM evaluation that relies solely on BoN will carry the burden of proof: the researcher must demonstrate that their model is not simply an ORM in PRM clothing (Figure 8, where EurusPRM-Stage1 has 54.6% of minimum scores at the final step).

The paper also shifts the practical bottleneck from data quantity to data quality, but in a specific and actionable way. Prior work treated the PRM data problem as one of scale: get more queries, generate more responses, run more completions. The paper's data scaling results (Tables 3–4 vs. Figures 3–4) show that pure MC estimation hits diminishing returns almost immediately — doubling from 440K to 860K samples barely moves PROCESSBENCH (28.9 → 40.1 F1), and tripling to 3M produces almost no further gain (40.1 → 40.2). The bottleneck is label noise, specifically the systematic false positives that arise when completion models "repair" errors from incorrect intermediate steps. This means that throwing more compute at the MC estimation pipeline is a dead end. The field's compute budget should shift from scaling MC-estimated data volume to investing in better annotation methods — LLM-as-a-judge, human annotation, or consensus filtering.

This makes the consensus filtering mechanism (Section 3.1.3) more significant than it might appear as a standalone method. It is not just a clever trick for cleaning data; it is the first practical demonstration that crossing two annotation sources with uncorrelated error modes can break through the quality ceiling that pure MC estimation hits. The retention rate of ~40–50% (Section 3.1.3) is not a cost — it is a measurement of how much noise exists in MC-estimated labels. The fact that the surviving 40% of data trains a PRM that matches full-dataset LLM-as-a-judge performance (Figure 2: 46.3 vs. 46.5 PB F1) means that most of the information in the MC-estimated dataset is redundant or actively harmful for the purpose of training process verifiers. This is a strong result that should change how practitioners think about data curation for reward modeling.

Research directions that become more attractive:

• Multi-source annotation triangulation. If two sources (MC + LLM-as-a-judge) produce substantial gains through consensus, what about three or four? Adding human annotation on a small subset, or using multiple judge models with different biases, or incorporating execution-based verification (running code in a sandbox) could further improve label quality. The paper's framework provides a natural experimental template: measure performance on both BoN and a step-level benchmark, vary the annotation sources, and track both metrics.

• Process-level evaluation as a standard. The paper makes a compelling case that PROCESSBENCH (or step-level error identification more broadly) must become a standard supplement to BoN in PRM evaluation. The evidence that BoN alone is actively misleading (Section 3.2.2) is strong enough that future PRM papers omitting step-level evaluation will face legitimate skepticism.

• Training explicit value models for search. The paper's distinction between PRMs (reward, current-step correctness) and value models (future outcome probability) in Section 3.1.1 opens the door to training both explicitly and using them for different purposes: value models for guiding search/planning, PRMs for verification and error identification. The Appendix A results (PRM-guided greedy search barely outperforms ORM BoN) underscore that PRMs alone are insufficient for search guidance. This is not a failure — it is a precise specification of what PRMs are and are not good for.

Research directions that become less attractive:

• Scaling pure MC estimation in hopes of matching human annotation quality. The paper's evidence that MC estimation hits a quality ceiling (PB F1 ~40 regardless of data scale) makes this approach difficult to justify without additional innovations.

• BoN-only PRM evaluation. The paper demonstrates that BoN scores are inflated, that BoN optimization drives PRMs toward ORM-like behavior, and that BoN rankings are inverted relative to process verification rankings. A researcher reporting only BoN results for a new PRM now faces the obvious objection: "have you checked whether your model can actually identify process errors, or is it just another ORM?"

• Simple ensembling of MC estimation runs. The paper shows that MC estimation's errors are systematic (false positives from completion model error correction), not just random. Taking the average of multiple MC estimation runs — or using more completions per step — would reduce variance but not address the systematic bias. The paper's diagnostic framework explains why this will not work.

Follow-Up Research This Work Enables

Cross-model-family replication of the BoN-PROCESSBENCH inverse relationship. The paper's central empirical finding — that MC estimation, LLM-as-a-judge, and human annotation produce PRMs with opposite performance orderings on BoN vs. PROCESSBENCH — is demonstrated entirely within the Qwen ecosystem: Qwen generators, Qwen completers, Qwen judges, Qwen policy models for evaluation. A strong follow-up would replicate this experiment using a different model family (e.g., Llama-3-based generators and completions, with both Llama-based and Qwen-based judges) and measure whether the inverse relationship persists. If it generalizes, the value-reward distinction becomes a robust finding about PRM training, not a Qwen-specific artifact. If it fails to generalize — if, for instance, a stronger completion model eliminates the false-positive problem in MC estimation — that would refine our understanding of when MC estimation is and is not appropriate. The key measurements would be Table 4 analog (per-category PROCESSBENCH F1 across data construction methods) and Figure 7 analog (extracted PROCESSBENCH detection accuracy vs. BoN for each training data source).

Consensus filtering with execution-based verification as a third annotation source. The paper uses two annotation sources (MC estimation and LLM-as-a-judge) that have complementary error modes. A natural extension adds a third source: execution-based verification, where mathematical reasoning steps are checked by running code (e.g., sympy for algebraic manipulation, numerical evaluation for arithmetic). For problems where reasoning can be expressed as executable code, each step's computations could be verified deterministically rather than probabilistically. This would provide a signal with yet another error mode — execution catches calculation errors that LLM-as-a-judge might miss, but cannot verify logical reasoning that is not executable. A three-way consensus filter (MC + LLM + execution) would test whether adding more independent annotation sources continues to improve quality, or whether two sources already capture most of the gain. The experiment would compare: (a) MC-only, (b) MC + LLM consensus (the paper's method), (c) MC + LLM + execution consensus, all at equal training data volume, on both BoN and PROCESSBENCH. The hypothesis is that execution-based verification should further reduce false positives on arithmetic-heavy problems (OlympiadBench, MATH), improving error detection recall.

Using the scoring-strategy diagnostic to audit black-box PRMs. The paper shows (Section 3.2.4, Figure 9) that the optimal solution-level scoring strategy — last-score vs. product vs. minimum — reveals whether a PRM has learned a value-like or reward-like function. This finding can be operationalized as a post-hoc audit tool for any trained PRM, including proprietary models accessible only via API. The procedure: for a given PRM, run BoN evaluation at N = 8 or 16 on a standard benchmark, but compute three separate prm@N scores using last-score, product, and minimum aggregation. If last-score substantially outperforms product, the model is likely value-trained (or has drifted toward outcome-based assessment). If product/minimum outperform last, it is likely reward-trained. This audit could be applied to models like GPT-4's internal verifier, Claude's process supervision, or future open-source PRMs to assess whether they are genuinely evaluating reasoning processes or have degenerated into ORMs, without requiring access to training data or PROCESSBENCH-style annotated error locations. A research paper conducting this audit across 10+ publicly available PRMs would provide a useful taxonomy of which models actually do process verification.

RL fine-tuning comparison: MC-trained PRM vs. consensus-filtered PRM as reward models. The paper explicitly identifies the lack of RL experiments as a limitation. A direct follow-up would train two policy models using RL (e.g., PPO or ReST-style iterative fine-tuning) with the same base policy but different reward models: (a) an MC-estimation-trained PRM (Qwen2.5-Math-7B-PRM-MC-hard from Table 1), and (b) the consensus-filtered PRM (Qwen2.5-Math-PRM-7B from Table 6). The evaluation would measure not just final-answer accuracy (standard RL benchmarks) but also process quality — perhaps by manually annotating the trained policy's solution steps for logical errors, or by using PROCESSBENCH to evaluate the policy's reasoning quality if it generates solutions to PROCESSBENCH problems. The paper's thesis predicts that the consensus-filtered PRM should produce policies with fewer process errors at equal or better final-answer accuracy, because its reward signal actually penalizes flawed intermediate steps rather than only rewarding answer correctness. A null result — both PRMs producing indistinguishable policy quality — would weaken the practical case for consensus filtering, while a positive result would demonstrate that the value-reward distinction has concrete downstream consequences beyond benchmark scores.

Difficulty-adaptive PRM deployment with confidence calibration. The paper shows (Table 7, Table 6) that PRM performance varies dramatically across difficulty levels — the 72B PRM achieves 87.3 F1 on GSM8K error detection but only 64.8 error recall on Omni-MATH. A practical follow-up would develop a confidence estimation mechanism for the PRM, where the model outputs not just a per-step correctness score but an estimate of its own uncertainty about that score (e.g., via Monte Carlo dropout, ensemble disagreement across multiple PRM variants, or a separate confidence head). At deployment time, a PRM with high confidence in its error detection could be trusted; a PRM with low confidence could escalate to a more expensive verification method (human review, a larger judge model, execution-based checking). The experiment would measure: (a) calibration of the confidence estimates (do low-confidence predictions correspond to higher error rates?), (b) the cost-accuracy tradeoff of an adaptive system that allocates verification budget based on confidence, compared to uniform PRM verification. This directly extends the paper's framework — the paper diagnoses that PRMs are unreliable on hard problems; the follow-up builds a system that knows when it is unreliable.

Scaling the LLM-as-a-judge annotator quality and measuring downstream PRM performance. The paper uses Qwen2.5-72B-Instruct as the judge model. A natural question: how does PRM quality scale with judge quality? This could be tested by using progressively stronger judges — Qwen2.5-72B-Instruct, QwQ-32B-Preview, GPT-4o, o1-mini — to annotate the same set of query-response pairs, then training separate PRMs on each judge's labels (both alone and in consensus with MC estimation). The hypothesis is that stronger judges produce higher-quality labels, which should improve PROCESSBENCH performance of the resulting PRM, but with diminishing returns as judge quality approaches the Bayes error rate for mathematical step verification. This experiment would also reveal whether the BoN-PROCESSBENCH inverse relationship persists with very high-quality judges — if even o1-mini-level annotation produces lower BoN than MC estimation, that would be strong evidence that the tradeoff is fundamental (reward models necessarily penalize some correct-answer-but-flawed-process solutions that BoN rewards), rather than an artifact of imperfect annotation.

Practical Applications and Downstream Use Cases

Automated error detection in math tutoring and education platforms. The PROCESSBENCH results (Table 7) demonstrate that Qwen2.5-Math-PRM-7B can identify erroneous steps in mathematical reasoning with 73.5 average F1, detecting 72.0% of errors on GSM8K-level problems and 55.7% on OlympiadBench-level problems. For an automated math tutoring system that checks student solutions step by step, this is a deployable capability: the PRM can flag likely errors for human tutor review or provide targeted feedback at the specific step where the reasoning went wrong, rather than only indicating that the final answer is incorrect. The 7B model size makes this feasible for on-device or low-latency deployment in educational software. The key number for this use case is the error recall (not just F1): 72.0% on GSM8K, 68.0% on MATH, 55.7% on OlympiadBench for the 7B model (Table 7). A tutoring platform could set expectations accordingly — catching ~70% of errors on standard problems, with the understanding that harder problems will see more missed errors.

Quality filtering for LLM-generated reasoning datasets. When constructing training data for math-tuned LLMs through rejection sampling or self-improvement pipelines (STaR, ReST), practitioners typically filter based on final-answer correctness — keeping solutions that reach the right answer and discarding those that don't. The paper's Figure 6 shows that this filter is insufficient: 11.9% of correct-answer MATH solutions and 27.4% of correct-answer OlympiadBench solutions contain process errors. Using Qwen2.5-Math-PRM-7B as an additional filter — discarding solutions where the PRM assigns a low score to any intermediate step, even if the final answer is correct — would improve the reasoning quality of the resulting training data at a modest computational cost (one PRM forward pass per solution, 7B parameters). The specific benefit: on OlympiadBench-level problems, pure answer-based filtering would retain ~27% of training examples with flawed reasoning; adding PRM-based filtering with 55.7% error recall (Table 7) would catch roughly half of those, reducing process-error contamination in the training set from ~27% to ~12% of correct-answer solutions. This is directly actionable for any team currently using MC-estimation-based data filtering.

PRM-based response selection in LLM evaluation and benchmarking. The BoN results in Table 6 show that Qwen2.5-Math-PRM-7B improves over majority voting by 1.4 percentage points on average across seven benchmarks when selecting among 8 candidate solutions from Qwen2.5-Math-7B-Instruct. For benchmark evaluation pipelines that report best-of-N or pass@N metrics, replacing majority voting with PRM-based selection provides a small but consistent accuracy improvement without changing the underlying policy model. The improvement varies by benchmark — from ~0 points on GSM8K (where maj@8 is already near ceiling) to +3.2 points on OlympiadBench (47.6 vs. 44.4). The practical value is most significant on harder benchmarks where majority voting is far from the pass@N ceiling: on Minerva Math, the 7B PRM achieves 42.6 vs. maj@8's 41.2, and the 72B PRM achieves 46.0 — a 4.8-point improvement that closes 60% of the gap to pass@8 (49.3). For organizations that regularly run LLM evaluations on mathematical reasoning benchmarks, swapping majority voting for PRM-based selection is a low-effort change with measurable gains.

When to Prefer This Method

The paper does not articulate a formal tradeoff matrix against named alternatives for deployment-time decisions. It positions consensus filtering as superior to pure MC estimation for training data construction, and it advocates dual evaluation (BoN + PROCESSBENCH) over BoN-only evaluation. However, the paper does not frame "when to use consensus-filtered PRMs vs. MC-estimated PRMs vs. ORMs vs. LLM-as-a-judge for response selection" as an explicit decision problem. The experimental results imply — but do not state as prescriptive rules — the following guidance:

• For training PRMs, prefer consensus filtering over pure MC estimation when process verification fidelity matters (which should be always, given that process verification is the purpose of a PRM). The cost is higher annotation compute (MC + LLM judging), but the quality gain is substantial: PB F1 improves from ~40 to ~66 at the 7B scale on the 3M dataset (Figure 4).

• For evaluating PRMs, use both BoN and a step-level benchmark. Evaluating on BoN alone will select for models that have drifted toward outcome-based assessment, as evidenced by the BoN-PROCESSBENCH inverse ranking (Tables 3–4).

• For response selection (BoN deployment), the paper's PRMs achieve better BoN than both majority voting and competing open-source PRMs (Table 6), and competitive or slightly better performance than ORMs. The choice between a PRM and an ORM for BoN depends on whether the additional per-step computation (vs. single-score ORM) is justified by the accuracy gain — which in the paper's results is modest (~0.4 points on average for the 72B PRM vs. the 72B ORM).

• For search-based decoding, the paper demonstrates (Appendix A, Table 8) that PRM-guided greedy search is not clearly superior to ORM-based BoN. The paper suggests that value models may be more appropriate for search, and the PRMs developed here are more suited for verification than for planning. This is not a prescriptive recommendation but an empirically grounded caution.

The paper does not provide a decision framework for choosing between its PRMs and LLM-as-a-judge for BoN selection, though Appendix B.5 provides the raw comparison data (Tables 11–12 show competitive performance between PRM-based selection and LLM-as-a-judge pairwise comparison). The tradeoff is computational: PRM scoring requires one forward pass per solution (N passes for N candidates), while pairwise LLM-as-a-judge requires N−1 comparisons, each a full LLM call. The paper does not analyze this cost-accuracy tradeoff explicitly.